# vCluster Documentation ## docs - [vCluster Docs](https://vcluster.com/docs/index.md) ### search - [Search the documentation](https://vcluster.com/docs/search.md) ### platform - [What is vCluster Platform?](https://vcluster.com/docs/platform.md) - [Access keys](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) - [Use the vCluster Platform as an OIDC provider](https://vcluster.com/docs/platform/administer/authentication/oidc-provider.md) - [Reset Admin Password](https://vcluster.com/docs/platform/administer/authentication/reset-admin-password.md) - [OIDC token exchange](https://vcluster.com/docs/platform/administer/authentication/token-exchange.md) - [Overview](https://vcluster.com/docs/platform/administer/bare-metal/overview.md) - [Configure the Agent](https://vcluster.com/docs/platform/administer/clusters/advanced/agent-config.md) - [Agent Upgrades](https://vcluster.com/docs/platform/administer/clusters/advanced/cluster-upgrades.md) - [DERP relay configuration](https://vcluster.com/docs/platform/administer/clusters/advanced/derp-relay.md) - [Platform External Database](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database.md) - [Deploy vCluster Platform with an External Database](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/deploy.md) - [Authenticate to an External PostgreSQL Database with Kerberos](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/kerberos.md) - [Restore a Platform Deployment with an External Database](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/restore.md) - [Troubleshooting External Database](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/troubleshoot.md) - [Upgrade Platform with an External Database](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/upgrade.md) - [Ingress Suffix](https://vcluster.com/docs/platform/administer/clusters/advanced/ingress-suffix.md) - [Deploy Multi-Region Platform on AWS (EKS)](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/deploy.md) - [Overview](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md) - [Recover from Regional Failover](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/recover.md) - [Restore the Kine Database from a Snapshot](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/restore.md) - [Upgrade a Multi-Region Platform Deployment](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/upgrade.md) - [Networking](https://vcluster.com/docs/platform/administer/clusters/advanced/networking.md) - [Enforcing Policies](https://vcluster.com/docs/platform/administer/clusters/advanced/policies.md) - [Regional Cluster Endpoints](https://vcluster.com/docs/platform/administer/clusters/advanced/regional-cluster-endpoints.md) - [Register clusters with Terraform](https://vcluster.com/docs/platform/administer/clusters/advanced/terraform-registration.md) - [Cluster Roles](https://vcluster.com/docs/platform/administer/clusters/cluster-roles.md) - [Connect a Cluster](https://vcluster.com/docs/platform/administer/clusters/connect-cluster.md) - [Database Connector](https://vcluster.com/docs/platform/administer/connector/database.md) - [IAM Database Connector](https://vcluster.com/docs/platform/administer/connector/iam_database.md) - [Inbox](https://vcluster.com/docs/platform/administer/inbox.md) - [Overview](https://vcluster.com/docs/platform/administer/node-profiles/overview.md) - [Nvidia Base Command Manager](https://vcluster.com/docs/platform/administer/node-providers/bcm.md) - [KubeVirt](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md) - [Metal3](https://vcluster.com/docs/platform/administer/node-providers/metal3.md) - [Metal3 Network Integration](https://vcluster.com/docs/platform/administer/node-providers/metal3-network-integration.md) - [Overview](https://vcluster.com/docs/platform/administer/node-providers/overview.md) - [Terraform](https://vcluster.com/docs/platform/administer/node-providers/terraform.md) - [Manage Allowed Clusters](https://vcluster.com/docs/platform/administer/projects/allowed/clusters.md) - [Manage Node Profiles](https://vcluster.com/docs/platform/administer/projects/allowed/node-profiles.md) - [Manage Allowed Node Types](https://vcluster.com/docs/platform/administer/projects/allowed/node-types.md) - [Manage Allowed Templates](https://vcluster.com/docs/platform/administer/projects/allowed/templates.md) - [Create a Project](https://vcluster.com/docs/platform/administer/projects/create.md) - [Manage Quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md) - [Create global secrets](https://vcluster.com/docs/platform/administer/secrets/global/create.md) - [Delete global secrets](https://vcluster.com/docs/platform/administer/secrets/global/delete.md) - [Create project secrets](https://vcluster.com/docs/platform/administer/secrets/project/create.md) - [Delete project secrets](https://vcluster.com/docs/platform/administer/secrets/project/delete.md) - [Retrieve project secrets](https://vcluster.com/docs/platform/administer/secrets/project/retrieve.md) - [Manage Certified Stacks](https://vcluster.com/docs/platform/administer/templates/certified-stacks.md) - [Create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) - [Create a template](https://vcluster.com/docs/platform/administer/templates/create-templates.md) - [Private Helm Repositories for Apps](https://vcluster.com/docs/platform/administer/templates/external-credentials.md) - [Templates overview](https://vcluster.com/docs/platform/administer/templates/overview.md) - [Parameters](https://vcluster.com/docs/platform/administer/templates/parameters.md) - [Versioning Templates](https://vcluster.com/docs/platform/administer/templates/versioning.md) - [Create teams in the platform](https://vcluster.com/docs/platform/administer/users-permissions/managing-teams/create.md) - [Create users in the platform](https://vcluster.com/docs/platform/administer/users-permissions/managing-users/create.md) - [Impersonate](https://vcluster.com/docs/platform/administer/users-permissions/managing-users/impersonate.md) - [Reset a user's password](https://vcluster.com/docs/platform/administer/users-permissions/managing-users/reset-password.md) - [Suspend](https://vcluster.com/docs/platform/administer/users-permissions/managing-users/suspend.md) - [Overview](https://vcluster.com/docs/platform/administer/users-permissions/overview.md) - [Cluster Access](https://vcluster.com/docs/platform/administer/users-permissions/permissions/cluster.md) - [Manage Access for a Namespace](https://vcluster.com/docs/platform/administer/users-permissions/permissions/namespace.md) - [Project Membership](https://vcluster.com/docs/platform/administer/users-permissions/permissions/project.md) - [Stack Permissions](https://vcluster.com/docs/platform/administer/users-permissions/permissions/stacks.md) - [Management access to the tenant cluster](https://vcluster.com/docs/platform/administer/users-permissions/permissions/vcluster.md) - [Data volumes](https://vcluster.com/docs/platform/administer/virtual-machines/data-volumes.md) - [Network attachment definitions](https://vcluster.com/docs/platform/administer/virtual-machines/network-attachment-definitions.md) - [Overview](https://vcluster.com/docs/platform/administer/virtual-machines/overview.md) - [App Instance](https://vcluster.com/docs/platform/api/resources/appinstance.md) - [Retrieve App Instance Logs](https://vcluster.com/docs/platform/api/resources/appinstancelog.md) - [App](https://vcluster.com/docs/platform/api/resources/apps.md) - [ArgoCD Application](https://vcluster.com/docs/platform/api/resources/argocdapplication.md) - [ArgoCD Application Template](https://vcluster.com/docs/platform/api/resources/argocdapplicationtemplate.md) - [Cluster Access](https://vcluster.com/docs/platform/api/resources/clusteraccess.md) - [Cluster Role Template](https://vcluster.com/docs/platform/api/resources/clusterroletemplate.md) - [Cluster](https://vcluster.com/docs/platform/api/resources/clusters.md) - [Loft Config](https://vcluster.com/docs/platform/api/resources/config.md) - [Direct Cluster Endpoint Token](https://vcluster.com/docs/platform/api/resources/directclusterendpointtoken.md) - [Machine Config Template](https://vcluster.com/docs/platform/api/resources/machineconfigtemplate.md) - [Node Claim](https://vcluster.com/docs/platform/api/resources/nodeclaim.md) - [Node Profile](https://vcluster.com/docs/platform/api/resources/nodeprofile.md) - [Node Provider](https://vcluster.com/docs/platform/api/resources/nodeprovider.md) - [OS Image](https://vcluster.com/docs/platform/api/resources/osimage.md) - [Owned Access Key](https://vcluster.com/docs/platform/api/resources/ownedaccesskey.md) - [Project](https://vcluster.com/docs/platform/api/resources/project.md) - [Retrieve Project Clusters](https://vcluster.com/docs/platform/api/resources/project/clusters.md) - [Import Namespace](https://vcluster.com/docs/platform/api/resources/project/importspace.md) - [Retrieve Project Members](https://vcluster.com/docs/platform/api/resources/project/members.md) - [Move Space To Other Project](https://vcluster.com/docs/platform/api/resources/project/migratespaceinstance.md) - [Move vCluster To Other Project](https://vcluster.com/docs/platform/api/resources/project/migratevirtualclusterinstance.md) - [Project Secret](https://vcluster.com/docs/platform/api/resources/projectsecret.md) - [Self](https://vcluster.com/docs/platform/api/resources/self.md) - [Global Secret](https://vcluster.com/docs/platform/api/resources/sharedsecret.md) - [Space Instance](https://vcluster.com/docs/platform/api/resources/spaceinstance.md) - [Space Template](https://vcluster.com/docs/platform/api/resources/spacetemplate.md) - [SSH Key](https://vcluster.com/docs/platform/api/resources/sshkey.md) - [Stack Instance](https://vcluster.com/docs/platform/api/resources/stackinstance.md) - [Retrieve Stack Instance Outputs](https://vcluster.com/docs/platform/api/resources/stackinstanceoutputs.md) - [Stack Template](https://vcluster.com/docs/platform/api/resources/stacktemplate.md) - [Team](https://vcluster.com/docs/platform/api/resources/team.md) - [User](https://vcluster.com/docs/platform/api/resources/user.md) - [Virtual Cluster Instance](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance.md) - [Retrieve Kube Config (Ingress)](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance/kubeconfig.md) - [Retrieve Resource Usage](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance/resourceusage.md) - [Virtual Cluster Template](https://vcluster.com/docs/platform/api/resources/virtualclustertemplate.md) - [Use API](https://vcluster.com/docs/platform/api/use-api.md) - [Configure CORS policy](https://vcluster.com/docs/platform/configure/advanced/cors-policy.md) - [Agent upgrades](https://vcluster.com/docs/platform/configure/agent-settings/agent-upgrade.md) - [Per-control plane cluster customization](https://vcluster.com/docs/platform/configure/agent-settings/customization.md) - [Least Privilege Mode](https://vcluster.com/docs/platform/configure/agent-settings/least-privilege-mode.md) - [Overview](https://vcluster.com/docs/platform/configure/agent-settings/overview.md) - [Configure security context](https://vcluster.com/docs/platform/configure/agent-settings/security-context.md) - [Configure External Access & TLS](https://vcluster.com/docs/platform/configure/installation-options/domain.md) - [Overview](https://vcluster.com/docs/platform/configure/installation-options/overview.md) - [Introduction](https://vcluster.com/docs/platform/configure/introduction.md) - [Audit Logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md) - [Cost Control Dashboard](https://vcluster.com/docs/platform/configure/platform-configs/cost-control.md) - [Custom branding](https://vcluster.com/docs/platform/configure/platform-configs/custom-branding.md) - [Overview](https://vcluster.com/docs/platform/configure/platform-configs/overview.md) - [SSO Providers](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md) - [Disable the local admin account after SSO setup](https://vcluster.com/docs/platform/how-to/disable-local-admin.md) - [Enforce storage quotas per tenant cluster](https://vcluster.com/docs/platform/how-to/enforce-storage-quotas.md) - [Expose the platform with Istio](https://vcluster.com/docs/platform/how-to/exposing-vCluster-platform-with-Istio.md) - [End-to-end GitOps with ArgoCD](https://vcluster.com/docs/platform/how-to/gitops-end-to-end.md) - [Build a product control plane on the Platform API](https://vcluster.com/docs/platform/how-to/product-control-plane.md) - [Use Go Template Functions](https://vcluster.com/docs/platform/how-to/use-go-template.md) - [With Offline License Key](https://vcluster.com/docs/platform/install/air-gapped.md) - [Platform Database](https://vcluster.com/docs/platform/install/air-gapped/platform-database.md) - [With Offline License Server](https://vcluster.com/docs/platform/install/air-gapped/with-offline-license-server.md) - [Install vCluster Platform on Amazon EKS](https://vcluster.com/docs/platform/install/environments/aws.md) - [Install vCluster Platform on Azure using AKS](https://vcluster.com/docs/platform/install/environments/azure.md) - [Install vCluster Platform on Google Cloud using GKE](https://vcluster.com/docs/platform/install/environments/gcp.md) - [Install with ArgoCD](https://vcluster.com/docs/platform/install/gitops.md) - [Install with Helm](https://vcluster.com/docs/platform/install/helm.md) - [Install with CLI](https://vcluster.com/docs/platform/install/quick-start-guide.md) - [Argo CD](https://vcluster.com/docs/platform/integrations/argocd.md) - [Connect to Akuity](https://vcluster.com/docs/platform/integrations/argocd/connect-akuity.md) - [Connect to Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md) - [Feature support and limitations](https://vcluster.com/docs/platform/integrations/argocd/considerations.md) - [Deploy applications](https://vcluster.com/docs/platform/integrations/argocd/deploy-applications.md) - [Argo CD (legacy)](https://vcluster.com/docs/platform/integrations/argocd/legacy.md) - [Certified Stacks](https://vcluster.com/docs/platform/integrations/certified-stacks.md) - [NVIDIA Run:ai](https://vcluster.com/docs/platform/integrations/certified-stacks/runai.md) - [GitLab CI](https://vcluster.com/docs/platform/integrations/gitlab-ci.md) - [HashiCorp Vault Secret Sync](https://vcluster.com/docs/platform/integrations/hashicorp-vault.md) - [Using Karpenter with vCluster Platform](https://vcluster.com/docs/platform/integrations/karpenter.md) - [Migrate to the vCluster Rancher operator](https://vcluster.com/docs/platform/integrations/rancher/migration-guide.md) - [Upgrade from legacy Rancher integration](https://vcluster.com/docs/platform/integrations/rancher/upgrade-guide.md) - [Architecture](https://vcluster.com/docs/platform/introduction/architecture.md) - [How Platform works](https://vcluster.com/docs/platform/introduction/how-platform-works.md) - [What is vCluster?](https://vcluster.com/docs/platform/introduction/vcluster_intro.md) - [Backup and restore vCluster Platform](https://vcluster.com/docs/platform/maintenance/backup-restore/backup-restore-platform.md) - [Restore license](https://vcluster.com/docs/platform/maintenance/backup-restore/preserve-license.md) - [Central HostPath Mapper](https://vcluster.com/docs/platform/maintenance/logging/central-hostpath-mapper.md) - [Logging](https://vcluster.com/docs/platform/maintenance/logging/overview.md) - [Platform Process Logging](https://vcluster.com/docs/platform/maintenance/logging/platform-logging.md) - [Aggregating Metrics](https://vcluster.com/docs/platform/maintenance/monitoring/aggregating-metrics.md) - [Fleet metrics architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md) - [Fleet monitoring with OpenTelemetry](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-monitoring-otel.md) - [Metrics](https://vcluster.com/docs/platform/maintenance/monitoring/metrics.md) - [Monitoring](https://vcluster.com/docs/platform/maintenance/monitoring/overview.md) - [Prometheus node metrics on shared nodes](https://vcluster.com/docs/platform/maintenance/monitoring/prometheus-shared-nodes.md) - [Configure edge collectors](https://vcluster.com/docs/platform/maintenance/observability/configure-edge-collectors.md) - [Configure metrics access](https://vcluster.com/docs/platform/maintenance/observability/configure-metrics-access.md) - [Deploy GPU observability with Argo CD](https://vcluster.com/docs/platform/maintenance/observability/gpu-observability-templates.md) - [Install the observability gateway](https://vcluster.com/docs/platform/maintenance/observability/install-observability-gateway.md) - [Fleet Observability labels reference](https://vcluster.com/docs/platform/maintenance/observability/labels-reference.md) - [Observe GPU nodes with NVSentinel](https://vcluster.com/docs/platform/maintenance/observability/nvsentinel-gpu-observability.md) - [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md) - [Query fleet metrics](https://vcluster.com/docs/platform/maintenance/observability/query-fleet-metrics.md) - [Fleet Observability quickstart](https://vcluster.com/docs/platform/maintenance/observability/quickstart.md) - [Troubleshooting Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/troubleshooting.md) - [High Availability Installation](https://vcluster.com/docs/platform/maintenance/reliability-scaling/high-availability.md) - [Uninstall](https://vcluster.com/docs/platform/maintenance/uninstall.md) - [Lifecycle policy](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/lifecycle-policy.md) - [Migrate the platform to a different cluster](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/migrate-to-hostcluster.md) - [Upgrade vCluster Platform](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade.md) - [Upgrade in an air-gapped environment](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade-air-gapped.md) - [Upgrade from v3 to v4](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade-v3-to-v4.md) - [Platform glossary](https://vcluster.com/docs/platform/reference/glossary.md) - [Platform annotations and labels reference](https://vcluster.com/docs/platform/reference/platform-annotations.md) - [Platform CLI hub](https://vcluster.com/docs/platform/reference/platform-cli.md) - [Search vCluster docs with Algolia MCP](https://vcluster.com/docs/platform/resources/algolia-mcp.md) - [Contribute to Platform Docs](https://vcluster.com/docs/platform/resources/contribute.md) - [Troubleshoot connected cluster connection failures](https://vcluster.com/docs/platform/troubleshoot/connected-cluster-connectivity.md) - [Troubleshoot Platform Database](https://vcluster.com/docs/platform/troubleshoot/platform-database.md) - [Troubleshoot quota and admission denials](https://vcluster.com/docs/platform/troubleshoot/quota-admission.md) - [Troubleshoot Stacks](https://vcluster.com/docs/platform/troubleshoot/stacks.md) - [Troubleshoot template and app sync failures](https://vcluster.com/docs/platform/troubleshoot/template-sync-failures.md) - [Resolve vCluster TSNet connection failures](https://vcluster.com/docs/platform/troubleshoot/tsnet-connectivity.md) - [Authentication and authorization](https://vcluster.com/docs/platform/understand/auth-explanation.md) - [What happens during control plane outages?](https://vcluster.com/docs/platform/understand/control-plane-outages.md) - [Platform-Managed vs. Externally Deployed Tenant Clusters](https://vcluster.com/docs/platform/understand/externally-vs-platform-deployed.md) - [How licensing works](https://vcluster.com/docs/platform/understand/licensing.md) - [Platform communication and connectivity](https://vcluster.com/docs/platform/understand/platform_communication.md) - [What are numeric input ranges in app templates?](https://vcluster.com/docs/platform/understand/ranges-app-template.md) - [What are Apps](https://vcluster.com/docs/platform/understand/what-are-apps.md) - [What are Connected Clusters](https://vcluster.com/docs/platform/understand/what-are-clusters.md) - [What Are Namespaces](https://vcluster.com/docs/platform/understand/what-are-namespaces.md) - [What are Projects](https://vcluster.com/docs/platform/understand/what-are-projects.md) - [What are Secrets](https://vcluster.com/docs/platform/understand/what-are-secrets.md) - [What are Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md) - [What are Templates](https://vcluster.com/docs/platform/understand/what-are-templates.md) - [What are Users and Teams](https://vcluster.com/docs/platform/understand/what-are-users-and-teams.md) - [View resources across projects](https://vcluster.com/docs/platform/use-platform/all-projects.md) - [Updating Apps](https://vcluster.com/docs/platform/use-platform/apps/updating.md) - [Use In Templates](https://vcluster.com/docs/platform/use-platform/apps/use-in-templates.md) - [Use On-Demand](https://vcluster.com/docs/platform/use-platform/apps/use-on-demand.md) - [Use Parameters with Apps](https://vcluster.com/docs/platform/use-platform/apps/use-parameters.md) - [Use a Stack](https://vcluster.com/docs/platform/use-platform/apps/use-stacks.md) - [Change your password](https://vcluster.com/docs/platform/use-platform/change-your-password.md) - [Machine Config Templates](https://vcluster.com/docs/platform/use-platform/machines/machine-config-templates.md) - [OS Images](https://vcluster.com/docs/platform/use-platform/machines/os-images.md) - [Overview](https://vcluster.com/docs/platform/use-platform/machines/overview.md) - [SSH Keys](https://vcluster.com/docs/platform/use-platform/machines/ssh-keys.md) - [Create Namespace Manually](https://vcluster.com/docs/platform/use-platform/namespaces/create/create-no-template.md) - [Create Namespace from Templates](https://vcluster.com/docs/platform/use-platform/namespaces/create/create-with-template.md) - [Custom links](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/custom-links.md) - [Isolation](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/isolation.md) - [Prevent Deletion](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/prevent-deletion.md) - [Manage namespace sleep and auto-delete](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/sleep-mode.md) - [Get kubeconfig File](https://vcluster.com/docs/platform/use-platform/namespaces/retrieve-kube-config.md) - [Sync Secrets](https://vcluster.com/docs/platform/use-platform/secrets/secret-sync.md) - [Troubleshooting](https://vcluster.com/docs/platform/use-platform/troubleshooting.md) - [Add existing tenant clusters](https://vcluster.com/docs/platform/use-platform/virtual-clusters/add-virtual-clusters.md) - [Manually configure a tenant cluster](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-no-template.md) - [Create a tenant cluster from a template](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-with-template.md) - [Custom Links](https://vcluster.com/docs/platform/use-platform/virtual-clusters/key-features/custom-links.md) - [Kubectl shell](https://vcluster.com/docs/platform/use-platform/virtual-clusters/key-features/kubectl-shell.md) - [Prevent Deletion](https://vcluster.com/docs/platform/use-platform/virtual-clusters/key-features/prevent-deletion.md) - [Manage sleep and auto-delete](https://vcluster.com/docs/platform/use-platform/virtual-clusters/key-features/sleep-mode.md) - [Grant access to a tenant cluster](https://vcluster.com/docs/platform/use-platform/virtual-clusters/manage-access.md) - [Get a kubeconfig file for your tenant cluster](https://vcluster.com/docs/platform/use-platform/virtual-clusters/retrieve-kube-config.md) ### vcluster - [Start Here](https://vcluster.com/docs/vcluster.md) - [Control Plane](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane.md) - [Backing Store](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store.md) - [Database](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database.md) - [External database](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md) - [Embedded SQLite](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/sqlite.md) - [etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd.md) - [Deployed etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md) - [Embedded etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/embedded.md) - [CoreDNS](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/coredns.md) - [Kubernetes distro](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/distro.md) - [Kubernetes distribution](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/distro/k8s.md) - [Host path mapper](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/host-path-mapper.md) - [Proxy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/proxy.md) - [Standalone](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/standalone.md) - [Ingress](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/deployment/ingress.md) - [Service](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/deployment/service.md) - [Service monitor](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/deployment/service-monitor.md) - [StatefulSet](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/deployment/statefulset.md) - [Advanced](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced.md) - [Default image registry](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/default-image-registry.md) - [Global metadata](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/global-metadata.md) - [Headless service](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/headless-service.md) - [Kube-vip](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/kube-vip.md) - [Service account](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/service-account.md) - [Virtual scheduler](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/virtual-scheduler.md) - [Workload service account](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/workload-service-account.md) - [Configure automatic deletion](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/deletion.md) - [Deploy vCluster AddOns](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/deploy.md) - [Experimental](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental.md) - [Denying proxy requests](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/deny-proxy-requests.md) - [Experimental deploy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/deploy.md) - [vCluster in Docker (vind)](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/docker.md) - [Resource Proxy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/proxy.md) - [Experimental sync settings](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/sync-settings.md) - [Export kubeconfig](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/export-kube-config.md) - [Integrations](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations.md) - [cert-manager](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/cert-manager.md) - [External secrets](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/eso.md) - [Istio](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/istio.md) - [KubeVirt](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/kubeVirt.md) - [Metrics server](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/metricsServer.md) - [Netris](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/netris.md) - [Logging](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/logging.md) - [Networking](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking.md) - [Advanced networking](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking/advanced.md) - [Replicate networking services](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking/replicate-services.md) - [Resolve DNS](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking/resolve-dns.md) - [Platform](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/platform.md) - [Plugins](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/plugins.md) - [Policies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies.md) - [Central admission control](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/admission-control.md) - [Limit range](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/limit-range.md) - [Network policy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md) - [Pod security standard](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/pod-security-standard.md) - [Resource quota](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/resource-quota.md) - [Private Nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes.md) - [Auto Nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/auto-nodes.md) - [vCluster VPN](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/vpn.md) - [RBAC](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/rbac.md) - [Configure auto sleep](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md) - [Snapshots](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/snapshots.md) - [Sync from host to virtual](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host.md) - [ConfigMaps](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/configmaps.md) - [CSI drivers](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-drivers.md) - [CSI nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-nodes.md) - [CSI storage capacities](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-storage-capacities.md) - [Custom resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/custom-resources.md) - [Device classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/device-classes.md) - [Events](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/events.md) - [Gateway classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/gateway-classes.md) - [Gateways and GatewayClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/gateways.md) - [Ingress classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/ingress-classes.md) - [Nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md) - [Priority classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/priority-classes.md) - [Runtime classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/runtime-classes.md) - [Secrets](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/secrets.md) - [Storage classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/storage-classes.md) - [Volume snapshot classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/volume-snapshot-classes.md) - [Patching](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) - [Sync to host from virtual](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host.md) - [Custom resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/custom-resources.md) - [Namespaces](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/namespaces.md) - [Pod disruption budgets](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/pod-disruption-budgets.md) - [Priority classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/priority-classes.md) - [Service accounts](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/service-accounts.md) - [Config maps](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/config-maps.md) - [Pods](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/pods.md) - [Hybrid scheduling](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/pods/hybrid-scheduling.md) - [Secrets](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/secrets.md) - [ResourceClaims](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaims.md) - [ResourceClaimTemplates](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaimtemplates.md) - [Endpoints](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/endpoints.md) - [EndpointSlices](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/endpointslices.md) - [Gateway API](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md) - [Ingresses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/ingresses.md) - [Network policies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/network-policies.md) - [Services](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/services.md) - [Persistent volume claims](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/persistent-volume-claims.md) - [Persistent volumes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/persistent-volumes.md) - [Storage Classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/storage-classes.md) - [Volume snapshots](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/volume-snapshots.md) - [Telemetry](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/telemetry.md) - [What is vcluster.yaml?](https://vcluster.com/docs/vcluster/configure/what-is-vcluster-yaml.md) - [Deploy vCluster](https://vcluster.com/docs/vcluster/deploy.md) - [Control plane benchmark methodology](https://vcluster.com/docs/vcluster/deploy/control-plane/benchmark-methodology.md) - [vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md) - [Create control plane](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) - [Deploy in high availability](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md) - [Manage vCluster Standalone Control Plane Nodes](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/manage.md) - [Node Requirements](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/node-requirements.md) - [Deploy vCluster in an air-gapped environment](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/security/air-gapped.md) - [FIPS 140-2 enablement](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/security/fips.md) - [Troubleshooting vCluster Standalone Control Plane Nodes](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/troubleshooting.md) - [Scale CoreDNS for large node counts](https://vcluster.com/docs/vcluster/deploy/control-plane/coredns-scaling.md) - [Deploy vind with restricted network access](https://vcluster.com/docs/vcluster/deploy/control-plane/docker-container/air-gapped.md) - [Deployment basics](https://vcluster.com/docs/vcluster/deploy/control-plane/docker-container/basics.md) - [Deployment basics](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/basics.md) - [Deploy vCluster on AKS](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/aks.md) - [Isolate clusters with Netris and NVIDIA BCM](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/bare-metal.md) - [Deploy vCluster on EKS](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/eks.md) - [Deploy vCluster on GKE](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/gke.md) - [Deploy on OpenShift](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/openshift.md) - [Deploy in high availability](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/high-availability.md) - [Deploy vCluster in an air-gapped environment](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/air-gapped.md) - [Deploy vCluster behind a corporate proxy](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/corporate-proxy.md) - [FIPS 140-2 enablement](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/fips.md) - [Control plane configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/control-plane.md) - [Control plane security configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/control-plane-components.md) - [etcd configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/etcd.md) - [Kubernetes policies](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/policies.md) - [Self-assessment guide for host nodes](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/self-assessment.md) - [Worker node configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/worker-node.md) - [CIS hardening overview](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/overview.md) - [Control plane configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/private-nodes/control-plane.md) - [Control plane security configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/private-nodes/control-plane-components.md) - [etcd configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/private-nodes/etcd.md) - [Kubernetes policies](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/private-nodes/policies.md) - [Self-assessment guide for private nodes](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/private-nodes/self-assessment.md) - [Worker node configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/private-nodes/worker-node.md) - [Deploy in rootless mode](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/rootless-mode.md) - [Control plane sizing and performance](https://vcluster.com/docs/vcluster/deploy/control-plane/sizing-and-performance.md) - [Nodes from the Control Plane Cluster](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes.md) - [Deploy with isolated workloads](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes/isolated-workloads.md) - [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) - [Automatically Provision and Join Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/auto-nodes.md) - [Deploy auto nodes with quick start templates](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/auto-nodes/quick-start-templates.md) - [Join Manually Provisioned Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md) - [kube-ovn integration](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/kube-ovn.md) - [Manage Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/manage.md) - [Node Requirements](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/node-requirements.md) - [Deploy Private Nodes in an air-gapped environment](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/air-gapped.md) - [Confidential compute](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/confidential-compute.md) - [FIPS 140-2 enablement](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/fips.md) - [SELinux support](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/selinux.md) - [vCluster VPN](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/vpn.md) - [cert-manager](https://vcluster.com/docs/vcluster/integrations/certManager.md) - [Custom Plugins](https://vcluster.com/docs/vcluster/integrations/custom-plugins.md) - [External Secrets Operator](https://vcluster.com/docs/vcluster/integrations/external-secrets-operator.md) - [GPU and inference autoscaling](https://vcluster.com/docs/vcluster/integrations/gpu-hpa-dcgm.md) - [Istio](https://vcluster.com/docs/vcluster/integrations/istio.md) - [KubeVirt](https://vcluster.com/docs/vcluster/integrations/kubevirt.md) - [Metrics server](https://vcluster.com/docs/vcluster/integrations/metrics-server.md) - [Inference serving stacks](https://vcluster.com/docs/vcluster/integrations/model-serving.md) - [CI testing with setup-vind](https://vcluster.com/docs/vcluster/integrations/setup-vind.md) - [Architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md) - [GPU and accelerator support](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md) - [Building a GPU cloud platform](https://vcluster.com/docs/vcluster/introduction/gpu-cloud-platform.md) - [Building an inference platform](https://vcluster.com/docs/vcluster/introduction/inference-platform.md) - [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md) - [What is vCluster?](https://vcluster.com/docs/vcluster/introduction/what-is-vcluster.md) - [Custom Links](https://vcluster.com/docs/vcluster/key-features/custom-links.md) - [Gateway API](https://vcluster.com/docs/vcluster/key-features/gateway-api.md) - [Ingress Access](https://vcluster.com/docs/vcluster/key-features/ingress-access.md) - [Isolation](https://vcluster.com/docs/vcluster/key-features/isolation.md) - [Kubectl shell](https://vcluster.com/docs/vcluster/key-features/kubectl-shell.md) - [Sleep tenant clusters](https://vcluster.com/docs/vcluster/key-features/sleep.md) - [Snapshots](https://vcluster.com/docs/vcluster/key-features/snapshots.md) - [Deploy vCluster in an air-gapped environment using a private web server](https://vcluster.com/docs/vcluster/learn-how-to/airgapped-vcluster-deployment.md) - [Use vCluster CLI as a kubectl plugin](https://vcluster.com/docs/vcluster/learn-how-to/cli-kubectl.md) - [Capture pprof for vCluster](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/capture-pprof-vcluster.md) - [Check connection to a vCluster or control plane cluster](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/check-vcluster.md) - [Configure custom CoreDNS](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/configure-custom-coredns.md) - [Configure CPU and memory limits for vCluster](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/configure-memory-limits.md) - [Configure webhook token authentication](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/configure-webhook-token-auth.md) - [Enable debug logging in vCluster](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/enable-debug-logging.md) - [Override the Alpine sidecar image in vCluster](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/override-alpine-sidecar.md) - [Retrieve the deployed vCluster configuration](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/retrieve-vcluster-config.md) - [Safely upgrade ETCD v3.5 -> v3.6](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/safely-upgrade-etcd.md) - [Use custom schedulers with Hybrid scheduling](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/use-custom-schedulers.md) - [Use a control plane cluster registry from a tenant cluster](https://vcluster.com/docs/vcluster/learn-how-to/in-cluster-registry.md) - [Set up the resource proxy](https://vcluster.com/docs/vcluster/learn-how-to/resource-proxy-setup.md) - [Sync Calico network policies to the control plane cluster](https://vcluster.com/docs/vcluster/learn-how-to/sync-calico-network-policies.md) - [Sync a namespaced custom resource from the control plane cluster](https://vcluster.com/docs/vcluster/learn-how-to/sync-custom-resource-from-host.md) - [Control node IP visibility](https://vcluster.com/docs/vcluster/learn-how-to/worker-nodes/host-nodes/control-node-ip-visibility.md) - [Access and expose vCluster](https://vcluster.com/docs/vcluster/manage/accessing-vcluster.md) - [Create snapshots](https://vcluster.com/docs/vcluster/manage/backup-restore/backup.md) - [Restore snapshots](https://vcluster.com/docs/vcluster/manage/backup-restore/restore.md) - [Using Velero](https://vcluster.com/docs/vcluster/manage/backup-restore/velero.md) - [Rotate certificates](https://vcluster.com/docs/vcluster/manage/certificates.md) - [Manage custom resources](https://vcluster.com/docs/vcluster/manage/custom-resources.md) - [Deploy configuration changes](https://vcluster.com/docs/vcluster/manage/deploy-changes.md) - [Migrate from ingress-nginx to Gateway API](https://vcluster.com/docs/vcluster/manage/ingress-nginx-to-gateway-api.md) - [Logging](https://vcluster.com/docs/vcluster/manage/logging.md) - [Migrate etcd backing store](https://vcluster.com/docs/vcluster/manage/migrate-etcd-backing-store.md) - [Operate the resource proxy](https://vcluster.com/docs/vcluster/manage/resource-proxy-operations.md) - [Pause and resume a tenant cluster](https://vcluster.com/docs/vcluster/manage/sleep-wakeup.md) - [Migrate from K3s to Kubernetes](https://vcluster.com/docs/vcluster/manage/upgrade/distro-migration.md) - [Lifecycle policy](https://vcluster.com/docs/vcluster/manage/upgrade/supported_versions.md) - [Upgrade vCluster](https://vcluster.com/docs/vcluster/manage/upgrade/upgrade-version.md) - [AI Cloud: Managed Kubernetes Service](https://vcluster.com/docs/vcluster/production-guide/ai-cloud.md) - [Choose a worker node model](https://vcluster.com/docs/vcluster/production-guide/choose-worker-node-model.md) - [CI/CD Platform](https://vcluster.com/docs/vcluster/production-guide/cicd-platform.md) - [Distributed Compute Aggregation](https://vcluster.com/docs/vcluster/production-guide/distributed-compute.md) - [Edge Distribution](https://vcluster.com/docs/vcluster/production-guide/edge-distribution.md) - [Enterprise AI Factory](https://vcluster.com/docs/vcluster/production-guide/enterprise-ai-factory.md) - [Inference Provider: Managed Model Serving](https://vcluster.com/docs/vcluster/production-guide/inference-provider.md) - [Internal Kubernetes Platform](https://vcluster.com/docs/vcluster/production-guide/internal-k8s-platform.md) - [Build for Production](https://vcluster.com/docs/vcluster/production-guide/overview.md) - [Single-Tenant Per Customer](https://vcluster.com/docs/vcluster/production-guide/single-tenant.md) - [Docker (vind) Quick Start](https://vcluster.com/docs/vcluster/quick-start/docker.md) - [Private Nodes Quick Start](https://vcluster.com/docs/vcluster/quick-start/private-nodes.md) - [Shared Nodes Quick Start](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md) - [Standalone Quick Start](https://vcluster.com/docs/vcluster/quick-start/standalone.md) - [Annotations and labels reference](https://vcluster.com/docs/vcluster/reference/annotations.md) - [vCluster glossary](https://vcluster.com/docs/vcluster/reference/glossary.md) - [v0.20 CLI Updates](https://vcluster.com/docs/vcluster/reference/migrations/0-20-cli-changes.md) - [Conversion Guide from pre-v0.20 to v0.20](https://vcluster.com/docs/vcluster/reference/migrations/0-20-migration.md) - [Loft CLI to vCluster CLI Migration](https://vcluster.com/docs/vcluster/reference/migrations/loft-cli-vcluster-cli-migration.md) - [Search vCluster docs with Algolia MCP](https://vcluster.com/docs/vcluster/resources/algolia-mcp.md) - [Contribute to vCluster Docs](https://vcluster.com/docs/vcluster/resources/contribute.md) - [Secure vCluster Deployments](https://vcluster.com/docs/vcluster/security.md) - [Shared-node admission policy examples](https://vcluster.com/docs/vcluster/security/shared-nodes-admission-examples.md) - [Shared-node security hardening](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) - [Deploy with Flux](https://vcluster.com/docs/vcluster/third-party-integrations/git-operations/flux.md) - [Preview environments](https://vcluster.com/docs/vcluster/third-party-integrations/github-actions/preview-environments.md) - [GitHub actions on pull requests](https://vcluster.com/docs/vcluster/third-party-integrations/github-actions/pull-requests.md) - [Netris](https://vcluster.com/docs/vcluster/third-party-integrations/netris.md) - [AKS Workload Identity](https://vcluster.com/docs/vcluster/third-party-integrations/pod-identity/aks-workload-identity.md) - [Integrate EKS Pod Identity with vCluster](https://vcluster.com/docs/vcluster/third-party-integrations/pod-identity/eks-pod-identity.md) - [Integrating GCP Workload Identity with vCluster](https://vcluster.com/docs/vcluster/third-party-integrations/pod-identity/gke-workload-identity.md) - [Install the Rancher integration in vCluster](https://vcluster.com/docs/vcluster/third-party-integrations/rancher.md) - [Configure NVIDIA KAI scheduler with vCluster](https://vcluster.com/docs/vcluster/third-party-integrations/scheduler/kai-scheduler.md) - [Clean obsolete nodes after changing the node selector](https://vcluster.com/docs/vcluster/troubleshoot/clean-up-orphaned-synced-nodes.md) - [Resolve CoreDNS DNS timeouts and SERVFAIL errors](https://vcluster.com/docs/vcluster/troubleshoot/coredns-dns-timeouts.md) - [Resolve etcd NOSPACE alarm in vCluster](https://vcluster.com/docs/vcluster/troubleshoot/etcd-compaction.md) - [Resolve Gateway API sync errors](https://vcluster.com/docs/vcluster/troubleshoot/gateway-api-sync.md) - [Resolve kubectl port-forward issues with Ingress](https://vcluster.com/docs/vcluster/troubleshoot/kubectl-port-forward-ingress-issues.md) - [Resolve permission denied errors with noexec volumes](https://vcluster.com/docs/vcluster/troubleshoot/noexec-emptydir-volumes.md) - [Resolve pods stuck at Ready=False with control plane and tenant version skew](https://vcluster.com/docs/vcluster/troubleshoot/pod-stuck-ready-false-version-skew.md) - [Resolve Pro feature license errors](https://vcluster.com/docs/vcluster/troubleshoot/pro-feature-license-error.md) - [Resolve ingress update conflicts with Rancher](https://vcluster.com/docs/vcluster/troubleshoot/resolve-rancher-ingress.md) - [Troubleshoot the resource proxy](https://vcluster.com/docs/vcluster/troubleshoot/resource-proxy.md) - [What happens during control plane outages?](https://vcluster.com/docs/vcluster/understand/control-plane-outages.md) - [Why does CoreDNS become a DNS bottleneck?](https://vcluster.com/docs/vcluster/understand/coredns-bottlenecks.md) - [What is vCluster pod creation?](https://vcluster.com/docs/vcluster/understand/pod-creation.md) - [How the resource proxy works](https://vcluster.com/docs/vcluster/understand/resource-proxy.md) - [What is allocatable in vCluster virtual scheduler](https://vcluster.com/docs/vcluster/understand/virtual-scheduler-allocatable.md) ### how-the-products-fit-together - [How vCluster and vMetal fit together](https://vcluster.com/docs/how-the-products-fit-together.md) ### ips - [vCluster Labs egress IP addresses](https://vcluster.com/docs/ips.md) --- # Full Documentation Content [Skip to main content](#__docusaurus_skipToContent_fallback) [![vCluster](/docs/media/rebranding/vCluster_horizontal-orange.svg)![vCluster](/docs/media/rebranding/vCluster_horizontal-orange.svg)](https://vcluster.com/docs/.md) [****](https://vcluster.com/docs/.md)[vCluster](https://vcluster.com/docs/vcluster/.md)[vCluster Platform](https://vcluster.com/docs/platform/.md)[vNode](https://www.vnode.com/docs)[vMetal](https://www.vmetal.ai/docs) [Blog](https://www.vcluster.com/blog)[Changelog](https://www.vcluster.com/releases/en/changelog?hideLogo=true\&hideMenu=true\&theme=dark\&embed=true\&c=vCluster)[](https://slack.loft.sh/)[](https://github.com/loft-sh/vcluster-docs) Search # Search the documentation Search defaults to the current stable docs for each product. Use the version filters to broaden results or target a specific release. Type your search here vClusterCurrent stable (v0.37 Stable) (\_\_current\_\_) PlatformCurrent stable (v4.12 Stable) (\_\_current\_\_) [](https://www.algolia.com/)   * [Create New Doc](https://github.com/loft-sh/vcluster-docs/new/main/staging?filename=your-doc-name.mdx\&value=---%0Atitle%3A%20Your%20Document%20Title%0Asidebar_label%3A%20Short%20Label%0Adescription%3A%20Brief%20description%20of%20the%20document%0A---%0A%0AWrite%20your%20content%20here...%0A) Copyright © 2026[vCluster Labs](https://www.vcluster.com/)|Documentation released under[CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/). --- # What is vCluster Platform? vCluster Platform is the management plane for your tenant cluster fleet. It provides a web UI, CLI, and API for deploying, configuring, and operating tenant clusters across one or more Control Plane Clusters. Access control, lifecycle automation, resource governance, and node management are all built in. Without Platform, operating tenant clusters at scale requires custom tooling for provisioning workflows, RBAC, quota enforcement, and cluster lifecycle. Platform replaces that custom work with a production-ready management layer. ![Tenant cluster dashboard showing clusters across multiple projects with running, starting, sleeping, and terminating statuses](/docs/assets/images/tenant-clusters-dashboard-status-variety-b0c8bdf9df8e1dbcdc6fb4186efe6f8e.png) The tenant cluster dashboard gives operators one place to review cluster lifecycle state, project ownership, templates, sleep state, and control plane placement. ## How Platform fits into the picture[​](#how-platform-fits-into-the-picture "Direct link to How Platform fits into the picture") vCluster creates and runs tenant clusters. Platform manages the fleet. The relationship is straightforward: * **vCluster** handles the tenant cluster, including the API server, control plane, [syncer](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync), and node connectivity - **Platform** handles everything around the tenant cluster. It controls who can access it, how it's configured, when it sleeps, and when nodes are provisioned or deprovisioned Platform installs into a Kubernetes cluster and connects to other clusters in your environment. Those connected clusters become the Control Plane Clusters where tenant cluster control planes run. Platform doesn't replace those clusters. It manages what runs inside them. [Understand the Platform architecture →](https://vcluster.com/docs/platform/introduction/architecture.md) ## Projects[​](#projects "Direct link to Projects") Projects are the primary organizational unit in Platform. Each project is a policy boundary that groups tenant clusters and the users or teams who access them. Inside a project you can: * Set resource quotas that cap compute and storage across all tenant clusters in the project * Assign default and allowed templates so teams can only create clusters that meet your requirements * Grant users and teams access to just this project, with no visibility into other projects For AI cloud operators serving paying customers, projects map naturally to customer accounts or organizational boundaries. For internal platforms, projects map to teams, cost centers, or environments. ![Tenant clusters dashboard with the project filter open, showing multiple projects available for filtering](/docs/assets/images/tenant-cluster-management-dashboard-with-project-filter-1639a2b6413dc27eb0ad0f1c27c34f59.png) Projects scope the tenant clusters that users can view and manage. [Learn about projects →](https://vcluster.com/docs/platform/understand/what-are-projects.md) ## Tenant cluster management[​](#tenant-cluster-management "Direct link to Tenant cluster management") Platform provides a complete lifecycle interface for tenant clusters. Create, configure, upgrade, connect, and delete tenant clusters from the UI or through the Platform API. ### Templates[​](#templates "Direct link to Templates") Templates define a baseline `vcluster.yaml` configuration applied to every tenant cluster created from them. Administrators define the templates available in each project. Users choose from those templates at creation time and can optionally customize within allowed bounds. Templates solve the consistency problem at scale. Instead of relying on users to configure settings correctly, you encode requirements into a template and enforce them across the fleet. ![Private nodes template configuration screen showing config options, a vCluster version selector, and the vcluster.yaml editor](/docs/assets/images/private-nodes-template-configuration-interface-d4e4eabe8f12d321e1c433375b12f9d5.png) Templates combine guided configuration fields with the generated `vcluster.yaml` that Platform applies to new tenant clusters. [Create templates →](https://vcluster.com/docs/platform/administer/templates/create-templates.md) ### Applications and stacks[​](#applications-and-stacks "Direct link to Applications and stacks") Tenant cluster templates configure the Kubernetes environment itself. [Apps](https://vcluster.com/docs/platform/understand/what-are-apps.md) install individual applications into that environment. When an environment needs several Apps or Argo CD Applications with dependency ordering, health gates, or output handoff, use a [Stack](https://vcluster.com/docs/platform/understand/what-are-stacks.md) to manage them as one unit. [Create a Stack template →](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) ### Deploy and connect[​](#deploy-and-connect "Direct link to Deploy and connect") Tenant clusters can be created from the Platform UI, CLI (`vcluster create`), or Platform API. This makes Platform a good fit for GitOps workflows and automated provisioning pipelines. Once created, Platform generates a `kubeconfig` scoped to the tenant cluster. Users access their cluster directly. They never need credentials for the underlying Control Plane Cluster. ![Template selection screen for creating a tenant cluster with template cards and a version selector](/docs/assets/images/tenant-cluster-template-selection-interface-ee508bf1e631ef691d2f9cd91ff7e757.png) Users start tenant cluster creation by choosing an approved template and version. After selecting a template, users review the resolved configuration, choose where the tenant cluster runs, and confirm the generated `vcluster.yaml` before deployment. ![Tenant cluster creation screen showing the cluster name, owner, control plane host selection, and computed vcluster.yaml](/docs/assets/images/create-tenant-cluster-configuration-interface-6a52db223f0d3cb31b6bee564dfff124.png) The create flow resolves the selected template into the final cluster configuration before deployment. [Create a tenant cluster →](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-no-template.md) ### Upgrade[​](#upgrade "Direct link to Upgrade") Platform coordinates vCluster version upgrades with rolling updates. Upgrades can be initiated from the UI or automated through Platform's GitOps integration. Configuration changes to `vcluster.yaml` can be applied at the same time as version upgrades. ## Sleep and wake[​](#sleep-and-wake "Direct link to Sleep and wake") Platform can automatically sleep idle tenant clusters, suspending the control plane and stopping synced workloads to free resources. When activity resumes, Platform wakes the cluster and restores it to its previous state. The tenant experience is seamless. Their next `kubectl` command triggers the wake automatically. Sleep can be configured three ways: * **Manual** — sleep or wake a cluster on demand from the UI or CLI * **Inactivity timeout** — sleep after a configurable period with no API activity * **Schedule** — sleep and wake on a CRON schedule, useful for environments with predictable working hours Auto-delete works the same way. Tenant clusters idle beyond a configured threshold are deleted automatically. This is useful for ephemeral developer and CI environments where clusters are created frequently and often abandoned. ![Tenant cluster details page showing a sleeping virtual cluster with a Wake Up action](/docs/assets/images/virtual-cluster-sleeping-status-dashboard-196558d9f07c0db032618467e4fe26cc.png) Sleeping tenant clusters remain visible in Platform and can be woken when users need them again. [Configure sleep mode →](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md) ## Access control[​](#access-control "Direct link to Access control") Platform uses a layered RBAC model. Administrators control access at the platform, project, and individual tenant cluster level. * **Platform roles** — control access to global settings, connected clusters, and user management * **Project roles** — control access to tenant clusters, spaces, and resources within a project * **Tenant cluster roles** — control what a user can do inside a specific tenant cluster Platform integrates with any OIDC provider for SSO, including Okta, Azure AD, Dex, and others. Teams and group memberships from your identity provider map directly to Platform roles. ![Project members and roles panel showing project users with assigned roles](/docs/assets/images/project-member-management-70b4e86e968171cbe135ee0434fd7fa1.png) Project membership controls who can access resources in the project and which project role they receive. [Configure SSO →](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md) ## Node management[​](#node-management "Direct link to Node management") For tenant clusters running on [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md), Platform manages node lifecycle. Nodes are provisioned and joined automatically based on workload demand, and deprovisioned when no longer needed. Node providers integrate with cloud APIs (AWS, GCP, Azure, and others) to provision VMs that join as private nodes. For bare metal infrastructure, [vMetal](https://vcluster.com/docs/platform/administer/bare-metal/overview.md) handles physical server provisioning using the same model. This is the operational foundation for AI cloud providers. Tenant clusters with GPU nodes scale up when a customer submits a job and scale down when it finishes, without manual intervention. ![Node provider configuration screen showing KubeVirt provider settings, node types, and the generated NodeProvider resource](/docs/assets/images/interface-for-creating-a-new-node-provider-e4afd460cadaf9550396c45e2746d8bc.png) Node providers define the infrastructure and node types Platform can use when tenant clusters need private nodes. [Configure node providers →](https://vcluster.com/docs/platform/administer/node-providers/overview.md) ## Connected clusters[​](#connected-clusters "Direct link to Connected clusters") Platform connects to as many Kubernetes clusters as your infrastructure requires. These become the Control Plane Clusters for your tenant cluster fleet. You can distribute tenant clusters across regions, cloud providers, or on-premises environments and manage all of them from one Platform instance. ![Host clusters dashboard showing multiple connected clusters with status, Kubernetes version, agent version, and connected time](/docs/assets/images/connected-clusters-agent-status-95cd81a261270eaf513122bd42f7f26b.png) Connected clusters report agent status and version information back to the Platform management plane. [Connect a cluster →](https://vcluster.com/docs/platform/understand/what-are-clusters.md) ## Next steps[​](#next-steps "Direct link to Next steps") * [Install vCluster Platform](https://vcluster.com/docs/platform/install/quick-start-guide.md) * [Create your first tenant cluster](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-no-template.md) * [Set up a project](https://vcluster.com/docs/platform/administer/projects/create.md) * [Configure templates](https://vcluster.com/docs/platform/administer/templates/create-templates.md) * [Standardize applications with Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md) * [Set up SSO](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md) --- # Access keys Access keys provide authentication credentials for the platform. Users can use these keys with the vCluster CLI or HTTP tools like curl to interact with the platform. ## How access keys work[​](#how-access-keys-work "Direct link to How access keys work") Access keys in vCluster platform function as authentication tokens that inherit the exact permissions of the user who creates them. When a user generates an access key, that key can perform any action the user has permission to perform. Any authenticated user on the vCluster platform can generate their own access keys through the user interface or API. When making requests through the vCluster CLI or direct API calls, these keys authenticate the request with the platform. The platform then processes the request with the same privilege level as if the generating user performed the action directly. ## Scoped access keys[​](#scoped-access-keys "Direct link to Scoped access keys") The platform supports limiting access key permissions by scoping them to specific resources. You can scope an access key to restrict its use to: A single project within the platform, limiting actions to only that project's resources. A particular tenant cluster, constraining operations to just that tenant cluster's components. A specific namespace, allowing actions only within that defined namespace boundary. Scoping provides an important security control by restricting what an access key can access. However, scoped access keys cannot access resources beyond what the generating user already has permission to access. The scoping mechanism reduces permissions rather than expanding them. ## Best practices[​](#best-practices "Direct link to Best practices") When implementing access keys in automated workflows, consider using scoped keys to improve security by limiting access to only the required resources. For example, an automated deployment process typically needs access to just one project or tenant cluster. By scoping the access key appropriately, you protect other resources in case the key is compromised. ## Example use cases[​](#example-use-cases "Direct link to Example use cases") CI/CD integration often requires access to platform resources. You can create project-scoped access keys for build pipelines to enable automated building and deployment while limiting potential security exposure. If your pipeline already issues OIDC tokens, [exchange one for a short-lived access key](https://vcluster.com/docs/platform/administer/authentication/token-exchange.md) instead of storing a long-lived key. Automated deployment tools can use tenant cluster-scoped keys to manage deployments without gaining unnecessary access to other parts of your infrastructure. Monitoring applications need to gather metrics and status information. Configure namespace-scoped keys for these systems to allow monitoring without granting broader permissions. ## Security considerations[​](#security-considerations "Direct link to Security considerations") Access key management requires careful attention to security. Rotate keys regularly to reduce the impact of potential breaches. Store keys securely using environment variables or a secrets management system rather than embedding them in code or configuration files. Apply the principle of least privilege by using the most restrictive scope possible for each access key. When a process no longer requires an access key, delete it promptly to reduce your security exposure. Changing or resetting a user's password deletes all of that user's existing access keys, including ones used by automation. If you rotate a password, plan to generate and distribute new access keys for any integrations that relied on the old ones. See [Change your password](https://vcluster.com/docs/platform/use-platform/change-your-password.md) and [Reset a user's password](https://vcluster.com/docs/platform/administer/users-permissions/managing-users/reset-password.md) for the exact effects of each method. ## Create an access key[​](#create-an-access-key "Direct link to Create an access key") 1. Go to Access & Secrets > Users & Roles. 2. Find the user you would like to generate an Access Key for from the list of users. Click the user. 3. In the User Access Key management screen click the Create Access Key button. 4. In the configuration sheet that opens, give the Access Key a display name in the Display Name field. 5. **Optional:** Expand the Limit Scope section to select the clusters, namespaces, and tenant clusters you want to limit the access key scope to. 6. Click Create. 7. In the pop-up window, copy the generated access key. You cannot retrieve it again after closing this dialog. 8. Click Ok. ## Use access keys[​](#use-access-keys "Direct link to Use access keys") ### Log in using CLI[​](#log-in-using-cli "Direct link to Log in using CLI") You can use an access key to login to vCluster Platform using the vCluster CLI: ``` vcluster platform login my-vcluster-platform-url.com --access-key ACCESS_KEY ``` ### Manually create kube config[​](#manually-create-kube-config "Direct link to Manually create kube config") Optionally, you can manually create your kube config by first generating an access key and then using the following template: my-kube-config.yaml ``` apiVersion: v1 kind: Config clusters: - cluster: server: https://$LOFT_URL/kubernetes/management # Optional: if the server uses an insecure certificate # insecure-skip-tls-verify: true name: loft contexts: - context: cluster: loft user: loft name: loft current-context: loft users: - name: loft user: token: $ACCESS_KEY ``` Replace the following placeholders: * `$LOFT_URL`: the loft host you connect to * `$ACCESS_KEY`: the access key Save the modified template as `my-kube-config.yaml`. You can access your tenant cluster using the newly created kube config file: ``` KUBECONFIG=my-kube-config.yaml kubectl get users ``` ## Related topics[​](#related-topics "Direct link to Related topics") * [Platform CLI hub](https://vcluster.com/docs/platform/reference/platform-cli.md): UI-to-CLI equivalents across common Platform tasks, including creating access keys. * [Access the management API](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md#access-the-management-api): on a [multi-region platform](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md), an access key is the only supported way to reach the management API. * [OIDC token exchange](https://vcluster.com/docs/platform/administer/authentication/token-exchange.md): exchange a short-lived OIDC token for an access key instead of storing a long-lived one. --- # Use the vCluster Platform as an OIDC provider | | | Enterprise | | | | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Platform as OIDC Provider](https://vcluster.com/docs/platform/administer/authentication/oidc-provider.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). vCluster Platform can act as an OpenID Connect provider for other services, e.g. a self-hosted container registry using [Harbor](https://goharbor.io). ## Configure vCluster Platform as an OIDC Provider[​](#configure-vcluster-platform-as-an-oidc-provider "Direct link to Configure vCluster Platform as an OIDC Provider") To tell vCluster Platform to act as OIDC provider, navigate to `Admin > Config` and add the `oidc` section to your config: ``` auth: ... oidc: enabled: true clients: - name: "Example Client" clientId: "loft" clientSecret: "MYCLIENTSECRET" redirectURIs: - http://my-allowed-redirect-uri # - ADD A 2nd OIDC CLIENT HERE # - ADD A 3rd OIDC CLIENT HERE ``` ## Add OIDC Clients to vCluster Platform OIDC Using Secrets[​](#adding-oidc-clients-to-vcluster-platform-oidc-using-secrets "Direct link to Add OIDC Clients to vCluster Platform OIDC Using Secrets") You can add OIDC clients to vCluster Platform using Kubernetes secrets. These secrets should contain the string fields `name`, `clientID`, `clientSecret`, and `redirectURIs`. Multiple redirect URIs should be delimitted by `\n`. vCluster Platform will only recognize an OIDC client secret if they posses the labels `clientID` and `component`. The value for `clientID` should match the `clientID` field's value and the value for `component` should be `oidcServer`. Here is an example of a yaml manifest that can be used to manage an OIDC client for vCluster Platform's OIDC provider: ``` apiVersion: v1 kind: Secret metadata: name: test-oidc-client namespace: vcluster-platform # Namespace where vCluster Platform is installed labels: clientID: loft2 component: oidcServer type: Opaque stringData: name: "Example Client" clientID: "loft2" clientSecret: "MYCLIENTSECRET" redirectURIs: http://my-allowed-redirect-uri\nhttp://my-allowed-redirect-uri2 # delimit multiple uris with newlines ``` ## Use vCluster OIDC Provider in Other Applications[​](#use-vcluster-oidc-provider-in-other-applications "Direct link to Use vCluster OIDC Provider in Other Applications") To configure vCluster Platform as an OIDC provider somewhere else, you can fill out the following fields with: * OIDC Provider Endpoint / Issuer: * OIDC Client ID: loft * OIDC Client Secret: MYCLIENTSECRET * Group Claim Name: groups * Available OIDC Scopes: offline\_access,openid,groups,email,profile With this configuration, your vCluster Platform users will be able to authenticate in another application using their vCluster Platform account (which may itself be based on another SSO provider). --- # Reset Admin Password If you're locked out of vCluster Platform, you can reset the admin password directly from the command line. If you still have UI access, use [Reset a user's password](https://vcluster.com/docs/platform/administer/users-permissions/managing-users/reset-password.md) instead. It works for the admin account too. No authentication methods on the login page If you don't see any authentication methods on the login page at all, password login was likely disabled and you lost access to SSO. Re-enable password login first, then come back to reset the password. See [Recovery](https://vcluster.com/docs/platform/how-to/disable-local-admin.md#recovery). ## vCluster CLI[​](#vcluster-cli "Direct link to vCluster CLI") If you are still authenticated using the vCluster CLI you can easily reset the admin password: ``` vcluster platform reset password --user=admin ``` Existing sessions and access keys are invalidated Resetting a password through the CLI deletes all of that user's existing access keys, including browser sessions, other CLI sessions, and any API tokens the user generated. Anyone using the old password or an existing access key for this user must log in again. This does not apply to the [Kubectl](#kubectl) method below, which patches the password secret directly and leaves existing sessions and access keys untouched. ## Kubectl[​](#kubectl "Direct link to Kubectl") If you don't have CLI access, you can reset the password by patching the underlying Kubernetes secret directly. ### Find the password secret[​](#find-the-password-secret "Direct link to Find the password secret") Run the following command to look up the admin user's password reference: ``` $ kubectl get user admin -o yaml apiVersion: management.loft.sh/v1 kind: User metadata: ... spec: passwordRef: key: password secretName: loft-user-secret-admin secretNamespace: vcluster-platform ... ``` Note the `secretName` and `secretNamespace` under `passwordRef`. Older installs may use the `loft` namespace instead of `vcluster-platform`. You'll use both values in the next step. ### Set a new password[​](#set-a-new-password "Direct link to Set a new password") Replace `SECRET_NAME`, `SECRET_NAMESPACE`, and `my-new-password` below with the values from the previous step and your new password, then run: ``` SECRET_NAME=loft-user-secret-admin SECRET_NAMESPACE=vcluster-platform NEW_PASSWORD_HASH=$(echo -n "my-new-password" | sha256sum | awk '{print $1}') kubectl get secret "$SECRET_NAME" -n "$SECRET_NAMESPACE" -o json \ | jq --arg password "$(printf '%s' "$NEW_PASSWORD_HASH" | base64)" '.data["password"]=$password' \ | kubectl apply -f - ``` After that you should be able to log in to vCluster Platform with the user `admin` and your new password. --- # OIDC token exchange The platform supports [RFC 8693](https://datatracker.ietf.org/doc/html/rfc8693) token exchange. You can exchange an external OIDC token for a platform [access key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md). Automation tools and CI/CD pipelines can use this flow to authenticate without storing long-lived secrets. ## How it works[​](#how-it-works "Direct link to How it works") An external identity provider issues a short-lived OIDC token to a workload. The workload sends the token to the platform. The platform verifies the token and returns an access key. The workload uses the access key to log in with [`vcluster platform login`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_login), or directly in API requests. The access key has the same permissions as the platform user that the token identity maps to. The exchange follows these steps: 1. The pipeline requests an OIDC token from its runtime, such as GitHub Actions or Azure DevOps. 2. The pipeline sends the token to the platform token exchange endpoint. 3. The platform verifies the token against the configured SSO connector and issues an access key. 4. The pipeline uses the access key to log in. ## Token exchange endpoint[​](#token-exchange-endpoint "Direct link to Token exchange endpoint") ``` POST https:///auth//token Content-Type: application/x-www-form-urlencoded ``` **Parameters:** | Parameter | Value | | -------------------- | ---------------------------------------------------------------------------------------------- | | `grant_type` | `urn:ietf:params:oauth:grant-type:token-exchange` | | `subject_token` | The OIDC token from the external provider | | `subject_token_type` | `urn:ietf:params:oauth:token-type:id_token` or `urn:ietf:params:oauth:token-type:access_token` | The `` in the URL must match the `id` field of the connector in your platform configuration. Use `oidc` if you authenticate with the primary OIDC provider configured under `auth.oidc`. **Response:** ``` { "access_token": "", "issued_token_type": "urn:ietf:params:oauth:token-type:access_token", "token_type": "Bearer", "expires_in": 300 } ``` `expires_in` is the lifetime of the access key in seconds. The platform caps the key to the remaining lifetime of the subject token. The value is the shorter of the platform login TTL and the time left on the subject token. A subject token that expires in approximately 5 minutes returns an `expires_in` value no greater than `300`. Don't cache the access key for longer than `expires_in` seconds. The access key expires at a fixed time. It has no inactivity extension and no identity refresh, so a pipeline that needs a longer session must exchange a new token. ## Token types[​](#token-types "Direct link to Token types") The platform accepts two token types. **ID tokens** (`urn:ietf:params:oauth:token-type:id_token`): The platform verifies the token against the provider JWKS endpoint. The `aud` claim must match the `clientId` value in the connector configuration. Use this type for GitHub Actions. **Access tokens** (`urn:ietf:params:oauth:token-type:access_token`): The platform verifies the token against the provider JWKS endpoint. The `aud` claim must match the `resource` value in the connector configuration. Set the `resource` field in the platform configuration before you exchange access tokens. Use this type for providers such as Microsoft Entra ID. ## Configure a connector for token exchange[​](#configure-a-connector-for-token-exchange "Direct link to Configure a connector for token exchange") Token exchange works with any OIDC SSO connector. Add the connector under `auth.connectors`. For the full list of supported providers, see [SSO providers](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md). Platform connector configuration ``` auth: connectors: # The id becomes part of the exchange URL: /auth/my-provider/token - id: my-provider oidc: issuerUrl: https://my-oidc-provider.example.com # Must match the aud claim in the tokens that the provider issues clientId: # Leave empty if the provider doesn't require a client secret clientSecret: "" # Claim that identifies the platform user usernameClaim: sub # Namespace the subject to prevent collisions with other connectors usernamePrefix: "my-provider:" loftUsernameClaim: sub groupsClaim: "" ``` note If your platform uses `auth.oidc` as the primary SSO provider, token exchange is available at `/auth/oidc/token`. ### GitHub Actions[​](#github-actions "Direct link to GitHub Actions") GitHub Actions issues OIDC tokens for workflows that have the `id-token: write` permission. The issuer URL is the same for all GitHub organizations. Platform connector configuration ``` auth: connectors: - id: github-actions oidc: # Global GitHub Actions OIDC issuer issuerUrl: https://token.actions.githubusercontent.com # Must match the audience that the workflow requests clientId: https:// clientSecret: "" # repository_id is a stable numeric ID; the platform matches returning # users against this claim, so use it instead of the repository name usernameClaim: repository_id # Namespace the subject to prevent collisions with other connectors usernamePrefix: "github-actions:" # Use the repository name for the initial platform username and display name loftUsernameClaim: repository groupsClaim: "" ``` * The `clientId` must match the `audience` parameter that the workflow uses when it requests the token. * `usernameClaim` is the claim the platform uses to match a returning identity to an existing user, so use the stable `repository_id` claim rather than `repository`. A renamed repository keeps the same `repository_id`. A repository name can be reused after deletion, though, which would otherwise let a new repository inherit the old one's platform identity and permissions. The `usernamePrefix` namespaces the numeric ID to prevent collisions with subjects from other SSO connectors. * `loftUsernameClaim` provides the initial platform username and display name. The human-readable `repository` claim (`/`) works well here, while returning-user matching continues to use `usernameClaim`. For all claims GitHub Actions includes in OIDC tokens, see the [GitHub Actions OIDC reference](https://docs.github.com/en/actions/reference/security/oidc). * This configuration maps every workflow and Git ref in one repository to the same platform user. Use a more specific stable claim if separate workflows need different platform permissions. **Workflow example:** .github/workflows/platform-login.yaml ``` jobs: deploy: runs-on: ubuntu-latest permissions: id-token: write contents: read steps: - name: Get OIDC token id: oidc run: | RESPONSE=$(curl -sS -w "\n%{http_code}" \ -H "Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \ "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=https://$PLATFORM_HOST") HTTP_CODE=$(echo "$RESPONSE" | tail -1) BODY=$(echo "$RESPONSE" | sed '$d') OIDC_TOKEN=$(echo "$BODY" | jq -r '.value // empty') if [ "$HTTP_CODE" != "200" ] || [ -z "$OIDC_TOKEN" ]; then echo "OIDC token request failed with HTTP $HTTP_CODE" exit 1 fi echo "::add-mask::$OIDC_TOKEN" echo "token=$OIDC_TOKEN" >> "$GITHUB_OUTPUT" env: PLATFORM_HOST: - name: Exchange token id: exchange run: | RESPONSE=$(curl -sS -X POST \ "https://$PLATFORM_HOST/auth/github-actions/token" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=urn:ietf:params:oauth:grant-type:token-exchange" \ -d "subject_token=${{ steps.oidc.outputs.token }}" \ -d "subject_token_type=urn:ietf:params:oauth:token-type:id_token") KEY=$(echo "$RESPONSE" | jq -r '.access_token // empty') if [ -z "$KEY" ]; then echo "Token exchange failed: $RESPONSE" exit 1 fi echo "::add-mask::$KEY" echo "key=$KEY" >> "$GITHUB_OUTPUT" env: PLATFORM_HOST: - name: Install the vCluster CLI uses: loft-sh/setup-vcluster@main - name: Log in to the platform run: | vcluster platform login "https://$PLATFORM_HOST" \ --access-key "${{ steps.exchange.outputs.key }}" env: PLATFORM_HOST: ``` The `::add-mask::` command keeps the access key out of the workflow logs. ### Azure DevOps[​](#azure-devops "Direct link to Azure DevOps") Azure DevOps issues OIDC tokens through a Generic service connection that uses workload identity federation. Unlike GitHub Actions, the audience is fixed, and the trust boundary is the issuer URL instead. Platform connector configuration ``` auth: connectors: - id: azure-devops oidc: # Includes your organization GUID; this is the actual trust boundary issuerUrl: https://vstoken.dev.azure.com/ # Fixed audience: Azure DevOps doesn't let you customize it per pipeline clientId: api://AzureADTokenExchange clientSecret: "" # Azure DevOps tokens have no email claim usernameClaim: sub # Namespace the subject to prevent collisions with other connectors usernamePrefix: "azure-devops:" loftUsernameClaim: sub groupsClaim: "" ``` * The `clientId` must be `api://AzureADTokenExchange`. Azure DevOps shares this audience across every organization, so the `issuerUrl`, which includes your organization GUID, is what actually scopes the trust to your organization. * Azure DevOps tokens have no `email` claim. Set `usernameClaim` and `loftUsernameClaim` to `sub`, which follows the format `sc:////`. * The `usernamePrefix` prevents the subject from matching a user created through another SSO connector. * The pipeline needs a Generic service connection, even though the token request doesn't reference it as a resource. Assign the service connection to at least one task in the pipeline so Azure DevOps authorizes the request. Any built-in task with a `connectedService` input works, such as `FtpUpload@2` in the example below. **Pipeline example:** azure-pipelines.yaml ``` steps: # A task must reference the service connection so Azure DevOps # authorizes this pipeline to request an OIDC token for it. - task: FtpUpload@2 displayName: "Authorize service connection for OIDC" continueOnError: true timeoutInMinutes: 1 inputs: serverEndpoint: rootDirectory: $(System.DefaultWorkingDirectory) filePatterns: "nonexistent" remoteDirectory: / - script: | RESPONSE=$(curl -sS -w "\n%{http_code}" \ -H "Authorization: Bearer $SYSTEM_ACCESSTOKEN" \ -H "Content-Type: application/json" -H "Content-Length: 0" \ "$SYSTEM_OIDCREQUESTURI?api-version=7.1-preview.1&serviceConnectionId=") HTTP_CODE=$(echo "$RESPONSE" | tail -1) BODY=$(echo "$RESPONSE" | sed '$d') OIDC_TOKEN=$(echo "$BODY" | jq -r '.oidcToken // empty') if [ "$HTTP_CODE" != "200" ] || [ -z "$OIDC_TOKEN" ]; then echo "OIDC token request failed with HTTP $HTTP_CODE" exit 1 fi echo "##vso[task.setvariable variable=OIDC_TOKEN;issecret=true]$OIDC_TOKEN" displayName: "Get OIDC token" env: SYSTEM_ACCESSTOKEN: $(System.AccessToken) - script: | RESPONSE=$(curl -sS -w "\n%{http_code}" -X POST \ "https:///auth/azure-devops/token" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=urn:ietf:params:oauth:grant-type:token-exchange" \ -d "subject_token=$OIDC_TOKEN" \ -d "subject_token_type=urn:ietf:params:oauth:token-type:id_token") HTTP_CODE=$(echo "$RESPONSE" | tail -1) BODY=$(echo "$RESPONSE" | sed '$d') PLATFORM_TOKEN=$(echo "$BODY" | jq -r '.access_token // empty') if [ "$HTTP_CODE" != "200" ] || [ -z "$PLATFORM_TOKEN" ]; then echo "Token exchange failed with HTTP $HTTP_CODE: $BODY" exit 1 fi echo "##vso[task.setvariable variable=PLATFORM_TOKEN;issecret=true]$PLATFORM_TOKEN" displayName: "Exchange token" env: OIDC_TOKEN: $(OIDC_TOKEN) - script: | VERSION=$(curl -s https://api.github.com/repos/loft-sh/vcluster/releases/latest | jq -r '.tag_name') curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download/${VERSION}/vcluster-linux-amd64" chmod +x vcluster sudo mv vcluster /usr/local/bin/ displayName: "Install the vCluster CLI" - script: | vcluster platform login "https://" --access-key "$PLATFORM_TOKEN" displayName: "Log in to the platform" env: PLATFORM_TOKEN: $(PLATFORM_TOKEN) ``` Azure DevOps doesn't expose pipeline variables to a script automatically. Map each one explicitly through `env`, including `System.AccessToken` and any variable a previous step set with `setvariable`, such as `OIDC_TOKEN` and `PLATFORM_TOKEN`. Setting `issecret=true` on both bearer credentials keeps them out of the pipeline logs, the same way `::add-mask::` does for GitHub Actions. The service connection must have **Grant access permission to all pipelines** enabled, or be explicitly authorized for this pipeline, or the OIDC token request fails. ### Active Directory Federation Services (ADFS)[​](#active-directory-federation-services-adfs "Direct link to Active Directory Federation Services (ADFS)") ADFS doesn't follow the OIDC standard for access tokens. It publishes an `access_token_issuer` field in its discovery document, and it signs access tokens with that value as the `iss` claim. ADFS doesn't use the standard `issuer` value in access tokens. **Enable ADFS access token support** Set this environment variable on the platform deployment: ``` LOFT_OIDC_TRUST_ACCESS_TOKEN_ISSUER=true ``` When this variable is set, the platform reads the `access_token_issuer` field from the connector discovery document. If the field is present, the platform accepts only that value as the issuer for access tokens. The platform no longer accepts the standard issuer for access tokens. ID token validation doesn't change. note This variable applies to all connectors. Every connector with an `access_token_issuer` field in its discovery document adopts the replace behavior. ## Troubleshoot token exchange errors[​](#troubleshoot-token-exchange-errors "Direct link to Troubleshoot token exchange errors") Token verification failures return a generic error code and description. Check the platform log for the specific cause. | Error | Cause | Fix | | --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `invalid_grant`: `token verification failed` | The `issuerUrl` or audience doesn't match, the `resource` field is missing or wrong, or the connector's discovery document doesn't trust the issuer | Check the platform log for the specific connector error. Decode the token (`echo "$TOKEN" \| jq -R 'split(".")[1] \| gsub("-"; "+") \| gsub("_"; "/") \| . + ("=" * ((4 - (length % 4)) % 4)) \| @base64d \| fromjson \| {iss, aud}'`) and compare the values with the platform configuration. | | `invalid_grant`: `access token has expired` | The subject token's remaining lifetime reached zero before the platform issued an access key | Get a new token and retry immediately. | | `invalid_grant`: `too many failed token exchange attempts` (HTTP 429) | Five token exchanges from the same client IP failed within 10 minutes | Correct the token or connector configuration, wait 10 minutes, and retry. | | HTTP 404 on `/auth//token` | The connector ID doesn't match, or the connector is disabled | Check that the connector `id` in the platform configuration matches the URL, and that the authentication method is enabled. | ### Platform log messages[​](#platform-log-messages "Direct link to Platform log messages") These appear only in the platform log, not in the response the client receives. | Log message | Cause | Fix | | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | `access token authentication requires the resource (audience) to be configured` | The `resource` field is missing from the platform configuration | Add `resource: ""` to `auth.oidc` or the connector configuration. | | `oidc: expected audience "api://..." got [...]` | The `resource` field uses the `api://` prefix, but the token contains the bare GUID | Set `resource` to the bare GUID. Don't use the `api://` prefix. | | `oidc: access token issued by a different provider` | ADFS signs access tokens with `access_token_issuer` as `iss` instead of the standard issuer | Set `LOFT_OIDC_TRUST_ACCESS_TOKEN_ISSUER=true` on the platform deployment. | --- # Overview | | | Enterprise | | | | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Metal3 Integration](https://vcluster.com/docs/platform/administer/bare-metal/overview.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). vCluster Platform integrates [Metal3](https://metal3.io/) and [Ironic](https://ironicbaremetal.org/) for bare metal server lifecycle management. Physical servers are represented as `BareMetalHost` resources on a control plane cluster. The platform manages their detection, provisioning, configuration, and decommissioning for reuse. ## When to use bare metal[​](#when-to-use-bare-metal "Direct link to When to use bare metal") Bare metal servers are a good fit when workloads need direct hardware access or predictable performance that virtualization cannot provide. Common use cases include: * **GPU and accelerator workloads**: Machine learning training, inference, or video processing that requires direct GPU passthrough without a hypervisor layer. * **Performance-sensitive applications**: Latency-critical databases, real-time analytics, or high-frequency trading systems that benefit from eliminating virtualization overhead. * **Hardware passthrough**: Workloads that need access to specific hardware such as FPGAs, NICs with SR-IOV, or storage controllers. * **Compliance requirements**: Environments where dedicated physical isolation is mandated by regulatory or security policies. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before managing bare metal servers, you need a [Metal3 node provider](https://vcluster.com/docs/platform/administer/node-providers/metal3.md) configured on a control plane cluster. The node provider can deploy Metal3, Ironic, and a DHCP server automatically, or you can install them yourself. Metal3 and the Bare Metal Operator rely on admission webhooks, which require [cert-manager](https://cert-manager.io/) on the control plane cluster. Install cert-manager before deploying Metal3. The platform checks for it and warns when it is missing. ## Add servers[​](#add-servers "Direct link to Add servers") Bare metal servers are added by creating `BareMetalHost` resources on the control plane cluster, either through the platform UI or by applying them directly with kubectl. Each BareMetalHost represents a physical server and requires BMC (Baseboard Management Controller) configuration for out-of-band management. ### BMC configuration[​](#bmc-configuration "Direct link to BMC configuration") Each BareMetalHost needs a BMC address and credentials. The address scheme determines which driver Metal3 uses to communicate with the server's BMC. Common address formats: * **Redfish**: `redfish://192.168.1.100` * **IPMI**: `ipmi://192.168.1.100:623` For the full list of supported BMC drivers and address formats, see the [Metal3 Bare Metal Operator documentation](https://book.metal3.io/bmo/introduction.html). BMC credentials are stored in a Kubernetes Secret: ``` apiVersion: v1 kind: Secret metadata: name: server-01-bmc namespace: metal3-system type: Opaque stringData: username: admin password: ``` ### Add a single server[​](#add-a-single-server "Direct link to Add a single server") You can add servers through the platform UI under **Bare Metal Servers**, or by creating the resources directly: ``` apiVersion: metal3.io/v1alpha1 kind: BareMetalHost metadata: name: server-01 namespace: metal3-system labels: role: compute spec: online: false bmc: address: redfish://192.168.1.100 credentialsName: server-01-bmc disableCertificateVerification: true bootMACAddress: "aa:bb:cc:dd:ee:01" ``` Key fields: | Field | Description | | ----------------------------------------- | ---------------------------------------------------------------------------------------------------- | | `spec.online` | Desired power state. Register servers with `false` and let Ironic power them on during provisioning. | | `spec.bmc.address` | BMC endpoint URL. The scheme (`redfish://`, `ipmi://`, etc.) selects the driver. | | `spec.bmc.credentialsName` | Name of the Secret containing `username` and `password` keys. Must be in the same namespace. | | `spec.bmc.disableCertificateVerification` | Set to `true` for BMCs with self-signed certificates. | | `spec.bootMACAddress` | MAC address of the NIC used for PXE boot. | | `metadata.labels` | Used by node type selectors to match servers to node types. | ### Bulk registration[​](#bulk-registration "Direct link to Bulk registration") Add multiple servers at once by applying BareMetalHost and Secret resources together: ``` --- apiVersion: v1 kind: Secret metadata: name: server-01-bmc namespace: metal3-system type: Opaque stringData: username: admin password: --- apiVersion: metal3.io/v1alpha1 kind: BareMetalHost metadata: name: server-01 namespace: metal3-system labels: role: compute rack: rack-a spec: bmc: address: redfish://192.168.1.100 credentialsName: server-01-bmc bootMACAddress: "aa:bb:cc:dd:ee:01" --- apiVersion: v1 kind: Secret metadata: name: server-02-bmc namespace: metal3-system type: Opaque stringData: username: admin password: --- apiVersion: metal3.io/v1alpha1 kind: BareMetalHost metadata: name: server-02 namespace: metal3-system labels: role: compute rack: rack-b spec: bmc: address: ipmi://192.168.1.101:623 credentialsName: server-02-bmc bootMACAddress: "aa:bb:cc:dd:ee:02" ``` ``` kubectl apply -f servers.yaml ``` ## Server lifecycle[​](#server-lifecycle "Direct link to Server lifecycle") BareMetalHost resources go through the following states: | State | Description | | ---------------- | ------------------------------------------------------------------------------------------------------------------ | | `registering` | The server is being registered in the Ironic database and BMC credentials are verified. | | `inspecting` | Hardware inventory is actively being collected — CPU, RAM, NICs, disks, firmware, and PCIe devices (such as GPUs). | | `available` | Server is ready to be provisioned. | | `provisioning` | OS image is being written and cloud-init is being configured. | | `provisioned` | Server is running with the configured OS. | | `deprovisioning` | Server is being cleaned and returned to available state. | | `error` | An error occurred. Check the BareMetalHost status for details. | When a Machine claims a server, the platform powers it on and it moves from `available` through `provisioning` to `provisioned`. When the claim is removed, the server is deprovisioned, powered off, and returned to `available`. Ironic controls server power through the BMC. Register a `BareMetalHost` with `spec.online: false` and let Ironic power the server on when a Machine claims it. Ironic powers it off again when the claim is removed. ## DHCP and PXE boot[​](#dhcp-and-pxe-boot "Direct link to DHCP and PXE boot") When the Metal3 node provider deploys a DHCP server, it automatically handles the PXE boot flow. The DHCP server acts as a proxy between the bare metal servers and Ironic, which may reside in a different network. It forwards the API communication needed for PXE boot and OS installation. No manual DHCP or PXE configuration is required when using the provider-managed DHCP server. --- # Configure the Agent As outlined in the 'What are Clusters' section, during the initial setup, vCluster Platform tries to install a vCluster Platform Agent on the connected cluster to handle further communication with the cluster. vCluster Platform allows certain level of control over the installation and upgradation of the vCluster Platform Agent release. ## Find agent pods[​](#find-agent-pods "Direct link to Find agent pods") The platform agent runs as a pod in each connected cluster. Locate agent pods to verify connectivity and troubleshoot issues: Modify the following with your specific values to generate a copyable command: MANAGEMENT\_NAMESPACEvcluster-platform List agent pods ``` kubectl get pods -n vcluster-platform -l app=loft ``` Example output: ``` NAME READY STATUS RESTARTS AGE loft-d6c6d5576-7kqwx 1/1 Running 0 9d ``` ## Agent values[​](#agent-values "Direct link to Agent values") The vCluster Platform Agent is installed on the connected cluster as a Helm Release. Provide additional values for the release by setting the `loft.sh/agent-values` annotation on the Cluster object. These values are merged with the default configuration values of the Agent's chart and subsequently applied to the Agent release. ### Set agent values using the UI[​](#set-agent-values-using-the-ui "Direct link to Set agent values using the UI") 1. Go to Infrastructure > Control Plane Clusters. 2. Click the Edit option for the cluster you want to modify. 3. In the configuration sheet that opens, click the Agent tab and enter your values in Extra Agent Values using YAML format. 4. Click Save Changes. ### Set agent values using the CLI[​](#set-agent-values-using-the-cli "Direct link to Set agent values using the CLI") Export the current Cluster resource (this is a platform custom resource, not vcluster.yaml): Modify the following with your specific values to generate a copyable command: CLUSTER\_NAMEmy-cluster Export Cluster resource ``` export CLUSTER_NAME=my-cluster kubectl get cluster $CLUSTER_NAME -o yaml > cluster-resource.yaml ``` Edit `cluster-resource.yaml` and add the `loft.sh/agent-values` annotation: cluster-resource.yaml excerpt ``` apiVersion: management.loft.sh/v1 kind: Cluster metadata: name: my-cluster annotations: # Add this annotation to configure agent loft.sh/agent-values: | resources: requests: memory: 256Mi cpu: 100m spec: displayName: My Cluster managementNamespace: vcluster-platform ``` info The Cluster resource is a custom resource managed by the platform. It represents a connected cluster and stores configuration like agent values. Apply the updated configuration: Apply updated configuration ``` kubectl apply -f cluster-resource.yaml ``` ### Security context configuration[​](#security-context-configuration "Direct link to Security context configuration") Security context is one of the common agent configurations. Agent pods support multiple levels of security context configuration. Security Context Precedence When multiple security contexts are configured, the agent uses this precedence order (highest to lowest): 1. Cluster-specific `agentValues` (using `loft.sh/agent-values` annotation) 2. Platform-wide `agentValues` (in platform values.yaml) 3. Platform default `securityContext` and `podSecurityContext` See [Platform configuration](https://vcluster.com/docs/platform/configure/platform-configs/overview.md) for platform-wide settings. Inspect the security context applied to agent pods: Modify the following with your specific values to generate a copyable command: MANAGEMENT\_NAMESPACEvcluster-platform View security contexts ``` kubectl get pod -n vcluster-platform -l app=loft -o yaml | grep -A 10 "securityContext:" ``` Example output: ``` securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL privileged: false runAsNonRoot: true ``` Container vs Pod Security Context * **Container Security Context** (`securityContext`): Applied to individual containers within a pod. Controls settings like user ID, privilege escalation, and capabilities. * **Pod Security Context** (`podSecurityContext`): Applied at the pod level. Controls settings like filesystem group, supplemental groups, and sysctls that affect all containers in the pod. Both Agent Pod and Agent Proxy Pod (deployed during upgrades) follow the same precedence order. #### Example: Platform-wide security context[​](#example-platform-wide-security-context "Direct link to Example: Platform-wide security context") Configure security contexts in your `vcluster-platform` values file during platform installation: vcluster-platform values.yaml ``` # Platform-wide default security contexts securityContext: allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 1000 capabilities: drop: - ALL podSecurityContext: fsGroup: 2000 runAsGroup: 3000 # Default agent values that apply to all connected clusters agentValues: securityContext: allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 1001 capabilities: drop: - ALL podSecurityContext: fsGroup: 2001 runAsGroup: 3001 ``` #### Example: Cluster-specific security context[​](#example-cluster-specific-security-context "Direct link to Example: Cluster-specific security context") Override security contexts for specific clusters using the `loft.sh/agent-values` annotation: cluster-specific-agent-values.yaml ``` # Applied using loft.sh/agent-values annotation on the Cluster resource securityContext: allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 1002 capabilities: drop: - ALL add: - NET_BIND_SERVICE podSecurityContext: fsGroup: 2002 runAsGroup: 3002 supplementalGroups: - 4000 - 5000 ``` To apply cluster-specific security context using the UI, follow the steps in [Set agent values using the UI](#set-agent-values-using-the-ui) and add your security context configuration in the Extra Agent Values field. ## Disable the vCluster Platform agent[​](#disable-the-vcluster-platform-agent "Direct link to Disable the vCluster Platform agent") There might be cases where you don't want vCluster Platform to automatically handle vCluster Platform agent updates. This can be achieved either by setting the environment variable `DISABLE_AGENT` to `true` in the vCluster Platform container or by setting the annotation `loft.sh/cluster-ignore-agent: 'true'` on a connected cluster. 1. Go to Infrastructure > Control Plane Clusters. 2. Click the Edit option for the cluster you want to modify. 3. In the configuration sheet that opens, click the Agent tab and enable Ignore Agent. 4. Click Save Changes. warning If you do not install vCluster Platform agent into a connected cluster at all, certain features, such as Spaces, Auto Sleep, Apps, Accounts, Account Quotas & Security Templates will not be available in the cluster ## Troubleshoot[​](#troubleshoot "Direct link to Troubleshoot") For issues configuring agent values or manual deployment, see [Troubleshoot connected cluster connection failures](https://vcluster.com/docs/platform/troubleshoot/connected-cluster-connectivity.md). --- # Agent Upgrades ## Platform-managed agents[​](#platform-managed-agents "Direct link to Platform-managed agents") Platform-managed agents automatically receive upgrades controlled by the control plane. These automatic upgrades also handle more significant agent changes and migrations. All agents connect using an egress-only connection to the platform. ### Upgrade process[​](#upgrade-process "Direct link to Upgrade process") When the platform initiates an upgrade on an egress-only agent, it will first deploy a proxy-only agent. As the name suggests, this proxy-only agent solely aims to proxy the communication between the platform and the connected cluster's Kubernetes API server. The platform deploys the proxy-only agent by creating a deployment in the connected cluster's agent's namespace. It additionally reuses the agent's service account and associated RBAC permissions. Once the proxy-only agent is healthy, the platform upgrades the agent deployment. In case of a failure during the upgrade, the platform can still connect using the proxy-only agent and perform a rollback to a previous agent version. When the platform starts the agent upgrade, the agent will be unavailable for a brief period until it restarts and reconnects. This proxy-only agent maintains a connection to the connected cluster's Kubernetes apiserver during that upgrade period and serves as a backup connection. The proxy-only agent exists for the entire duration of the upgrade process, and the platform uses it to communicate with the connected cluster's Kubernetes API server. By doing so, the platform ensures that the connected cluster remains operational and can continue to receive updates and perform operations. Once the agent upgrade is complete, the platform will remove the proxy-only agent and the associated resources from the connected cluster. ## Self-managed agents[​](#self-managed-agents "Direct link to Self-managed agents") If the platform does not manage your agents, you can still manually upgrade them. * CLI * Helm 1. You can verify that the [vCluster Platform Agent](https://vcluster.com/docs/platform/administer/clusters/advanced/agent-config.md) is installed in the cluster by running: ``` vcluster platform list clusters ``` The expected output would be: ``` CLUSTER | AGE --------------------+-------- cluster-name | 7s ``` In this example, the cluster where the platform agent is installed is named `cluster-name`. 2. By providing the Helm chart version and values configuration, you can install or upgrade the existing platform agent deployment for the `cluster-name` cluster using: ``` vcluster platform add cluster cluster-name \ --helm-chart-version v4.5 \ --helm-values my-helm-values \ --context cluster-name-context ``` Choose the appropriate version in `--helm-chart-version` and Helm chart configuration in `--helm-values`. Replace `--context` with the Kubernetes context of your target cluster. If you are already operating within that cluster’s context, you can omit the --context flag — the command will automatically install or upgrade the platform agent in the current cluster. You can find more details about the available flags for the `vcluster platform add cluster` command in the [CLI documentation](https://vcluster.com/docs/vcluster/cli/vcluster_platform_add_cluster). platform login **Connect to the Platform**: to use any `vcluster platform` command, you need to be connected to the platform. See [authentication documentation](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) for more details. When you started the platform, you would have automatically connected to the platform when you ran `vcluster platform start`. If you aren't sure if you are connected and already have a running platform, then you can login. Before running this command, be sure that your kubecontext is set to the cluster running the platform. Login to the platform. ``` vcluster platform login ``` If your kubectl context is not set to the cluster with running platform, it's possible to login to the platform using the CLI and access key. If you haven't already, you need to [create access key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md#create-an-access-key) to be able to login to the platform. Login to the platform using access key ``` vcluster platform login [platform-url] --access-key [access-key] ``` 1. If you prefer to use Helm, ensure that you have a valid access key. You can only obtain an access key after the cluster has been connected to the Platform. 2. To retrieve the access key for a connected cluster, run: ``` vcluster platform get cluster-access-key cluster-name ``` The expected output is: ``` 14:43:35 info vCluster platform host: https://loftg2fei.loft.host 14:43:35 info Access Key: xxxxxxxx ``` In this example, `cluster-name` refers to the cluster containing the [vCluster Platform Agent](https://vcluster.com/docs/platform/administer/clusters/advanced/agent-config.md) to be upgraded. Here, `https://loftg2fei.loft.host` represents the `url`, and `xxxxxxxx` is the `token`. 3. Next, use the following Helm command to upgrade the platform agent by providing the retrieved `url` and `token`: ``` helm upgrade loft vcluster-platform \ --repo https://charts.loft.sh \ --install \ --create-namespace \ --namespace vcluster-platform \ --set agentOnly=true \ --set url=https://loftg2fei.loft.host \ --set token=xxxxxxxx ``` Ensure that the namespace specified with `--namespace` exists. This is the namespace where the platform agent is installed. If you deploy the platform agent’s resources into a different namespace, make sure to set `--namespace` accordingly. You can also retrieve pre-configured agent values from your platform by appending the `--values` flag with platform endpoint `--values "https://$HOST/clusters/agent-values/$CLUSTER_NAME?token=$TOKEN"`. * `$HOST` — your platform domain. * `$CLUSTER_NAME` — the name of the cluster you are connecting. * `$TOKEN` — the authentication token. In this example, `$HOST=loftg2fei.loft.host`, `$CLUSTER_NAME=cluster-name` and `$TOKEN=xxxxxxxx`. The endpoint returns a YAML file with agent values that are configured in the `agentValues` field of your vCluster Platform Helm values, merged with any cluster-specific overrides from the `loft.sh/agent-values` annotation (when present). If the platform agent is not yet installed, you can include the `--install` and `--create-namespace` flags. The combination of `--create-namespace` and `--namespace vcluster-platform` will create the `vcluster-platform` namespace in the control plane cluster if it does not already exist. All platform agent resources will then be deployed and run within that namespace. --- # DERP relay configuration info This feature is available from the **Platform** version v4.3.0 ## What is DERP?[​](#what-is-derp "Direct link to What is DERP?") DERP (Designated Encrypted Relay for Packets) is a relay protocol used by Tailscale when direct WireGuard peer-to-peer connections cannot be established. This happens when firewall rules, NAT, or restrictive network policies block direct communication between agents. By default, vCluster Platform hosts a central DERP server that all agents use as a fallback relay. When agents cannot connect directly, traffic routes through this central relay. The DERP server only forwards encrypted WireGuard packets and cannot decrypt the traffic. ## Why configure additional DERP relays?[​](#why-configure-additional-derp-relays "Direct link to Why configure additional DERP relays?") A single central DERP relay creates two problems for geographically distributed deployments: 1. **Latency**: All relayed traffic routes through one location. An agent in Tokyo communicating with an agent in Sydney must relay through a platform in Frankfurt, adding significant round-trip time. 2. **Single point of failure**: If the central DERP becomes unavailable, agents that cannot establish direct connections lose communication entirely. Configuring connected cluster agents as additional DERP relays solves both problems. Tailscale automatically selects the optimal relay based on latency, and multiple relays provide redundancy if one becomes unavailable. Fallback behavior Additional DERP relays supplement the central DERP server rather than replacing it. Agents fall back to the central relay if all custom relays become unavailable. **Use cases:** * **Multi-region deployments**: Clusters spread across continents benefit from regional relays that minimize cross-region traffic. * **High availability requirements**: Critical workloads need relay redundancy to maintain agent communication during outages. * **Latency-sensitive operations**: `kubectl` commands, log streaming, and exec sessions perform better with nearby relays. ## Configure a cluster as a DERP relay[​](#configure-a-cluster-as-a-derp-relay "Direct link to Configure a cluster as a DERP relay") To enable a connected cluster's agent as an additional DERP relay, add annotations to the Cluster resource. The agent must be publicly accessible for other agents to use it as a relay. ### Cluster annotations[​](#cluster-annotations "Direct link to Cluster annotations") | Annotation | Required | Description | | -------------------------------- | -------- | --------------------------------------------------------------------------------------------------- | | `loft.sh/derp-endpoint` | Yes | Publicly accessible endpoint of the DERP relay (domain name or IP address reachable by all agents). | | `loft.sh/derp-endpoint-insecure` | No | Set to `true` for self-signed certificates. Default: `false`. | ### Example: Basic DERP relay configuration[​](#example-basic-derp-relay-configuration "Direct link to Example: Basic DERP relay configuration") Add the DERP endpoint annotation to enable a cluster as a relay: Cluster resource with DERP relay ``` apiVersion: management.loft.sh/v1 kind: Cluster metadata: name: eu-west-cluster annotations: loft.sh/derp-endpoint: eu-west-cluster.example.com spec: config: {} ``` After applying this configuration, other agents can use this cluster as a DERP relay. Tailscale automatically selects the optimal relay based on latency. ### Example: DERP relay with self-signed certificate[​](#example-derp-relay-with-self-signed-certificate "Direct link to Example: DERP relay with self-signed certificate") For development or internal environments using self-signed certificates: Cluster resource with insecure DERP relay ``` apiVersion: management.loft.sh/v1 kind: Cluster metadata: name: dev-cluster annotations: loft.sh/derp-endpoint: dev-cluster.internal.example.com loft.sh/derp-endpoint-insecure: 'true' spec: config: {} ``` warning Self-signed certificates reduce security. Use valid TLS certificates in production environments. ## Expose the agent for DERP relay[​](#expose-the-agent-for-derp-relay "Direct link to Expose the agent for DERP relay") Before adding the DERP annotations, expose the vCluster Platform agent through an Ingress or LoadBalancer. The endpoint must be reachable by all agents in the platform. Create an Ingress for the agent: ingress.yaml ``` apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: loft-agent-derp annotations: nginx.ingress.kubernetes.io/proxy-read-timeout: "43200" nginx.ingress.kubernetes.io/proxy-send-timeout: "43200" cert-manager.io/cluster-issuer: my-cluster-issuer spec: rules: - host: eu-west-cluster.example.com http: paths: - path: / pathType: ImplementationSpecific backend: service: name: loft-agent port: number: 80 tls: - hosts: - eu-west-cluster.example.com secretName: derp-relay-tls ``` Apply the Ingress: ``` kubectl apply -n vcluster-platform -f ingress.yaml ``` Verify the endpoint is accessible: ``` curl -I https://eu-west-cluster.example.com/healthz ``` After confirming connectivity, add the `loft.sh/derp-endpoint` annotation to the Cluster resource. ## Verify DERP relay configuration[​](#verify-derp-relay-configuration "Direct link to Verify DERP relay configuration") Enable verbose TSNet logging to verify relay activity, as described in [Enable verbose agent and platform logs](https://vcluster.com/docs/platform/troubleshoot/connected-cluster-connectivity.md#enable-verbose-agent-and-platform-logs). `LOFT_LOG_TSNET_CONTROL_PLANE` covers the platform's DERP relay and mesh traffic, and `LOFT_LOG_TSNET_AGENT` covers the equivalent logging for an agent acting as a relay. Check the platform logs: ``` kubectl logs -n vcluster-platform -l app=loft | grep -i derp ``` ## Related[​](#related "Direct link to Related") * [Regional Cluster Endpoints](https://vcluster.com/docs/platform/administer/clusters/advanced/regional-cluster-endpoints.md): Reduce API latency by connecting directly to clusters * [Networking](https://vcluster.com/docs/platform/administer/clusters/advanced/networking.md): Required ports for platform communication * [Troubleshoot connected cluster connection failures](https://vcluster.com/docs/platform/troubleshoot/connected-cluster-connectivity.md): Debug connectivity issues --- # Platform External Database info This feature is available from the **Platform** version v4.8.0 Unrelated to the Platform Database feature This page covers the Kine-backed external database used for high-availability multi-replica deployments. If you are looking for the local PostgreSQL instance used for usage reporting in air-gapped environments, see [Platform Database](https://vcluster.com/docs/platform/install/air-gapped/platform-database.md). That is a separate, lighter-weight feature gated to air-gapped installs with offline license keys. ## What is an external database deployment?[​](#what-is-an-external-database-deployment "Direct link to What is an external database deployment?") An external database deployment runs multiple vCluster Platform replicas in a single Kubernetes cluster, all backed by an external database through [Kine](https://github.com/k3s-io/kine), which provides an etcd-compatible API backed by relational databases. Leader election ensures that only one replica writes to the database at a time, while the remaining replicas serve read traffic and stand ready to take over if the leader fails. This eliminates the platform as a single point of failure within the region. ## Why deploy with an external database?[​](#why-deploy-with-an-external-database "Direct link to Why deploy with an external database?") The default single-replica platform deployment works for most environments. Consider an external database deployment when you need one or more of the following: * **Control plane resilience**: If the platform pod or its node goes down, another replica takes over leadership automatically. Connected clusters continue operating through the surviving replicas. * **Strict RBAC environments**: Run the platform on a control plane cluster with limited permissions, similar to [Least Privilege Mode](https://vcluster.com/docs/platform/configure/agent-settings/least-privilege-mode.md) for agents. An external database lets the platform replicas use a shared backend without requiring broad cluster-level access. * **Protection against node failures**: Spreading replicas across nodes (using pod anti-affinity or topology spread constraints) ensures that a single node failure does not take down the platform. ### How it differs from other deployment modes[​](#how-it-differs-from-other-deployment-modes "Direct link to How it differs from other deployment modes") | | External database | [Multi-region platform](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md) | [Regional Cluster Endpoints](https://vcluster.com/docs/platform/administer/clusters/advanced/regional-cluster-endpoints.md) | | ---------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | **What is replicated** | Multiple platform replicas in a single cluster | The platform itself (full replicas in each region) | Only the agent endpoints (platform stays in one region) | | **Shared database** | Yes — all replicas share a single Kine-backed database | Yes — all regions share a single Kine-backed database | No — single platform database | | **Failover** | Automatic using leader election within the cluster | Automatic using DNS health checks | No platform failover | | **Use case** | Platform HA within a single region | Platform HA, low-latency platform API access | Low-latency `kubectl` access to clusters | ### Trade-offs[​](#trade-offs "Direct link to Trade-offs") * **External database operations**: An external database deployment requires provisioning and managing a database (RDS or equivalent), adding operational overhead compared to the default embedded PostgreSQL. * **[Cost control](https://vcluster.com/docs/platform/configure/platform-configs/cost-control.md) unavailable**: The cost control feature requires the embedded single-region database and is not compatible with the external Kine backend. warning Converting an existing single-replica platform installation to an external database deployment is not supported. An external database deployment must be configured as a fresh installation following the steps below. ### Authentication[​](#authentication "Direct link to Authentication") A PostgreSQL backend can authenticate over Kerberos/GSSAPI with a mounted keytab instead of a password or cloud IAM token, for environments that require Kerberos for all database access. See [Authenticate to PostgreSQL with Kerberos](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/kerberos.md). --- # Deploy vCluster Platform with an External Database info This feature is available from the **Platform** version v4.8.0 Modify the following with your specific values to replace on the whole page and generate copyable commands: PLATFORM\_DOMAINplatform.example.com ACCOUNT\_ID123456789012 AWS\_REGIONus-east-1 DB\_VPC\_IDvpc-xxxxxxxxx DB\_SG\_IDsg-xxxxxxxxx CLUSTER\_CONTEXTarn:aws:eks:us-east-1:123456789012:cluster/platform-ha note These instructions are tested on AWS. An external database deployment can run on other cloud providers with comparable capabilities (managed Kubernetes, a managed database, and a load balancer). These steps walk you through setting up a high-availability vCluster Platform in a single AWS region using EKS, an external Amazon Relational Database Service (RDS) database as a Kine backend, Application Load Balancer (ALB) ingress, and multiple platform replicas. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you begin, ensure you have: * An **AWS account** with sufficient IAM permissions * A **registered domain** (example: `platform.example.com`) * A **public Route 53 hosted zone** (or equivalent DNS) * The following tools installed: * `eksctl` * `kubectl` * `awscli` * `helm` * `vcluster` ## Step 1 - Create an EKS cluster[​](#step-1---create-an-eks-cluster "Direct link to Step 1 - Create an EKS cluster") Create an EKS cluster with at least three nodes to spread platform replicas across failure domains. ``` eksctl create cluster \ --name platform-ha \ --region us-east-1 \ --nodes 3 \ --managed \ --with-oidc ``` ### Install the Amazon EBS CSI driver[​](#install-the-amazon-ebs-csi-driver "Direct link to Install the Amazon EBS CSI driver") The EBS CSI driver is required for dynamic provisioning of persistent volumes on EKS. Without it, tenant cluster StatefulSet pods remain in `Pending` state because the `gp2` storage class cannot provision volumes. ``` eksctl create addon --name aws-ebs-csi-driver \ --cluster platform-ha \ --region us-east-1 ``` For more details on EKS prerequisites for running tenant clusters, see the [EKS environment setup guide](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/eks.md). ### Note the EKS VPC ID and CIDR[​](#note-the-eks-vpc-id-and-cidr "Direct link to Note the EKS VPC ID and CIDR") `eksctl` creates a VPC automatically. Note the VPC ID and CIDR — you need these when creating the database security group (Step 3) and VPC peering (Step 4). ``` aws eks describe-cluster \ --name platform-ha \ --region us-east-1 \ --query 'cluster.resourcesVpcConfig.vpcId' \ --output text ``` To find the CIDR range: Modify the following with your specific values to generate a copyable command: EKS\_VPC\_IDvpc-yyyyyyyyy ``` aws ec2 describe-vpcs \ --vpc-ids vpc-yyyyyyyyy \ --region us-east-1 \ --query 'Vpcs[0].CidrBlock' \ --output text ``` The default `eksctl` CIDR is `192.168.0.0/16`. Use the actual value from the output above in subsequent steps. ## Step 2 - Install AWS load balancer controller[​](#step-2---install-aws-load-balancer-controller "Direct link to Step 2 - Install AWS load balancer controller") ### Create IAM policy[​](#create-iam-policy "Direct link to Create IAM policy") ``` curl -o iam_policy.json \ https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json aws iam create-policy \ --policy-name AWSLoadBalancerControllerIAMPolicy \ --policy-document file://iam_policy.json ``` ### Create IAM service account[​](#create-iam-service-account "Direct link to Create IAM service account") Create an IAM Roles for Service Accounts (IRSA) association to allow the load balancer controller pods to assume the IAM role. ``` eksctl create iamserviceaccount \ --cluster platform-ha \ --namespace kube-system \ --name aws-load-balancer-controller \ --attach-policy-arn arn:aws:iam::123456789012:policy/AWSLoadBalancerControllerIAMPolicy \ --approve ``` ### Install with Helm[​](#install-with-helm "Direct link to Install with Helm") ``` helm repo add eks https://aws.github.io/eks-charts helm repo update ``` ``` helm install aws-load-balancer-controller eks/aws-load-balancer-controller \ -n kube-system \ --set clusterName=platform-ha \ --set serviceAccount.create=false \ --set serviceAccount.name=aws-load-balancer-controller ``` ## Step 3 - Create the database (Kine backend)[​](#step-3---create-the-database-kine-backend "Direct link to Step 3 - Create the database (Kine backend)") Create an RDS instance (MariaDB) in its own **isolated VPC**, separate from the EKS cluster VPC. This keeps the database network boundary independent from the cluster workloads and simplifies security group rules. ### Create the database VPC[​](#create-the-database-vpc "Direct link to Create the database VPC") Create a VPC with a CIDR that does not overlap with the EKS cluster VPC. Enable DNS support and DNS hostnames so the RDS endpoint resolves correctly across the VPC peering connection. ``` aws ec2 create-vpc \ --cidr-block 10.1.0.0/16 \ --region us-east-1 \ --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=ha-platform-db-vpc}]' ``` Note the `VpcId` from the output, then enable DNS settings: ``` aws ec2 modify-vpc-attribute \ --vpc-id vpc-xxxxxxxxx \ --enable-dns-support '{"Value":true}' \ --region us-east-1 aws ec2 modify-vpc-attribute \ --vpc-id vpc-xxxxxxxxx \ --enable-dns-hostnames '{"Value":true}' \ --region us-east-1 ``` ### Create subnets and DB subnet group[​](#create-subnets-and-db-subnet-group "Direct link to Create subnets and DB subnet group") Create subnets in at least two availability zones (required by RDS), then create a DB subnet group. ``` aws ec2 create-subnet \ --vpc-id vpc-xxxxxxxxx \ --cidr-block 10.1.1.0/24 \ --availability-zone us-east-1a \ --region us-east-1 \ --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=ha-platform-db-subnet-1a}]' aws ec2 create-subnet \ --vpc-id vpc-xxxxxxxxx \ --cidr-block 10.1.2.0/24 \ --availability-zone us-east-1b \ --region us-east-1 \ --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=ha-platform-db-subnet-1b}]' ``` Modify the following with your specific values to generate a copyable command: DB\_SUBNET\_1A\_IDsubnet-xxxxxxxxx DB\_SUBNET\_1B\_IDsubnet-yyyyyyyyy ``` aws rds create-db-subnet-group \ --db-subnet-group-name ha-platform-db-subnet \ --db-subnet-group-description "Isolated VPC subnet group for platform RDS" \ --subnet-ids subnet-xxxxxxxxx subnet-yyyyyyyyy \ --region us-east-1 ``` ### Create a security group for the database[​](#create-a-security-group-for-the-database "Direct link to Create a security group for the database") Create a security group that allows inbound MariaDB traffic (port 3306) from the EKS cluster VPC CIDR range. ``` aws ec2 create-security-group \ --group-name ha-platform-db-sg \ --description "Allow MariaDB from EKS VPC via peering" \ --vpc-id vpc-xxxxxxxxx \ --region us-east-1 ``` Modify the following with your specific values to generate a copyable command: EKS\_VPC\_CIDR10.0.0.0/16 ``` aws ec2 authorize-security-group-ingress \ --group-id sg-xxxxxxxxx \ --protocol tcp --port 3306 \ --cidr 10.0.0.0/16 \ --region us-east-1 ``` ### Create the RDS instance[​](#create-the-rds-instance "Direct link to Create the RDS instance") tip Enable automated backups by adding `--backup-retention-period 7` (or your preferred retention in days) to the command below. Without this, RDS does not retain automated snapshots and you must create manual snapshots for disaster recovery. Modify the following with your specific values to generate a copyable command: ADMIN\_PASSWORDyour-password ``` aws rds create-db-instance \ --engine mariadb \ --db-instance-identifier mariadb-ha-platform \ --allocated-storage 20 \ --region us-east-1 \ --db-instance-class db.t3.medium \ --master-username admin \ --master-user-password your-password \ --db-subnet-group-name ha-platform-db-subnet \ --vpc-security-group-ids sg-xxxxxxxxx \ --no-publicly-accessible \ --enable-iam-database-authentication ``` Wait for the instance to become available: ``` aws rds wait db-instance-available \ --db-instance-identifier mariadb-ha-platform \ --region us-east-1 ``` ### Create an IAM policy for RDS access[​](#create-an-iam-policy-for-rds-access "Direct link to Create an IAM policy for RDS access") Create an IAM policy that grants the `rds-db:connect` permission for the `kine` database user. Replace `` with the RDS instance resource ID. To find the value of the resource ID, use `aws rds describe-db-instances --query 'DBInstances[0].DbiResourceId'`. Modify the following with your specific values to generate a copyable command: DBI\_RESOURCE\_IDdb-XXXXXXXXXXXXXXXXXXXXXXXXXX ``` aws iam create-policy \ --policy-name RDSIAMAuthKine \ --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "rds-db:connect", "Resource": "arn:aws:rds-db:us-east-1:123456789012:dbuser:db-XXXXXXXXXXXXXXXXXXXXXXXXXX/kine" } ] }' ``` ### Create an IAM role for Pod Identity[​](#create-an-iam-role-for-pod-identity "Direct link to Create an IAM role for Pod Identity") Create an IAM role with a trust policy that allows the EKS Pod Identity agent to assume it. ``` aws iam create-role \ --role-name PlatformKineRDSRole \ --assume-role-policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "pods.eks.amazonaws.com" }, "Action": [ "sts:AssumeRole", "sts:TagSession" ] } ] }' aws iam attach-role-policy \ --role-name PlatformKineRDSRole \ --policy-arn arn:aws:iam::123456789012:policy/RDSIAMAuthKine ``` ### Install the EKS Pod Identity Agent[​](#install-the-eks-pod-identity-agent "Direct link to Install the EKS Pod Identity Agent") Install the Pod Identity Agent add-on. Without this add-on, the Pod Identity association has no effect and pods cannot obtain IAM credentials. ``` aws eks create-addon \ --cluster-name platform-ha \ --addon-name eks-pod-identity-agent \ --region us-east-1 ``` Verify the add-on is `ACTIVE` before proceeding: ``` aws eks describe-addon \ --cluster-name platform-ha \ --addon-name eks-pod-identity-agent \ --region us-east-1 \ --query 'addon.status' ``` ### Create Pod Identity association[​](#create-pod-identity-association "Direct link to Create Pod Identity association") Associate the IAM role with the `loft` service account in the `vcluster-platform` namespace. This allows the platform pods to generate temporary IAM authentication tokens for the database connection. ``` aws eks create-pod-identity-association \ --cluster-name platform-ha \ --role-arn arn:aws:iam::123456789012:role/PlatformKineRDSRole \ --namespace vcluster-platform \ --service-account loft \ --region us-east-1 ``` For more details on IAM database authentication with EKS Pod Identity, see [IAM database authentication](https://vcluster.com/docs/platform/administer/connector/iam_database.md). ## Step 4 - Create VPC peering to the database[​](#step-4---create-vpc-peering-to-the-database "Direct link to Step 4 - Create VPC peering to the database") Create a VPC peering connection between the EKS cluster VPC and the database VPC. This gives the cluster network access to the RDS instance. ### Create peering connection[​](#create-peering-connection "Direct link to Create peering connection") Modify the following with your specific values to generate a copyable command: EKS\_VPC\_IDvpc-yyyyyyyyy ``` aws ec2 create-vpc-peering-connection \ --vpc-id vpc-xxxxxxxxx \ --peer-vpc-id vpc-yyyyyyyyy \ --region us-east-1 \ --tag-specifications 'ResourceType=vpc-peering-connection,Tags=[{Key=Name,Value=db-vpc-to-eks-ha-platform}]' ``` ### Accept the peering connection[​](#accept-the-peering-connection "Direct link to Accept the peering connection") Modify the following with your specific values to generate a copyable command: PEERING\_IDpcx-xxxxxxxxx ``` aws ec2 accept-vpc-peering-connection \ --vpc-peering-connection-id pcx-xxxxxxxxx \ --region us-east-1 ``` ### Enable DNS resolution[​](#enable-dns-resolution "Direct link to Enable DNS resolution") Enable DNS resolution on the peering connection so the RDS endpoint hostname resolves to its private IP from the EKS VPC. Modify the following with your specific values to generate a copyable command: PEERING\_IDpcx-xxxxxxxxx ``` aws ec2 modify-vpc-peering-connection-options \ --vpc-peering-connection-id pcx-xxxxxxxxx \ --requester-peering-connection-options AllowDnsResolutionFromRemoteVpc=true \ --accepter-peering-connection-options AllowDnsResolutionFromRemoteVpc=true \ --region us-east-1 ``` ### Add routes[​](#add-routes "Direct link to Add routes") Add routes so traffic can flow between the EKS VPC and the database VPC. warning Add routes to **every** route table whose subnets host EKS nodes, including public route tables. If your EKS nodes run in public subnets (the `eksctl` default), missing routes on the public route table cause database connection timeouts even though VPC peering is active. **Database VPC route table** — route to the EKS VPC CIDR: Modify the following with your specific values to generate a copyable command: DB\_VPC\_ROUTE\_TABLE\_IDrtb-xxxxxxxxx EKS\_VPC\_CIDR10.0.0.0/16 PEERING\_IDpcx-xxxxxxxxx ``` aws ec2 create-route \ --route-table-id rtb-xxxxxxxxx \ --destination-cidr-block 10.0.0.0/16 \ --vpc-peering-connection-id pcx-xxxxxxxxx \ --region us-east-1 ``` **EKS VPC route tables** — route to the database VPC CIDR. Repeat for every route table associated with subnets that host EKS nodes: Modify the following with your specific values to generate a copyable command: EKS\_ROUTE\_TABLE\_IDrtb-yyyyyyyyy PEERING\_IDpcx-xxxxxxxxx ``` aws ec2 create-route \ --route-table-id rtb-yyyyyyyyy \ --destination-cidr-block 10.1.0.0/16 \ --vpc-peering-connection-id pcx-xxxxxxxxx \ --region us-east-1 ``` To list all route tables for a VPC and identify which subnets they serve: Modify the following with your specific values to generate a copyable command: VPC\_IDvpc-xxxxxxxxx ``` aws ec2 describe-route-tables \ --filters "Name=vpc-id,Values=vpc-xxxxxxxxx" \ --region us-east-1 \ --query 'RouteTables[].{RouteTableId:RouteTableId,Name:Tags[?Key==`Name`].Value|[0],Subnets:Associations[].SubnetId}' ``` ### Verify connectivity[​](#verify-connectivity "Direct link to Verify connectivity") Launch a test pod in the EKS cluster and verify it can reach the RDS endpoint: Modify the following with your specific values to generate a copyable command: RDS\_ENDPOINTmariadb-ha-platform.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com ``` kubectl run dbtest --image=busybox --restart=Never --rm -it -- \ nc -zv mariadb-ha-platform.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com 3306 ``` The connection should report `open`. If it times out, verify that routes exist on the correct route tables and that the database security group allows ingress from the cluster's VPC CIDR. ### Create the Kine database and IAM user[​](#create-the-kine-database-and-iam-user "Direct link to Create the Kine database and IAM user") Now that VPC peering is active, connect to the RDS instance through a temporary pod in the EKS cluster to create the Kine database and IAM-authenticated user. Modify the following with your specific values to generate a copyable command: RDS\_ENDPOINTmariadb-ha-platform.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com ADMIN\_PASSWORDyour-password ``` kubectl run mariadb-client --image=mariadb:latest --restart=Never --rm -i -- \ mariadb -h mariadb-ha-platform.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com \ -u admin -pyour-password -e " CREATE DATABASE IF NOT EXISTS kine; CREATE USER IF NOT EXISTS 'kine'@'%' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS'; GRANT ALL PRIVILEGES ON kine.* TO 'kine'@'%'; FLUSH PRIVILEGES; " ``` ## Step 5 - Deploy vCluster platform[​](#step-5---deploy-vcluster-platform "Direct link to Step 5 - Deploy vCluster platform") Deploy the platform with multiple replicas and the external database configuration. Unlike multi-region, no `multiRegion` block is needed — only `config.database` and `replicaCount`. Create a values file (`platform-ha-values.yaml`): Modify the following with your specific values to generate a copyable command: ADMIN\_EMAILadmin\@example.com DATABASE\_URLmariadb-ha-platform.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com ``` admin: email: admin@example.com replicaCount: 3 config: loftHost: platform.example.com database: enabled: true dataSource: "mysql://kine@tcp(mariadb-ha-platform.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com:3306)/kine" identityProvider: "aws" extraArgs: - --datastore-max-open-connections=20 # Set to 0 because IAM auth tokens expire, making idle connections stale. - --datastore-max-idle-connections=0 # Cost control requires the embedded single-region database and is not # compatible with the external Kine backend. costControl: enabled: false # Run multiple agent replicas for resilience on the connected control plane cluster. agentValues: replicaCount: 3 ``` Setting `config.database.enabled=true` with `replicaCount > 1` automatically configures: * Embedded Kubernetes (Kine) with the external database * Leader election across replicas * `RollingUpdate` deployment strategy Install the platform: * vCluster CLI * Helm ``` vcluster platform start \ --namespace vcluster-platform \ --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ --values platform-ha-values.yaml \ --no-tunnel ``` ``` helm upgrade loft vcluster-platform --install --create-namespace --repository-config='' \ --namespace vcluster-platform \ --repo "https://charts.loft.sh/" \ --kube-context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ -f platform-ha-values.yaml ``` Wait for all replicas to become ready: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ rollout status deployment/loft -n vcluster-platform ``` ## Step 6 - Configure HTTPS and Ingress[​](#step-6---configure-https-and-ingress "Direct link to Step 6 - Configure HTTPS and Ingress") ### Request an ACM certificate[​](#request-an-acm-certificate "Direct link to Request an ACM certificate") Request an AWS Certificate Manager (ACM) certificate for your domain. ``` aws acm request-certificate \ --region us-east-1 \ --domain-name "platform.example.com" \ --validation-method DNS ``` Get the certificate ARN from the output and describe the certificate to obtain DNS validation records: Modify the following with your specific values to generate a copyable command: CERTIFICATE\_ARN\_IDxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ``` aws acm describe-certificate \ --region us-east-1 \ --certificate-arn arn:aws:acm:us-east-1:123456789012:certificate/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ``` Add DNS validation CNAMEs in Route 53 and wait until certificate status is ISSUED. note You can also do this step by setting up cert-manager in the cluster instead of using AWS ACM. ### Create the Ingress (ALB)[​](#create-the-ingress-alb "Direct link to Create the Ingress (ALB)") Apply the Ingress manifest to provision an AWS ALB: Modify the following with your specific values to generate a copyable command: CERT\_IDxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ``` apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: loft namespace: vcluster-platform annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]' alb.ingress.kubernetes.io/certificate-arn: "arn:aws:acm:us-east-1:123456789012:certificate/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" alb.ingress.kubernetes.io/ssl-redirect: "443" alb.ingress.kubernetes.io/idle-timeout: "3600" alb.ingress.kubernetes.io/healthcheck-path: /healthz alb.ingress.kubernetes.io/success-codes: "200" spec: ingressClassName: alb rules: - host: platform.example.com http: paths: - path: / pathType: Prefix backend: service: name: loft port: number: 80 ``` ### Create a DNS record[​](#create-a-dns-record "Direct link to Create a DNS record") Create an A (Alias) record pointing your domain to the ALB: Modify the following with your specific values to generate a copyable command: HOSTED\_ZONE\_IDZ1234567890ABC ALB\_HOSTED\_ZONE\_IDZXXXXXXXXXX ALB\_DNS\_NAMEk8s-vcluster-loft-xxxxxxxxxx-xxxxxxxxx.us-east-1.elb.amazonaws.com ``` aws route53 change-resource-record-sets \ --hosted-zone-id Z1234567890ABC \ --change-batch '{ "Changes": [ { "Action": "UPSERT", "ResourceRecordSet": { "Name": "platform.example.com", "Type": "A", "AliasTarget": { "HostedZoneId": "ZXXXXXXXXXX", "DNSName": "k8s-vcluster-loft-xxxxxxxxxx-xxxxxxxxx.us-east-1.elb.amazonaws.com", "EvaluateTargetHealth": true } } } ] }' ``` ## Result[​](#result "Direct link to Result") You now have a fully operational **vCluster Platform external database deployment** with: * One EKS cluster with three nodes. * Three platform replicas with leader election. * External Kine database (RDS MariaDB) in an isolated VPC with IAM-based authentication (Pod Identity). * VPC peering between the cluster VPC and the database VPC. * ALB with extended idle timeout and ACM certificate. * `RollingUpdate` deployment strategy for zero-downtime upgrades. * PodDisruptionBudget (created automatically by the Helm chart) ensuring at least one replica is always available. --- # Authenticate to an External PostgreSQL Database with Kerberos info This feature is available from the **Platform** version v4.10.0 Modify the following with your specific values to replace on the whole page and generate copyable commands: REALMEXAMPLE.COM KDC\_HOSTkdc.example.com PRINCIPALkrb5-db-user PG\_FQDNpostgres.example.com DB\_NAMEkine ## Overview[​](#overview "Direct link to Overview") When you run vCluster Platform with an [external database](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/.md), the platform's embedded database connects to your PostgreSQL backend. Instead of a static password or a cloud IAM token, that connection can authenticate with **Kerberos/GSSAPI** using a mounted keytab. This suits enterprises that already operate a Kerberos realm (an MIT KDC or Active Directory) and require Kerberos for all database access. Authentication relies on existing Helm values. A passwordless `config.database.dataSource` carries the client principal and points at a mounted keytab. The generic `volumes`, `volumeMounts`, and `env` values mount the keytab and `krb5.conf`. No platform-specific Kerberos fields are required. note This applies to the platform's **external PostgreSQL** backend only. Kerberos authentication for the [Database Connector](https://vcluster.com/docs/platform/administer/connector/database.md) (the tenant cluster datastore path) is not covered here. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you begin, ensure you have: * vCluster Platform **v4.10.0 or later**. Earlier platform builds do not include the GSSAPI-capable database and fail to start against a Kerberos datasource. * An existing **Kerberos environment**: * A KDC and realm (for example `EXAMPLE.COM`). * A **client principal** for the platform, for example `krb5-db-user@EXAMPLE.COM`, that you can export to a keytab. * A **PostgreSQL server built with GSSAPI support**, reachable from the control plane cluster at a stable fully qualified domain name (FQDN), with: * The service principal `postgres/@REALM` present in the server's keytab and referenced by `krb_server_keyfile`. * A `pg_hba.conf` rule that authenticates the platform connection with `gss`. * A login role and database that the client principal maps to (see [Step 2](#step-2---create-the-postgresql-role-and-database)). * The platform already configured for an **external database deployment** (`config.database.enabled`). Follow the [external database overview](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/.md) and the [deployment walkthrough](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/deploy.md) for the surrounding high-availability setup. This page covers only the Kerberos authentication wiring. ## Step 1 - Create the database principal and export a keytab[​](#step-1---create-the-database-principal-and-export-a-keytab "Direct link to Step 1 - Create the database principal and export a keytab") Create the client principal in your KDC and export it to a keytab file. With MIT Kerberos, use `kadmin`: ``` kadmin -q "addprinc -randkey krb5-db-user@EXAMPLE.COM" kadmin -q "ktadd -k krb5-db-user.keytab krb5-db-user@EXAMPLE.COM" ``` This produces a `krb5-db-user.keytab` file containing credentials for the principal. Keep this file secure. Anyone holding it can authenticate as the principal. ## Step 2 - Create the PostgreSQL role and database[​](#step-2---create-the-postgresql-role-and-database "Direct link to Step 2 - Create the PostgreSQL role and database") Create a passwordless login role whose name matches the principal's short name and a database the role can use. Because the `pg_hba.conf` rule below uses `include_realm=0`, PostgreSQL strips the `@REALM` suffix, so the principal `krb5-db-user@EXAMPLE.COM` maps to the role `krb5-db-user`. ``` CREATE ROLE "krb5-db-user" LOGIN; CREATE DATABASE kine OWNER "krb5-db-user"; ``` Point PostgreSQL at its service keytab in `postgresql.conf`: ``` krb_server_keyfile = '/etc/postgres-keytab/krb5.keytab' ``` Add a `gss` rule to `pg_hba.conf` so remote TCP connections from the platform authenticate over Kerberos. PostgreSQL uses the first matching `pg_hba.conf` rule, so place this rule before broader `host` rules that use another authentication method. Scope the address range to the platform network where possible: ``` host all all 0.0.0.0/0 gss include_realm=0 krb_realm=EXAMPLE.COM ``` info `krbsrvname` in the datasource (Step 4) selects the `postgres` service principal, and `krb_realm` restricts which realm's principals the rule accepts. Keep `include_realm=0` and the role name aligned, otherwise the mapped role will not match and the connection is rejected. ## Step 3 - Create the keytab Secret and krb5.conf ConfigMap[​](#step-3---create-the-keytab-secret-and-krb5conf-configmap "Direct link to Step 3 - Create the keytab Secret and krb5.conf ConfigMap") Create a Kubernetes Secret holding the client keytab and a ConfigMap holding the client `krb5.conf`, both in the platform namespace (`vcluster-platform`). ``` kubectl create secret generic kine-keytab \ --from-file=krb5.keytab=krb5-db-user.keytab \ --namespace vcluster-platform ``` Write a local `krb5.conf` file that points at your KDC. When your service DNS names lack matching PTR records, disable DNS canonicalization and reverse lookups so the service principal name matches the host the platform connects to: ``` [libdefaults] default_realm = EXAMPLE.COM dns_lookup_realm = false dns_lookup_kdc = false dns_canonicalize_hostname = false rdns = false [realms] EXAMPLE.COM = { kdc = kdc.example.com admin_server = kdc.example.com } ``` Create the ConfigMap from that file: ``` kubectl create configmap krb5-conf \ --from-file=krb5.conf=krb5.conf \ --namespace vcluster-platform ``` ## Step 4 - Configure the platform Helm values[​](#step-4---configure-the-platform-helm-values "Direct link to Step 4 - Configure the platform Helm values") Add the Kerberos datasource and the keytab/`krb5.conf` mounts to your platform values file (`platform-kerberos-values.yaml`). The datasource carries **no password**. The principal lives in the user part of the connection string, `krb5_keytab` points at the mounted keytab, and `krbsrvname` selects the PostgreSQL service principal. ``` config: database: enabled: true # Passwordless GSSAPI datasource. The principal in the user part is URL-encoded # (@ becomes %40); krb5_keytab points at the mounted keytab. dataSource: "postgres://krb5-db-user%40EXAMPLE.COM@postgres.example.com:5432/kine?krb5_keytab=/etc/kine-keytab/krb5.keytab&krbsrvname=postgres" extraArgs: - --datastore-max-open-connections=5 - --datastore-max-idle-connections=2 # KRB5_CONFIG tells the GSSAPI client where to read the realm configuration. env: KRB5_CONFIG: /etc/krb5.conf # Mount the client keytab and krb5.conf into the platform pod. volumes: - name: kine-keytab secret: secretName: kine-keytab - name: krb5-conf configMap: name: krb5-conf volumeMounts: - name: kine-keytab mountPath: /etc/kine-keytab readOnly: true - name: krb5-conf mountPath: /etc/krb5.conf subPath: krb5.conf readOnly: true ``` warning The principal in the datasource user part must be **URL-encoded**. Write the `@` between the principal and the realm as `%40` (for example `krb5-db-user%40EXAMPLE.COM`). The datasource **host must exactly match** the host in the PostgreSQL service principal (`postgres/@REALM`) baked into the server keytab. A mismatch makes the GSSAPI handshake fail even when every other value is correct. note Kerberos authenticates the database connection, but it does not replace TLS. Add the PostgreSQL TLS options your environment requires to the datasource, such as `sslmode=verify-full`, and configure the corresponding database CA or client certificate values as needed. Merge these values with the rest of your external database configuration (`replicaCount`, ingress, and so on) from the [deployment walkthrough](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/deploy.md). ## Step 5 - Install and verify[​](#step-5---install-and-verify "Direct link to Step 5 - Install and verify") Install or upgrade the platform with the values file: * vCluster CLI * Helm ``` vcluster platform start \ --namespace vcluster-platform \ --values platform-kerberos-values.yaml ``` ``` helm upgrade loft vcluster-platform --install --create-namespace --repository-config='' \ --namespace vcluster-platform \ --repo "https://charts.loft.sh/" \ -f platform-kerberos-values.yaml ``` Wait for the deployment to become ready: ``` kubectl rollout status deployment/loft -n vcluster-platform ``` A ready replica confirms the GSSAPI handshake succeeded when the matching `pg_hba.conf` rule permits only `gss` for the platform connection. The datasource carries no password, so Kerberos is the only authentication path available. If the deployment stays unready, check the platform pod logs: ``` kubectl logs -n vcluster-platform -l app=loft --tail=100 ``` A wrong or absent principal surfaces a `KDC_ERR_C_PRINCIPAL_UNKNOWN` error. The KDC has no such principal, so the credential could not be acquired. Confirm that the principal in the datasource exactly matches the one exported into the mounted keytab. See [Troubleshooting External Database](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/troubleshoot.md) for more Kerberos failure modes. --- # Restore a Platform Deployment with an External Database info This feature is available from the **Platform** version v4.8.0 Modify the following with your specific values to replace on the whole page and generate copyable commands: PLATFORM\_DOMAINplatform.example.com ACCOUNT\_ID123456789012 AWS\_REGIONus-east-1 DB\_VPC\_IDvpc-xxxxxxxxx DB\_SG\_IDsg-xxxxxxxxx CLUSTER\_CONTEXTarn:aws:eks:us-east-1:123456789012:cluster/platform-ha CURRENT\_DB\_INSTANCE\_IDmariadb-ha-platform NEW\_DB\_INSTANCE\_IDmariadb-ha-platform-restored SNAPSHOT\_NAMEkine-backup-YYYY-MM-DD This procedure describes how to restore the Kine database from an RDS snapshot to a new instance and update the platform to use the restored database. Use this procedure for disaster recovery, database migration (for example, enabling IAM authentication), or point-in-time recovery. warning The platform deployment must be scaled down before switching the data source to prevent split-brain writes to the old and new databases. ## Step 1 - Create a snapshot of the current database[​](#step-1---create-a-snapshot-of-the-current-database "Direct link to Step 1 - Create a snapshot of the current database") Skip this step if you already have a snapshot to restore from. ``` aws rds create-db-snapshot \ --db-instance-identifier mariadb-ha-platform \ --db-snapshot-identifier kine-backup-YYYY-MM-DD \ --region us-east-1 ``` Wait for the snapshot to become available: ``` aws rds wait db-snapshot-available \ --db-snapshot-identifier kine-backup-YYYY-MM-DD \ --region us-east-1 ``` ## Step 2 - Restore the snapshot to a new RDS instance[​](#step-2---restore-the-snapshot-to-a-new-rds-instance "Direct link to Step 2 - Restore the snapshot to a new RDS instance") Restore the snapshot to a new instance in the database VPC. Use the same DB subnet group and security group from the original setup. Include `--enable-iam-database-authentication` if the new instance should use IAM auth. ``` aws rds restore-db-instance-from-db-snapshot \ --db-instance-identifier mariadb-ha-platform-restored \ --db-snapshot-identifier kine-backup-YYYY-MM-DD \ --db-instance-class db.t3.medium \ --db-subnet-group-name ha-platform-db-subnet \ --vpc-security-group-ids sg-xxxxxxxxx \ --no-publicly-accessible \ --enable-iam-database-authentication \ --region us-east-1 ``` Wait for the new instance to become available: ``` aws rds wait db-instance-available \ --db-instance-identifier mariadb-ha-platform-restored \ --region us-east-1 ``` Note the new endpoint: ``` aws rds describe-db-instances \ --db-instance-identifier mariadb-ha-platform-restored \ --query 'DBInstances[0].Endpoint.Address' \ --output text \ --region us-east-1 ``` If using IAM authentication, note the `DbiResourceId` of the new instance and update the `RDSIAMAuthKine` IAM policy to include it. Without this, platform pods fail with `Access denied for user 'kine'` errors because the IAM `rds-db:connect` permission is scoped to a specific RDS instance resource ID. ``` aws rds describe-db-instances \ --db-instance-identifier mariadb-ha-platform-restored \ --query 'DBInstances[0].DbiResourceId' \ --output text \ --region us-east-1 ``` Add the new resource ID to the policy's `Resource` array: Modify the following with your specific values to generate a copyable command: OLD\_DBI\_RESOURCE\_IDdb-OLDXXXXXXXXXXXXXXXXXXXXXXXXXX NEW\_DBI\_RESOURCE\_IDdb-NEWXXXXXXXXXXXXXXXXXXXXXXXXXX ``` aws iam create-policy-version \ --policy-arn arn:aws:iam::123456789012:policy/RDSIAMAuthKine \ --set-as-default \ --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "rds-db:connect", "Resource": [ "arn:aws:rds-db:us-east-1:123456789012:dbuser:db-OLDXXXXXXXXXXXXXXXXXXXXXXXXXX/kine", "arn:aws:rds-db:us-east-1:123456789012:dbuser:db-NEWXXXXXXXXXXXXXXXXXXXXXXXXXX/kine" ] } ] }' ``` ## Step 3 - Scale down the platform[​](#step-3---scale-down-the-platform "Direct link to Step 3 - Scale down the platform") Scale the deployment to zero to stop all writes to the old database. ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ scale deployment -n vcluster-platform loft --replicas=0 ``` Wait for all pods to stop: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ get pods -n vcluster-platform -l app=loft --watch ``` ## Step 4 - Update the values file[​](#step-4---update-the-values-file "Direct link to Step 4 - Update the values file") Update the `dataSource` in the values file to point to the new RDS endpoint: Modify the following with your specific values to generate a copyable command: NEW\_DATABASE\_URLmariadb-ha-platform-restored.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com ``` config: database: dataSource: "mysql://kine@tcp(mariadb-ha-platform-restored.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com:3306)/kine" ``` ## Step 5 - Upgrade the platform[​](#step-5---upgrade-the-platform "Direct link to Step 5 - Upgrade the platform") Apply the updated values file. The upgrade reapplies `replicaCount` from the values file, so this also brings the platform back up, now pointing at the restored database. * vCluster CLI * Helm ``` vcluster platform start \ --namespace vcluster-platform \ --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ --values platform-ha-values.yaml \ --upgrade \ --no-tunnel ``` Modify the following with your specific values to generate a copyable command: CHART\_VERSION4.12.0 ``` helm upgrade loft vcluster-platform --install --create-namespace --repository-config='' \ --namespace vcluster-platform \ --repo "https://charts.loft.sh/" \ --version 4.12.0 \ --kube-context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ -f platform-ha-values.yaml ``` ## Step 6 - Wait for the platform to come back up[​](#step-6---wait-for-the-platform-to-come-back-up "Direct link to Step 6 - Wait for the platform to come back up") Wait for all pods to become ready: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ rollout status deployment/loft -n vcluster-platform ``` ## Step 7 - Verify the restore[​](#step-7---verify-the-restore "Direct link to Step 7 - Verify the restore") Confirm the platform is healthy: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ get pods -n vcluster-platform -l app=loft ``` Verify the platform UI is accessible through the domain and that all replicas are running. --- # Troubleshooting External Database info This feature is available from the **Platform** version v4.8.0 Modify the following with your specific values to replace on the whole page and generate copyable commands: PLATFORM\_DOMAINplatform.example.com ACCOUNT\_ID123456789012 AWS\_REGIONus-east-1 DB\_VPC\_IDvpc-xxxxxxxxx DB\_SG\_IDsg-xxxxxxxxx CLUSTER\_CONTEXTarn:aws:eks:us-east-1:123456789012:cluster/platform-ha ## Platform pods are crashlooping[​](#platform-pods-are-crashlooping "Direct link to Platform pods are crashlooping") Check pod logs for errors: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ logs -n vcluster-platform -l app=loft --tail=50 ``` Common causes: * **Database connectivity errors**: Verify that VPC peering is active and routes exist on the correct route tables. Include public route tables if nodes are in public subnets. Confirm the database security group allows port 3306 from the cluster VPC CIDR. * **IAM authentication failures**: Confirm the Pod Identity association exists, the Pod Identity Agent add-on is `ACTIVE`, and the `RDSIAMAuthKine` IAM policy contains the correct `DbiResourceId`. * **Resource exhaustion**: Check node resource usage with `kubectl top nodes` and pod resource usage with `kubectl top pods -n vcluster-platform`. ## Leader election is stuck[​](#leader-election-is-stuck "Direct link to Leader election is stuck") If the platform is unresponsive but pods are running, the leader lease may be stuck. Check the lease status: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ get lease -n vcluster-platform -o wide ``` If the lease holder is a pod that no longer exists, restart the deployment to force a new leader election: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ rollout restart deployment/loft -n vcluster-platform ``` ## ALB health checks failing[​](#alb-health-checks-failing "Direct link to ALB health checks failing") Check the ingress status: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ get ingress -n vcluster-platform loft ``` Verify the ALB target group health in the AWS console or with: Modify the following with your specific values to generate a copyable command: TARGET\_GROUP\_ARNarn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/k8s-vcluster-loft-xxxxxxxxxx/xxxxxxxxxx ``` aws elbv2 describe-target-health \ --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/k8s-vcluster-loft-xxxxxxxxxx/xxxxxxxxxx ``` Common causes: * The platform pods are not ready (check `kubectl get pods`). * The `/healthz` endpoint is not reachable from the ALB (check security groups and the ALB target type matches the service configuration). ## Database connection timeouts[​](#database-connection-timeouts "Direct link to Database connection timeouts") If pods start but log database connection errors: 1. Verify VPC peering is active: ``` aws ec2 describe-vpc-peering-connections \ --filters "Name=status-code,Values=active" \ --region us-east-1 \ --query 'VpcPeeringConnections[].{ID:VpcPeeringConnectionId,Requester:RequesterVpcInfo.VpcId,Accepter:AccepterVpcInfo.VpcId}' ``` 2. Verify routes from the EKS VPC to the database VPC exist: Modify the following with your specific values to generate a copyable command: EKS\_VPC\_IDvpc-yyyyyyyyy ``` aws ec2 describe-route-tables \ --filters "Name=vpc-id,Values=vpc-yyyyyyyyy" \ --region us-east-1 \ --query 'RouteTables[].Routes[?DestinationCidrBlock==`10.1.0.0/16`]' ``` 3. Test connectivity from within the cluster: Modify the following with your specific values to generate a copyable command: RDS\_ENDPOINTmariadb-ha-platform.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com ``` kubectl run dbtest --image=busybox --restart=Never --rm -it -- \ nc -zv mariadb-ha-platform.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com 3306 ``` ## Kerberos authentication failures[​](#kerberos-authentication-failures "Direct link to Kerberos authentication failures") When the platform authenticates to a PostgreSQL backend over [Kerberos](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/kerberos.md), the GSSAPI handshake output appears in the platform pod logs. Check the current logs with: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ logs -n vcluster-platform -l app=loft --tail=100 ``` If the pod restarted, check the previous container logs as well: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ logs -n vcluster-platform -l app=loft --tail=100 --previous ``` Common causes: * **`KDC_ERR_C_PRINCIPAL_UNKNOWN`**: The KDC has no such principal. The principal in the datasource does not match the one exported into the mounted keytab, or it was never created in the KDC. Confirm both name and realm. * **`no GSSAPI provider registered`**: The platform build predates GSSAPI support. Upgrade to v4.10.0 or later. * **Handshake fails despite a valid principal**: The datasource host does not match the host in the PostgreSQL service principal (`postgres/@REALM`) baked into the server keytab. Make the datasource host and the service principal host identical, or disable DNS canonicalization in `krb5.conf` (`dns_canonicalize_hostname = false`, `rdns = false`). * **`gss authentication failed` from PostgreSQL**: The `pg_hba.conf` rule did not map the principal to a login role, or PostgreSQL matched a broader `host` rule before the Kerberos rule. Verify the `gss` rule appears before broader rules, the role name matches the principal short name, and `include_realm=0` and `krb_realm` are set correctly. * **TLS or certificate errors**: Kerberos authenticates the database connection, but TLS is configured separately. Verify the datasource `sslmode` and any database CA or client certificate values match your PostgreSQL server requirements. --- # Upgrade Platform with an External Database info This feature is available from the **Platform** version v4.8.0 Modify the following with your specific values to replace on the whole page and generate copyable commands: PLATFORM\_DOMAINplatform.example.com ACCOUNT\_ID123456789012 AWS\_REGIONus-east-1 DB\_VPC\_IDvpc-xxxxxxxxx DB\_SG\_IDsg-xxxxxxxxx CLUSTER\_CONTEXTarn:aws:eks:us-east-1:123456789012:cluster/platform-ha This procedure describes how to upgrade vCluster Platform with minimal downtime. Because all replicas run in a single cluster, a standard rolling upgrade handles the transition automatically. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Updated values file (`platform-ha-values.yaml`) * `kubectl` context configured for the cluster * The target chart version available in the Helm repository ## Step 1 - Upgrade the deployment[​](#step-1---upgrade-the-deployment "Direct link to Step 1 - Upgrade the deployment") * vCluster CLI * Helm ``` vcluster platform start \ --namespace vcluster-platform \ --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ --values platform-ha-values.yaml \ --upgrade \ --no-tunnel ``` Modify the following with your specific values to generate a copyable command: CHART\_VERSION4.12.0 ``` helm upgrade loft vcluster-platform --install --create-namespace --repository-config='' \ --namespace vcluster-platform \ --repo "https://charts.loft.sh/" \ --version 4.12.0 \ --kube-context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ -f platform-ha-values.yaml ``` If you use Helm 4 with `--server-side=true`, see the note on server-side apply conflicts in [Upgrade vCluster Platform](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade.md). ## Step 2 - Wait for the rollout to complete[​](#step-2---wait-for-the-rollout-to-complete "Direct link to Step 2 - Wait for the rollout to complete") ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha \ rollout status deployment/loft -n vcluster-platform ``` ## Step 3 - Verify the deployment[​](#step-3---verify-the-deployment "Direct link to Step 3 - Verify the deployment") Confirm that all platform and agent deployments are running the expected image: ``` echo -n "Platform: " && kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-ha get deployment -n vcluster-platform loft \ -o jsonpath='{.spec.template.spec.containers[0].image}' && echo ``` The platform automatically upgrades connected agents after the platform upgrade completes. --- # Ingress Suffix As outlined in the 'Tenant Clusters -> Advanced Topics -> Ingress Access' section, you can enable the 'AccessPoint' feature to access a tenant cluster API server directly by avoiding the vCluster Platform proxy. This requires a valid ingress controller to be present on the control plane cluster and a valid domain to be set in the `loft.sh/ingress-suffix` annotation on the Cluster Object. Once this is done, vCluster Platform creates a connection to the tenant cluster through an ingress instead of the default vCluster Platform proxy. This can be useful, if you want to handout the tenant cluster kubeconfig to users that do not belong to vCluster Platform. The ingress name URL is calculated in the following way: `[VirtualClusterInstance Name]-[Project Name].[ingressSuffix]`
You can set the required ingress suffix in the vCluster Platform UI: 1. Go to Infrastructure > Control Plane Clusters. 2. Click the Edit option for the cluster you want to modify. 3. In the configuration sheet that opens, click the Direct Access tab. Provide the desired domain under the Tenant Cluster Ingress Suffix field. 4. Click the Save Changes button. Pre-Requisites This will require an ingress-controller in the cluster and a wildcard DNS record on the above configured domain. Make sure to enable ssl-passthrough on the ingress controller as well or install the ingress-nginx vCluster Platform app (deprecated). --- # Deploy Multi-Region Platform on AWS (EKS) info This feature is available from the **Platform** version v4.8.0 note These instructions are tested on AWS. Multi-region platform can run on other cloud providers with comparable capabilities (managed Kubernetes, cross-region networking, latency-based DNS, and a shared database). These steps walk you through setting up vCluster Platform across two AWS regions using EKS, cross-region VPC peering, shared database (Kine), ALB ingress, and Route 53 latency-based routing. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you begin, ensure you have: * An **AWS account** with sufficient IAM permissions * A **registered domain** (example: `multi-region.example.com`) * A **public Route 53 hosted zone** * The following tools installed: * [`eksctl`](https://eksctl.io/installation/) * [`kubectl`](https://kubernetes.io/docs/tasks/tools/) * [`awscli`](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) * [`helm`](https://helm.sh/docs/intro/install/) * [`vcluster`](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md#install-the-vcluster-cli) ## Configure your values[​](#configure-your-values "Direct link to Configure your values") This guide uses resource identifiers that are unique to your AWS environment: account IDs, VPC IDs, security group IDs, cluster ARNs, ALB host names, and DNS names. Fill in the fields below once and every command on this page updates to use your values, ready to copy and run without manual substitution. Variables prefixed with `US_` and `EU_` are named for the example regions in this guide. You can deploy to any two AWS regions. Set `AWS_REGION` and `EU_REGION` to your actual region codes and update the other `US_`/`EU_` values accordingly. Expand to set page variables Modify the following with your specific values to replace on the whole page and generate copyable commands: ACCOUNT\_ID123456789012 AWS\_REGIONus-east-1 EU\_REGIONeu-west-1 DB\_VPC\_IDvpc-xxxxxxxxx DB\_SG\_IDsg-xxxxxxxxx US\_PEERING\_IDpcx-xxxxxxxxx EU\_PEERING\_IDpcx-yyyyyyyyy DB\_VPC\_ROUTE\_TABLE\_IDrtb-xxxxxxxxx US\_VPC\_CIDR10.0.0.0/16 EU\_VPC\_CIDR172.21.0.0/16 DB\_VPC\_CIDR10.1.0.0/16 US\_CLUSTER\_NAMEplatform-multi-region-us-east-1 EU\_CLUSTER\_NAMEplatform-multi-region-eu-west-1 DATABASE\_URLmariadb-multi-region.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com ADMIN\_EMAILadmin\@example.com REGIONus HOSTED\_ZONE\_IDZ1234567890ABC US\_ALB\_HOSTED\_ZONE\_IDZXXXXXXXXUS US\_ALB\_DNS\_NAMEk8s-vcluster-loft-xxxxxxxxxx-xxxxxxxxx.us-east-1.elb.amazonaws.com EU\_ALB\_HOSTED\_ZONE\_IDZXXXXXXXXEU EU\_ALB\_DNS\_NAMEk8s-vcluster-loft-xxxxxxxxxx-xxxxxxxxx.eu-west-1.elb.amazonaws.com HEALTH\_CHECK\_IDxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PLATFORM\_URLmulti-region.example.com US\_VALUES\_FILEplatform-us-east-1-values.yaml EU\_VALUES\_FILEplatform-eu-west-1-values.yaml ACCESS\_TOKENyour-access-token ## Step 1 - Create two EKS clusters[​](#step-1---create-two-eks-clusters "Direct link to Step 1 - Create two EKS clusters") Important All three VPCs (two EKS clusters and one database) must use **different, non-overlapping CIDR ranges** to allow VPC peering. **U.S. region** ``` eksctl create cluster \ --name platform-multi-region-us-east-1 \ --region us-east-1 \ --nodes 2 \ --managed \ --vpc-cidr 10.0.0.0/16 \ --with-oidc ``` **E.U. region** ``` eksctl create cluster \ --name platform-multi-region-eu-west-1 \ --region eu-west-1 \ --nodes 2 \ --managed \ --vpc-cidr 172.21.0.0/16 \ --with-oidc ``` #### Install the Amazon EBS CSI driver[​](#install-the-amazon-ebs-csi-driver "Direct link to Install the Amazon EBS CSI driver") The EBS CSI driver is required for dynamic provisioning of persistent volumes on EKS. Without it, tenant cluster StatefulSet pods remain in `Pending` state because the `gp2` storage class can't provision volumes. Install the driver as an EKS managed add-on on **each** cluster: ``` eksctl create addon --name aws-ebs-csi-driver \ --cluster platform-multi-region-us-east-1 \ --region us-east-1 eksctl create addon --name aws-ebs-csi-driver \ --cluster platform-multi-region-eu-west-1 \ --region eu-west-1 ``` For more details on EKS prerequisites for running tenant clusters, see the [EKS environment setup guide](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/eks.md). ## Step 2 - Install AWS load balancer controller[​](#step-2---install-aws-load-balancer-controller "Direct link to Step 2 - Install AWS load balancer controller") #### Create IAM policy[​](#create-iam-policy "Direct link to Create IAM policy") ``` curl -o iam_policy.json \ https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json aws iam create-policy \ --policy-name AWSLoadBalancerControllerIAMPolicy \ --policy-document file://iam_policy.json ``` #### Create IAM service account (IRSA)[​](#create-iam-service-account-irsa "Direct link to Create IAM service account (IRSA)") **U.S. region** ``` eksctl create iamserviceaccount \ --cluster platform-multi-region-us-east-1 \ --namespace kube-system \ --name aws-load-balancer-controller \ --attach-policy-arn arn:aws:iam::123456789012:policy/AWSLoadBalancerControllerIAMPolicy \ --approve ``` **E.U. region** ``` eksctl create iamserviceaccount \ --cluster platform-multi-region-eu-west-1 \ --namespace kube-system \ --name aws-load-balancer-controller \ --attach-policy-arn arn:aws:iam::123456789012:policy/AWSLoadBalancerControllerIAMPolicy \ --approve ``` #### Install using Helm[​](#install-using-helm "Direct link to Install using Helm") ``` helm repo add eks https://aws.github.io/eks-charts helm repo update ``` **U.S. region** ``` helm install aws-load-balancer-controller eks/aws-load-balancer-controller \ -n kube-system \ --set clusterName=platform-multi-region-us-east-1 \ --set serviceAccount.create=false \ --set serviceAccount.name=aws-load-balancer-controller ``` **E.U. region** ``` helm install aws-load-balancer-controller eks/aws-load-balancer-controller \ -n kube-system \ --set clusterName=platform-multi-region-eu-west-1 \ --set serviceAccount.create=false \ --set serviceAccount.name=aws-load-balancer-controller ``` ## Step 3 - Create the database (Kine backend)[​](#step-3---create-the-database-kine-backend "Direct link to Step 3 - Create the database (Kine backend)") Create an RDS instance (MariaDB) in its own **isolated VPC**, separate from the EKS cluster VPCs. This keeps the database network boundary independent from the cluster workloads and simplifies security group rules. Important For production multi-region deployments, configure RDS in an active-active setup so the database automatically fails over between regions. See [Active-Active Replication on Amazon RDS for MySQL](https://aws.amazon.com/solutions/guidance/active-active-replication-on-amazon-rds-for-mysql/) for guidance on setting this up. #### Create the database VPC[​](#create-the-database-vpc "Direct link to Create the database VPC") Create a VPC with a CIDR that doesn't overlap with either EKS cluster VPC. Enable DNS support and DNS hostnames so the RDS endpoint resolves correctly across VPC peering connections. ``` aws ec2 create-vpc \ --cidr-block 10.1.0.0/16 \ --region us-east-1 \ --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=multi-region-db-vpc}]' ``` Note the `VpcId` from the output, then enable DNS settings: ``` aws ec2 modify-vpc-attribute \ --vpc-id vpc-xxxxxxxxx \ --enable-dns-support '{"Value":true}' \ --region us-east-1 aws ec2 modify-vpc-attribute \ --vpc-id vpc-xxxxxxxxx \ --enable-dns-hostnames '{"Value":true}' \ --region us-east-1 ``` #### Create subnets and DB subnet group[​](#create-subnets-and-db-subnet-group "Direct link to Create subnets and DB subnet group") Create subnets in at least two availability zones (required by RDS), then create a DB subnet group. ``` aws ec2 create-subnet \ --vpc-id vpc-xxxxxxxxx \ --cidr-block 10.1.1.0/24 \ --availability-zone us-east-1a \ --region us-east-1 \ --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=multi-region-db-subnet-1a}]' aws ec2 create-subnet \ --vpc-id vpc-xxxxxxxxx \ --cidr-block 10.1.2.0/24 \ --availability-zone us-east-1b \ --region us-east-1 \ --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=multi-region-db-subnet-1b}]' ``` Modify the following with your specific values to generate a copyable command: DB\_SUBNET\_1A\_IDsubnet-xxxxxxxxx DB\_SUBNET\_1B\_IDsubnet-yyyyyyyyy ``` aws rds create-db-subnet-group \ --db-subnet-group-name multi-region-db-subnet \ --db-subnet-group-description "Isolated VPC subnet group for multi-region RDS" \ --subnet-ids subnet-xxxxxxxxx subnet-yyyyyyyyy \ --region us-east-1 ``` #### Create a security group for the database[​](#create-a-security-group-for-the-database "Direct link to Create a security group for the database") Create a security group that allows inbound MariaDB traffic (port 3306) from both EKS cluster VPC CIDR ranges. ``` aws ec2 create-security-group \ --group-name multi-region-db-sg \ --description "Allow MariaDB from EKS VPCs via peering" \ --vpc-id vpc-xxxxxxxxx \ --region us-east-1 ``` ``` aws ec2 authorize-security-group-ingress \ --group-id sg-xxxxxxxxx \ --protocol tcp --port 3306 \ --cidr 10.0.0.0/16 \ --region us-east-1 aws ec2 authorize-security-group-ingress \ --group-id sg-xxxxxxxxx \ --protocol tcp --port 3306 \ --cidr 172.21.0.0/16 \ --region us-east-1 ``` #### Create the RDS instance[​](#create-the-rds-instance "Direct link to Create the RDS instance") Modify the following with your specific values to generate a copyable command: ADMIN\_PASSWORDyour-password ``` aws rds create-db-instance \ --engine mariadb \ --db-instance-identifier mariadb-multi-region \ --allocated-storage 20 \ --region us-east-1 \ --db-instance-class db.t3.medium \ --master-username admin \ --master-user-password your-password \ --db-subnet-group-name multi-region-db-subnet \ --vpc-security-group-ids sg-xxxxxxxxx \ --no-publicly-accessible \ --enable-iam-database-authentication ``` Wait for the instance to become available: ``` aws rds wait db-instance-available \ --db-instance-identifier mariadb-multi-region \ --region us-east-1 ``` #### Create the Kine database and IAM user[​](#create-the-kine-database-and-iam-user "Direct link to Create the Kine database and IAM user") Connect to the database instance as the `admin` user and create the `kine` database and user. The user authenticates with the `AWSAuthenticationPlugin` instead of a password. ``` CREATE DATABASE IF NOT EXISTS kine; CREATE USER 'kine'@'%' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS'; GRANT ALL PRIVILEGES ON kine.* TO 'kine'@'%'; FLUSH PRIVILEGES; ``` #### Create an IAM policy for RDS access[​](#create-an-iam-policy-for-rds-access "Direct link to Create an IAM policy for RDS access") Create an IAM policy that grants the `rds-db:connect` permission for the `kine` database user. Replace `` with the RDS instance resource ID (run `aws rds describe-db-instances --query 'DBInstances[0].DbiResourceId'` to retrieve it). Modify the following with your specific values to generate a copyable command: DBI\_RESOURCE\_IDdb-XXXXXXXXXXXXXXXXXXXXXXXXXX ``` aws iam create-policy \ --policy-name RDSIAMAuthKine \ --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "rds-db:connect", "Resource": "arn:aws:rds-db:us-east-1:123456789012:dbuser:db-XXXXXXXXXXXXXXXXXXXXXXXXXX/kine" } ] }' ``` #### Create an IAM role for Pod Identity[​](#create-an-iam-role-for-pod-identity "Direct link to Create an IAM role for Pod Identity") Create an IAM role with a trust policy that allows the EKS Pod Identity agent to assume it. The `pods.eks.amazonaws.com` trust principal is region-agnostic, so a single role works for clusters in any region. ``` aws iam create-role \ --role-name PlatformKineRDSRole \ --assume-role-policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "pods.eks.amazonaws.com" }, "Action": [ "sts:AssumeRole", "sts:TagSession" ] } ] }' aws iam attach-role-policy \ --role-name PlatformKineRDSRole \ --policy-arn arn:aws:iam::123456789012:policy/RDSIAMAuthKine ``` #### Install the EKS Pod Identity Agent[​](#install-the-eks-pod-identity-agent "Direct link to Install the EKS Pod Identity Agent") Install the Pod Identity Agent add-on on **every** cluster. Without this add-on, the Pod Identity association has no effect and pods can't obtain IAM credentials. ``` aws eks create-addon \ --cluster-name platform-multi-region-us-east-1 \ --addon-name eks-pod-identity-agent \ --region us-east-1 aws eks create-addon \ --cluster-name platform-multi-region-eu-west-1 \ --addon-name eks-pod-identity-agent \ --region eu-west-1 ``` Verify the add-on is `ACTIVE` on each cluster before proceeding: ``` aws eks describe-addon \ --cluster-name platform-multi-region-us-east-1 \ --addon-name eks-pod-identity-agent \ --region us-east-1 \ --query 'addon.status' ``` #### Create Pod Identity associations[​](#create-pod-identity-associations "Direct link to Create Pod Identity associations") Associate the IAM role with the `loft` service account in the `vcluster-platform` namespace on **every** cluster. This allows the platform pods to generate temporary IAM authentication tokens for the database connection. Important You must create a Pod Identity association on **each** cluster, not just the cluster in the same region as the database. Without the association, the platform pods in that cluster can't authenticate with the RDS instance and fail with `Access denied` errors. **U.S. region** ``` aws eks create-pod-identity-association \ --cluster-name platform-multi-region-us-east-1 \ --role-arn arn:aws:iam::123456789012:role/PlatformKineRDSRole \ --namespace vcluster-platform \ --service-account loft \ --region us-east-1 ``` **E.U. region** ``` aws eks create-pod-identity-association \ --cluster-name platform-multi-region-eu-west-1 \ --role-arn arn:aws:iam::123456789012:role/PlatformKineRDSRole \ --namespace vcluster-platform \ --service-account loft \ --region eu-west-1 ``` For more details on IAM database authentication with EKS Pod Identity, see [IAM database authentication](https://vcluster.com/docs/platform/administer/connector/iam_database.md). ## Step 4 - Create VPC peering to the database[​](#step-4---create-vpc-peering-to-the-database "Direct link to Step 4 - Create VPC peering to the database") Create VPC peering connections between each EKS cluster VPC and the database VPC. This gives both clusters network access to the RDS instance without requiring direct cluster-to-cluster peering. #### Create peering connections[​](#create-peering-connections "Direct link to Create peering connections") **Database VPC to U.S. cluster VPC** (same region): Modify the following with your specific values to generate a copyable command: VPC\_ID\_US\_CLUSTERvpc-yyyyyyyyy ``` aws ec2 create-vpc-peering-connection \ --vpc-id vpc-xxxxxxxxx \ --peer-vpc-id vpc-yyyyyyyyy \ --region us-east-1 \ --tag-specifications 'ResourceType=vpc-peering-connection,Tags=[{Key=Name,Value=db-vpc-to-eks-us-east-1}]' ``` **Database VPC to E.U. cluster VPC** (cross-region): Modify the following with your specific values to generate a copyable command: VPC\_ID\_EU\_CLUSTERvpc-zzzzzzzzz ``` aws ec2 create-vpc-peering-connection \ --vpc-id vpc-xxxxxxxxx \ --peer-vpc-id vpc-zzzzzzzzz \ --peer-region eu-west-1 \ --region us-east-1 \ --tag-specifications 'ResourceType=vpc-peering-connection,Tags=[{Key=Name,Value=db-vpc-to-eks-eu-west-1}]' ``` #### Accept the peering connections[​](#accept-the-peering-connections "Direct link to Accept the peering connections") The same-region peering is accepted in `us-east-1`. The cross-region peering must be accepted in `eu-west-1`. ``` aws ec2 accept-vpc-peering-connection \ --vpc-peering-connection-id pcx-xxxxxxxxx \ --region us-east-1 aws ec2 accept-vpc-peering-connection \ --vpc-peering-connection-id pcx-yyyyyyyyy \ --region eu-west-1 ``` #### Enable DNS resolution[​](#enable-dns-resolution "Direct link to Enable DNS resolution") Enable DNS resolution on both peering connections so the RDS endpoint hostname resolves to its private IP from the EKS VPCs. ``` aws ec2 modify-vpc-peering-connection-options \ --vpc-peering-connection-id pcx-xxxxxxxxx \ --requester-peering-connection-options AllowDnsResolutionFromRemoteVpc=true \ --accepter-peering-connection-options AllowDnsResolutionFromRemoteVpc=true \ --region us-east-1 ``` For the cross-region peering, set the requester side (DB VPC) in `us-east-1` and the accepter side (E.U. cluster VPC) in `eu-west-1`: ``` aws ec2 modify-vpc-peering-connection-options \ --vpc-peering-connection-id pcx-yyyyyyyyy \ --requester-peering-connection-options AllowDnsResolutionFromRemoteVpc=true \ --region us-east-1 aws ec2 modify-vpc-peering-connection-options \ --vpc-peering-connection-id pcx-yyyyyyyyy \ --accepter-peering-connection-options AllowDnsResolutionFromRemoteVpc=true \ --region eu-west-1 ``` #### Add routes[​](#add-routes "Direct link to Add routes") Add routes so traffic can flow between each EKS VPC and the database VPC. Important Add routes to **every** route table whose subnets host EKS nodes, including public route tables. If your EKS nodes run in public subnets (the `eksctl` default), missing routes on the public route table cause database connection timeouts even though VPC peering is active. **Database VPC route table** — route to both EKS VPC CIDRs: ``` aws ec2 create-route \ --route-table-id rtb-xxxxxxxxx \ --destination-cidr-block 10.0.0.0/16 \ --vpc-peering-connection-id pcx-xxxxxxxxx \ --region us-east-1 aws ec2 create-route \ --route-table-id rtb-xxxxxxxxx \ --destination-cidr-block 172.21.0.0/16 \ --vpc-peering-connection-id pcx-yyyyyyyyy \ --region us-east-1 ``` **U.S. EKS VPC route tables** — route to the database VPC CIDR. Repeat for every route table associated with subnets that host EKS nodes: Modify the following with your specific values to generate a copyable command: US\_EKS\_ROUTE\_TABLE\_IDrtb-yyyyyyyyy ``` aws ec2 create-route \ --route-table-id rtb-yyyyyyyyy \ --destination-cidr-block 10.1.0.0/16 \ --vpc-peering-connection-id pcx-xxxxxxxxx \ --region us-east-1 ``` **E.U. EKS VPC route tables** — route to the database VPC CIDR. Repeat for every route table associated with subnets that host EKS nodes: Modify the following with your specific values to generate a copyable command: EU\_EKS\_ROUTE\_TABLE\_IDrtb-zzzzzzzzz ``` aws ec2 create-route \ --route-table-id rtb-zzzzzzzzz \ --destination-cidr-block 10.1.0.0/16 \ --vpc-peering-connection-id pcx-yyyyyyyyy \ --region eu-west-1 ``` To list all route tables for a VPC and identify which subnets they serve: Modify the following with your specific values to generate a copyable command: VPC\_IDvpc-xxxxxxxxx ``` aws ec2 describe-route-tables \ --filters "Name=vpc-id,Values=vpc-xxxxxxxxx" \ --region us-east-1 \ --query 'RouteTables[].{RouteTableId:RouteTableId,Name:Tags[?Key==`Name`].Value|[0],Subnets:Associations[].SubnetId}' ``` #### Verify connectivity[​](#verify-connectivity "Direct link to Verify connectivity") Launch a test pod in each EKS cluster and verify it can reach the RDS endpoint: Modify the following with your specific values to generate a copyable command: RDS\_ENDPOINTmariadb-multi-region.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com ``` kubectl run dbtest --image=busybox --restart=Never --rm -it -- \ nc -zv mariadb-multi-region.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com 3306 ``` The connection should report `open`. If it times out, verify that routes exist on the correct route tables and that the database security group allows ingress from the cluster's VPC CIDR. ## Step 5 - Deploy vCluster platform[​](#step-5---deploy-vcluster-platform "Direct link to Step 5 - Deploy vCluster platform") Deploy the platform on each cluster using the `multiRegion` Helm values. Each cluster needs a **region-specific** `multiRegion.host` value (used as the custom DERP server endpoint), while `config.loftHost` should be the **shared DNS domain**. Create a values file for each region. The only differences between regions are `multiRegion.host` and `multiRegion.region`. **U.S. region** (save as `US_VALUES_FILE`) ``` admin: email: admin@example.com multiRegion: enabled: true host: us.multi-region.example.com region: us-east-1 replicaCount: 3 config: loftHost: multi-region.example.com database: enabled: true dataSource: "mysql://kine@tcp(mariadb-multi-region.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com:3306)/kine" identityProvider: "aws" extraArgs: - --datastore-max-open-connections=20 - --datastore-max-idle-connections=0 # Cost control requires a single-region database and is not compatible # with the shared Kine backend used in multi-region deployments. costControl: enabled: false agentValues: replicaCount: 3 ``` **E.U. region** (save as `EU_VALUES_FILE`) ``` admin: email: admin@example.com multiRegion: enabled: true host: eu.multi-region.example.com region: eu-west-1 replicaCount: 3 config: loftHost: multi-region.example.com database: enabled: true dataSource: "mysql://kine@tcp(mariadb-multi-region.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com:3306)/kine" identityProvider: "aws" extraArgs: - --datastore-max-open-connections=20 - --datastore-max-idle-connections=0 # Cost control requires a single-region database and is not compatible # with the shared Kine backend used in multi-region deployments. costControl: enabled: false agentValues: replicaCount: 3 ``` Install the platform on each cluster using its values file. * vCluster CLI * Helm **U.S. region** Modify the following with your specific values to generate a copyable command: US\_REGION\_CONTEXTarn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 ``` vcluster platform start \ --namespace vcluster-platform \ --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ --values platform-us-east-1-values.yaml \ --no-tunnel ``` **E.U. region** Modify the following with your specific values to generate a copyable command: EU\_REGION\_CONTEXTarn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1 ``` vcluster platform start \ --namespace vcluster-platform \ --context arn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1 \ --values platform-eu-west-1-values.yaml \ --no-tunnel ``` **U.S. region** ``` helm upgrade loft vcluster-platform --install --create-namespace --repository-config='' \ --namespace vcluster-platform \ --repo "https://charts.loft.sh/" \ -f platform-us-east-1-values.yaml ``` **E.U. region** ``` helm upgrade loft vcluster-platform --install --create-namespace --repository-config='' \ --namespace vcluster-platform \ --repo "https://charts.loft.sh/" \ -f platform-eu-west-1-values.yaml ``` Setting `multiRegion.enabled=true` automatically configures: * Embedded Kubernetes (Kine) with the shared database * Leader election across regions * Multi-region platform license check ## Step 6 - Configure HTTPS with AWS Certificate Manager (ACM)[​](#step-6---configure-https-with-aws-certificate-manager-acm "Direct link to Step 6 - Configure HTTPS with AWS Certificate Manager (ACM)") Request a certificate in each region that covers both the shared DNS domain and the region-specific subdomain. The wildcard `*.multi-region.example.com` covers the regional subdomains (`us.multi-region.example.com`, `eu.multi-region.example.com`), and `multi-region.example.com` is added as a SAN to cover the apex. Run this command in each region. The command uses the `AWS_REGION` page variable — run it a second time after setting `EU_REGION` as the region value to request the certificate there too. ``` aws acm request-certificate \ --region us-east-1 \ --domain-name "multi-region.example.com" \ --subject-alternative-names "*.multi-region.example.com" \ --validation-method DNS ``` Get the certificate ARN from the output and describe the certificate. Replace `` with the ARN ID generated from the command above. Modify the following with your specific values to generate a copyable command: CERTIFICATE\_ARN\_IDxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ``` aws acm describe-certificate \ --region us-east-1 \ --certificate-arn arn:aws:acm:us-east-1:123456789012:certificate/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ``` Add DNS validation CNAMEs in Route 53 and wait until certificate status is ISSUED. note You can also do this step by setting up cert-manager in the clusters instead of using AWS ACM. ## Step 7 - Create Ingress (ALB)[​](#step-7---create-ingress-alb "Direct link to Step 7 - Create Ingress (ALB)") Apply the Ingress manifest in each cluster to provision an ALB and route traffic to the loft service. Modify the following with your specific values to generate a copyable command: CERT\_IDxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ``` apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: loft namespace: vcluster-platform annotations: alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]' alb.ingress.kubernetes.io/certificate-arn: "arn:aws:acm:us-east-1:123456789012:certificate/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" alb.ingress.kubernetes.io/ssl-redirect: "443" alb.ingress.kubernetes.io/idle-timeout: "3600" alb.ingress.kubernetes.io/healthcheck-path: /healthz alb.ingress.kubernetes.io/success-codes: "200" spec: ingressClassName: alb rules: - host: us.multi-region.example.com http: paths: - path: / pathType: Prefix backend: service: name: loft port: number: 80 - host: multi-region.example.com http: paths: - path: / pathType: Prefix backend: service: name: loft port: number: 80 ``` ## Step 8 - Configure Route 53 latency-based routing[​](#step-8---configure-route-53-latency-based-routing "Direct link to Step 8 - Configure Route 53 latency-based routing") #### Create health checks[​](#create-health-checks "Direct link to Create health checks") Create an HTTPS health check for each regional ALB. Route 53 uses these to detect when a region is unavailable and fail over traffic automatically. **U.S. region** ``` aws route53 create-health-check \ --caller-reference "us-east-1-platform-healthz" \ --health-check-config '{ "FullyQualifiedDomainName": "us.multi-region.example.com", "Port": 443, "Type": "HTTPS", "ResourcePath": "/healthz", "RequestInterval": 10, "FailureThreshold": 3, "EnableSNI": true }' ``` **E.U. region** ``` aws route53 create-health-check \ --caller-reference "eu-west-1-platform-healthz" \ --health-check-config '{ "FullyQualifiedDomainName": "eu.multi-region.example.com", "Port": 443, "Type": "HTTPS", "ResourcePath": "/healthz", "RequestInterval": 10, "FailureThreshold": 3, "EnableSNI": true }' ``` Note the health check IDs from the output. You can tag them for easier identification: ``` aws route53 change-tags-for-resource \ --resource-type healthcheck --resource-id xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \ --add-tags Key=Name,Value=us-platform-healthz ``` #### Create DNS records[​](#create-dns-records "Direct link to Create DNS records") Create latency-based A (Alias) records in Route 53 pointing to each regional ALB. Attach the health checks created above so that Route 53 can fail over traffic when a region becomes unavailable. Replace `` and `` with the IDs from the previous step, and ``, `` with the canonical hosted zone IDs of each ALB. Modify the following with your specific values to generate a copyable command: US\_HEALTH\_CHECK\_IDxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx EU\_HEALTH\_CHECK\_IDyyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy ``` aws route53 change-resource-record-sets \ --hosted-zone-id Z1234567890ABC \ --change-batch '{ "Changes": [ { "Action": "UPSERT", "ResourceRecordSet": { "Name": "multi-region.example.com", "Type": "A", "SetIdentifier": "us-east-1", "Region": "us-east-1", "HealthCheckId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "AliasTarget": { "HostedZoneId": "ZXXXXXXXXUS", "DNSName": "k8s-vcluster-loft-xxxxxxxxxx-xxxxxxxxx.us-east-1.elb.amazonaws.com", "EvaluateTargetHealth": true } } }, { "Action": "UPSERT", "ResourceRecordSet": { "Name": "multi-region.example.com", "Type": "A", "SetIdentifier": "eu-west-1", "Region": "eu-west-1", "HealthCheckId": "yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy", "AliasTarget": { "HostedZoneId": "ZXXXXXXXXEU", "DNSName": "k8s-vcluster-loft-xxxxxxxxxx-xxxxxxxxx.eu-west-1.elb.amazonaws.com", "EvaluateTargetHealth": true } } } ] }' ``` Also create A records for the region-specific subdomains used by the custom DERP server and health checks. Replace ``, `` with the canonical hosted zone IDs and ``, `` with the DNS names of each ALB. ``` aws route53 change-resource-record-sets \ --hosted-zone-id Z1234567890ABC \ --change-batch '{ "Changes": [ { "Action": "UPSERT", "ResourceRecordSet": { "Name": "us.multi-region.example.com", "Type": "A", "AliasTarget": { "HostedZoneId": "ZXXXXXXXXUS", "DNSName": "k8s-vcluster-loft-xxxxxxxxxx-xxxxxxxxx.us-east-1.elb.amazonaws.com", "EvaluateTargetHealth": true } } }, { "Action": "UPSERT", "ResourceRecordSet": { "Name": "eu.multi-region.example.com", "Type": "A", "AliasTarget": { "HostedZoneId": "ZXXXXXXXXEU", "DNSName": "k8s-vcluster-loft-xxxxxxxxxx-xxxxxxxxx.eu-west-1.elb.amazonaws.com", "EvaluateTargetHealth": true } } } ] }' ``` #### Verify health checks[​](#verify-health-checks "Direct link to Verify health checks") Confirm both health checks are healthy before proceeding: ``` aws route53 get-health-check-status --health-check-id xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \ --query 'HealthCheckObservations[].{Region:Region,Status:StatusReport.Status}' \ --output table ``` Important Without health checks attached to the latency-based records, Route 53 can't detect regional failures and continues routing traffic to an unavailable region. Always verify that health checks are healthy before proceeding. ## Step 9 - Deploy the agent on the clusters[​](#step-9---deploy-the-agent-on-the-clusters "Direct link to Step 9 - Deploy the agent on the clusters") The agent must be deployed in a **different namespace** than the platform (for example, `vcluster-agent`). The `agentValues` block in the platform values files (Step 5) controls the agent's replica count, leader election, and other settings. * vCluster CLI * Helm Use the vCluster CLI to add each cluster as a connected cluster. Generate an access token from the platform UI under **Users & Teams > Access Keys**, or by running `vcluster platform get token`. Modify the following with your specific values to generate a copyable command: DISPLAY\_NAMEmy-cluster ``` vcluster platform add cluster platform-multi-region-us-east-1 \ --namespace vcluster-agent \ --display-name my-cluster ``` The CLI fetches the agent values for the cluster from the platform itself, so you don't pass a values URL. Repeat on the second cluster, substituting `EU_CLUSTER_NAME` for the cluster name and providing a new `DISPLAY_NAME`. Go to the platform UI and under Infrastructure > Control Plane Clusters, add a connected cluster and select the Helm (advanced) option. You get a Helm command with a `--values` URL that contains agent-specific configuration. ``` helm upgrade loft vcluster-platform --install --create-namespace --repository-config='' \ --namespace vcluster-agent \ --repo "https://charts.loft.sh/" \ --set agentOnly=true \ --set url=multi-region.example.com \ --set token=your-access-token \ --values 'https://multi-region.example.com/clusters/agent-values/platform-multi-region-us-east-1?token=your-access-token' ``` Repeat on the second cluster, substituting `EU_CLUSTER_NAME` for the cluster name. ### Result[​](#result "Direct link to Result") You now have a fully operational **multi-region vCluster Platform deployment** with: * Two EKS clusters (U.S. + E.U.) * Shared Kine database in an isolated VPC with IAM-based authentication (Pod Identity) * VPC peering between each cluster VPC and the database VPC * Regional ALBs with extended idle timeout * ACM wildcard certificates * Route 53 latency-based routing with health checks for automatic failover * Platform deployed on both clusters sharing the same Kine database --- # Overview | | | Enterprise | | | | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Multi-Region Mode](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md) | | | | | | [Multi-Region Platform](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). info This feature is available from the **Platform** version v4.8.0 ## What is multi-region platform?[​](#what-is-multi-region-platform "Direct link to What is multi-region platform?") A multi-region platform deployment runs vCluster Platform instances in two or more regions, all backed by a single shared database. A leader election mechanism ensures that only one platform instance writes to the shared database at a time. A custom DERP server provides encrypted relay connectivity between regions. Health-checking DNS configuration ensures that failover occurs seamlessly during a regional outage and routes clients to the lowest-latency region. Multi-region platform architecture with shared Kine database, latency-based routing, and DERP relay ## How it works[​](#how-it-works "Direct link to How it works") Two mechanisms keep the platform available and consistent across regions: health-checked DNS routing handles failover, and leader election coordinates database writes. Health-checked DNS routing and leader election for write coordination ### Failover[​](#failover "Direct link to Failover") Route 53 runs health checks against each region's ALB every 10 seconds. After three consecutive failures, it removes that region from the routing pool. Traffic shifts automatically to the remaining healthy region. When the failed region recovers and its health checks pass, Route 53 reinstates it. No configuration changes are required. ### Write coordination[​](#write-coordination "Direct link to Write coordination") The two embedded k8s API servers compete for a write lease stored in the shared RDS database. The region that holds the lease is the leader and processes all writes. The other region is the follower. It serves reads and forwards writes to the leader. If the leader fails, the follower acquires the lease and becomes the new leader. The leader role is dynamic. Either region can hold it at any time. All writes go through the leader to a single database, so the follower incurs cross-region latency for write operations. Read-heavy workloads are less affected. ## Why deploy multi-region?[​](#why-deploy-multi-region "Direct link to Why deploy multi-region?") A single-region platform works for most deployments. Consider multi-region when you need one or more of the following: * **Regional failover**: If a region goes down, Route 53 health checks detect the failure and automatically redirect traffic to a healthy region. The shared database ensures no state is lost during failover. * **High availability for the control plane**: Running platform replicas across regions eliminates the platform as a single point of failure. Connected clusters continue operating through the surviving region. ### How it differs from other deployment modes[​](#how-it-differs-from-other-deployment-modes "Direct link to How it differs from other deployment modes") | | Multi-region platform | [Regional Cluster Endpoints](https://vcluster.com/docs/platform/administer/clusters/advanced/regional-cluster-endpoints.md) | [Platform External Database](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/.md) | | ---------------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | **What is replicated** | The platform itself (full replicas in each region) | Only the agent endpoints (platform stays in one region) | Multiple platform replicas in a single cluster | | **Shared database** | Yes — all regions share a single Kine-backed database | No — single platform database | Yes — all replicas share a single Kine-backed database | | **Failover** | Automatic through DNS health checks | No platform failover | Automatic through leader election within the cluster | | **Use case** | Platform HA, low-latency platform API access | Low-latency `kubectl` access to clusters | Platform HA within a single region | Both features can be used together: multi-region platform provides platform-level HA, while Regional Cluster Endpoints provide low-latency `kubectl` access to workloads. ### Trade-offs[​](#trade-offs "Direct link to Trade-offs") * **Operational complexity**: Multi-region requires managing VPC peering, cross-region networking, shared database infrastructure, and coordinated upgrades. * **Database latency**: The non-leader region incurs cross-region latency for database writes, since all writes go through the leader to a single database. Read-heavy workloads are less affected. * **Cost control unavailable**: The cost control feature requires a single-region database and isn't compatible with the shared Kine backend. * **Fresh install only**: Converting an existing single-region installation to multi-region isn't supported. Related docs For routing `kubectl` traffic directly to clusters without replicating the platform, see [Regional Cluster Endpoints](https://vcluster.com/docs/platform/administer/clusters/advanced/regional-cluster-endpoints.md). For details on how DERP relays provide cross-region connectivity, see [DERP relay](https://vcluster.com/docs/platform/administer/clusters/advanced/derp-relay.md). Important Converting an existing single-region platform installation to multi-region isn't supported. Multi-region must be configured as a fresh installation. See [Deploy (AWS/EKS)](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/deploy.md) for step-by-step setup instructions. ## Access the management API[​](#access-the-management-api "Direct link to Access the management API") Multi-region platforms run an embedded Kubernetes API server inside each region's vCluster Platform pod. The `v1.management.loft.sh` aggregated APIService isn't registered on the control plane EKS cluster. Each region registers it inside its own embedded API server instead. This changes how automation and integrations such as Argo CD or Terraform call the management API. The aggregated APIService is cluster-scoped, so a single control plane EKS cluster can register only one. With multiple region pods sharing the same control plane cluster, that one registration can't route correctly to all of them. Each region therefore registers the APIService locally, and the platform's own HTTPS endpoint is the integration surface. ### Compare standard and multi-region behavior[​](#compare-standard-and-multi-region-behavior "Direct link to Compare standard and multi-region behavior") | Aspect | Standard platform | Multi-region platform | | --------------------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | APIService location | Aggregated on the host EKS cluster | Local, inside each region's embedded API server | | Authentication | Host cluster bearer token (for example, an EKS IAM token) | [Platform access key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) | | APIService visibility on host EKS | Shows the registration | Shows nothing | The endpoint shape differs by deployment type: * **Standard platform**: `${EKS_ENDPOINT}/apis/management.loft.sh/v1/` * **Multi-region platform**: `https:///kubernetes/management/apis/management.loft.sh/v1/` ### Call the management API[​](#call-the-management-api "Direct link to Call the management API") Build requests against: ``` https:///kubernetes/management/apis/management.loft.sh/v1/ ``` Authenticate with a platform access key as a bearer token. For instructions on [creating an access key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md#create-an-access-key), scope it to the project, user, or tenant cluster the caller needs. #### Use kubectl[​](#use-kubectl "Direct link to Use kubectl") Modify the following with your specific values to generate a copyable command: PLATFORM\_HOSTplatform.example.com ``` kubectl --server https://platform.example.com/kubernetes/management \ --token "$ACCESS_KEY" \ get virtualclusterinstances -A ``` #### Use curl[​](#use-curl "Direct link to Use curl") Modify the following with your specific values to generate a copyable command: PLATFORM\_HOSTplatform.example.com ``` curl -H "Authorization: Bearer $ACCESS_KEY" \ https://platform.example.com/kubernetes/management/apis/management.loft.sh/v1/projects ``` #### Register an Argo CD cluster[​](#register-an-argo-cd-cluster "Direct link to Register an Argo CD cluster") Register the platform as an Argo CD cluster by creating a `cluster`-type secret in the Argo CD namespace. Argo CD treats `server` as the API endpoint and authenticates with `bearerToken`: Modify the following with your specific values to generate a copyable command: REGIONregion-a PLATFORM\_HOSTplatform.example.com ACCESS\_KEY\ ``` apiVersion: v1 kind: Secret metadata: name: platform-region-a namespace: argocd labels: argocd.argoproj.io/secret-type: cluster stringData: name: platform-region-a server: https://platform.example.com/kubernetes/management config: | { "bearerToken": "", "tlsClientConfig": { "insecure": false } } ``` For the wider Argo CD integration (project import, SSO, AppProject sync), see [Argo CD integration](https://vcluster.com/docs/platform/integrations/argocd.md). ### Upgrade from 4.7.x to 4.8.0[​](#upgrade-from-47x-to-480 "Direct link to Upgrade from 4.7.x to 4.8.0") This routing change shipped in platform version 4.8.0 and applies to multi-region deployments only. Automation that called the management API using the host EKS endpoint stops working after the upgrade. There's no in-place compatibility shim. Update callers to use the platform HTTPS endpoint. Switch authentication from your host-cluster bearer token (for example, an EKS IAM token) to a platform access key. Standard (non-multi-region) platform deployments are unaffected. The host EKS APIService keeps working as before. ### Troubleshoot common errors[​](#troubleshoot-common-errors "Direct link to Troubleshoot common errors") | Symptom | Cause | Resolution | | -------------------------------------------------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | `404` from the host EKS endpoint | Caller is using the host EKS endpoint on a multi-region platform | Switch to the platform HTTPS endpoint shown in [Call the management API](#call-the-management-api) | | The host EKS cluster has no `v1.management.loft.sh` APIService | Expected on multi-region, as the APIService lives inside the embedded API server | None; verify from inside the embedded API server if needed | | `401` or `403` from the platform endpoint | Access key is invalid, expired, or scoped without permission for the requested resource | Regenerate the key or widen its scope; check the owning user's role bindings | | TLS verification errors against the platform endpoint | Caller doesn't trust the platform's certificate chain | Configure the same CA bundle the platform UI uses; avoid `insecure=true` in production | --- # Recover from Regional Failover info This feature is available from the **Platform** version v4.8.0 When a region goes down, Route 53 health checks detect the failure and automatically redirect traffic to the healthy region. This runbook describes how to diagnose the failed region, restore it, and confirm that both regions are serving traffic again. ## Configure your values[​](#configure-your-values "Direct link to Configure your values") The commands in this runbook reference your cluster context ARNs, the failed region's values file, and its health check ID. Set them below once and all commands update automatically. Expand to set page variables Modify the following with your specific values to replace on the whole page and generate copyable commands: FIRST\_REGION\_CONTEXTarn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 SECOND\_REGION\_CONTEXTarn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1 FAILED\_REGION\_CONTEXTarn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 FAILED\_REGION\_VALUES\_FILEplatform-us-east-1-values.yaml FAILED\_REGION\_HEALTH\_CHECK\_IDxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ## Step 1 - Confirm failover is active[​](#step-1---confirm-failover-is-active "Direct link to Step 1 - Confirm failover is active") Verify which health check is failing: ``` aws route53 get-health-check-status --health-check-id xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \ --query 'HealthCheckObservations[].{Region:Region,Status:StatusReport.Status}' \ --output table ``` Confirm the healthy region is still passing: Modify the following with your specific values to generate a copyable command: HEALTHY\_REGION\_HEALTH\_CHECK\_IDyyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy ``` aws route53 get-health-check-status --health-check-id yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy \ --query 'HealthCheckObservations[].{Region:Region,Status:StatusReport.Status}' \ --output table ``` note During failover, the platform continues to operate normally through the healthy region. Users may experience slightly higher latency if they are geographically closer to the failed region. ## Step 2 - Diagnose the failed region[​](#step-2---diagnose-the-failed-region "Direct link to Step 2 - Diagnose the failed region") Check the state of the platform pods in the failed region: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ get pods -n vcluster-platform -l app=loft ``` Check pod logs for errors: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ logs -n vcluster-platform -l app=loft --tail=50 ``` Check the ALB and ingress status: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ get ingress -n vcluster-platform loft ``` Common failure causes: * **Pod crashes (CrashLoopBackOff)**: Check logs for database connectivity errors or resource exhaustion. * **Node failures**: Check `kubectl get nodes` for `NotReady` nodes. * **ALB unhealthy**: Verify the ALB target group health in the AWS console or with `aws elbv2 describe-target-health`. * **Network partition**: Verify that VPC peering between the cluster VPC and the database VPC is active, that routes exist on the correct route tables (including public route tables if nodes are in public subnets), and that the database security group allows port 3306 from the cluster VPC CIDR. ## Step 3 - Restore the failed region[​](#step-3---restore-the-failed-region "Direct link to Step 3 - Restore the failed region") The recovery steps depend on the failure cause. **If pods are crashlooping or unhealthy**, restart the deployment: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ rollout restart deployment/loft -n vcluster-platform ``` **If the deployment is scaled to zero or missing**, re-apply the values file: * vCluster CLI * Helm ``` vcluster platform start \ --namespace vcluster-platform \ --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ --values platform-us-east-1-values.yaml \ --upgrade \ --no-tunnel ``` Modify the following with your specific values to generate a copyable command: CHART\_VERSION4.12.0 ``` helm upgrade loft vcluster-platform --install --create-namespace --repository-config='' \ --namespace vcluster-platform \ --repo "https://charts.loft.sh/" \ --version 4.12.0 \ --kube-context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ -f platform-us-east-1-values.yaml ``` Re-applying the values file restores `replicaCount` from the file, so no separate scale-up is needed. **If the EKS cluster itself is down**, follow the AWS documentation to restore the cluster or create a replacement, then redeploy the platform using the region's values file. ## Step 4 - Wait for the rollout to complete[​](#step-4---wait-for-the-rollout-to-complete "Direct link to Step 4 - Wait for the rollout to complete") ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ rollout status deployment/loft -n vcluster-platform ``` ## Step 5 - Verify recovery[​](#step-5---verify-recovery "Direct link to Step 5 - Verify recovery") Confirm the Route 53 health check for the restored region returns healthy: ``` aws route53 get-health-check-status --health-check-id xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \ --query 'HealthCheckObservations[].{Region:Region,Status:StatusReport.Status}' \ --output table ``` note Route 53 health checks run every 10 seconds with a failure threshold of 3. After the platform becomes healthy, expect up to 30 seconds before the health check status updates. DNS clients with cached responses may take an additional 60 seconds (the ALB alias record TTL) to start resolving to the restored region. Verify both regions are serving traffic: ``` for CTX in arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 arn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1; do echo "=== $CTX ===" kubectl --context "$CTX" get pods -n vcluster-platform -l app=loft echo done ``` Confirm the custom DERP server is operational by checking for `NetworkPeer` resources: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ get networkpeers.storage.loft.sh -A ``` Both peers should show `Online: true`. --- # Restore the Kine Database from a Snapshot info This feature is available from the **Platform** version v4.8.0 This runbook covers restoring the shared Kine database from an RDS snapshot to a new instance and updating both platform regions to use it. Use this procedure for disaster recovery, database migration (for example, enabling IAM authentication), or point-in-time recovery. Important Both platform regions must be scaled down before switching the data source to prevent split-brain writes to the old and new databases. ## Configure your values[​](#configure-your-values "Direct link to Configure your values") This runbook references AWS resource IDs, cluster context ARNs, and file names specific to your deployment. Set them below once and all commands update automatically. Expand to set page variables Modify the following with your specific values to replace on the whole page and generate copyable commands: ACCOUNT\_ID123456789012 AWS\_REGIONus-east-1 DB\_SG\_IDsg-xxxxxxxxx FIRST\_REGION\_CONTEXTarn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 SECOND\_REGION\_CONTEXTarn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1 SNAPSHOT\_NAMEkine-backup-YYYY-MM-DD NEW\_DB\_INSTANCE\_IDmariadb-multi-region-restored FIRST\_REGION\_VALUES\_FILEplatform-us-east-1-values.yaml SECOND\_REGION\_VALUES\_FILEplatform-eu-west-1-values.yaml ## Step 1 - Create a snapshot of the current database[​](#step-1---create-a-snapshot-of-the-current-database "Direct link to Step 1 - Create a snapshot of the current database") Skip this step if you already have a snapshot to restore from. Modify the following with your specific values to generate a copyable command: CURRENT\_DB\_INSTANCE\_IDmariadb-multi-region ``` aws rds create-db-snapshot \ --db-instance-identifier mariadb-multi-region \ --db-snapshot-identifier kine-backup-YYYY-MM-DD \ --region us-east-1 ``` Wait for the snapshot to become available: ``` aws rds wait db-snapshot-available \ --db-snapshot-identifier kine-backup-YYYY-MM-DD \ --region us-east-1 ``` ## Step 2 - Restore the snapshot to a new RDS instance[​](#step-2---restore-the-snapshot-to-a-new-rds-instance "Direct link to Step 2 - Restore the snapshot to a new RDS instance") Restore the snapshot to a new instance in the database VPC. Use the same DB subnet group and security group from the original setup. Include `--enable-iam-database-authentication` if the new instance should use IAM auth. ``` aws rds restore-db-instance-from-db-snapshot \ --db-instance-identifier mariadb-multi-region-restored \ --db-snapshot-identifier kine-backup-YYYY-MM-DD \ --db-instance-class db.t3.medium \ --db-subnet-group-name multi-region-db-subnet \ --vpc-security-group-ids sg-xxxxxxxxx \ --no-publicly-accessible \ --enable-iam-database-authentication \ --region us-east-1 ``` Wait for the new instance to become available: ``` aws rds wait db-instance-available \ --db-instance-identifier mariadb-multi-region-restored \ --region us-east-1 ``` Note the new endpoint: ``` aws rds describe-db-instances \ --db-instance-identifier mariadb-multi-region-restored \ --query 'DBInstances[0].Endpoint.Address' \ --output text \ --region us-east-1 ``` If using IAM authentication, note the `DbiResourceId` of the new instance and update the `RDSIAMAuthKine` IAM policy to include it. Without this, platform pods fail with `Access denied for user 'kine'` errors because the IAM `rds-db:connect` permission is scoped to a specific RDS instance resource ID. ``` aws rds describe-db-instances \ --db-instance-identifier mariadb-multi-region-restored \ --query 'DBInstances[0].DbiResourceId' \ --output text \ --region us-east-1 ``` Add the new resource ID to the policy's `Resource` array: Modify the following with your specific values to generate a copyable command: OLD\_DBI\_RESOURCE\_IDdb-OLDXXXXXXXXXXXXXXXXXXXXXXXXXX NEW\_DBI\_RESOURCE\_IDdb-NEWXXXXXXXXXXXXXXXXXXXXXXXXXX ``` aws iam create-policy-version \ --policy-arn arn:aws:iam::123456789012:policy/RDSIAMAuthKine \ --set-as-default \ --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "rds-db:connect", "Resource": [ "arn:aws:rds-db:us-east-1:123456789012:dbuser:db-OLDXXXXXXXXXXXXXXXXXXXXXXXXXX/kine", "arn:aws:rds-db:us-east-1:123456789012:dbuser:db-NEWXXXXXXXXXXXXXXXXXXXXXXXXXX/kine" ] } ] }' ``` ## Step 3 - Scale down both regions[​](#step-3---scale-down-both-regions "Direct link to Step 3 - Scale down both regions") Scale both platform deployments to zero to stop all writes to the old database. ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ scale deployment -n vcluster-platform loft --replicas=0 kubectl --context arn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1 \ scale deployment -n vcluster-platform loft --replicas=0 ``` Wait for all pods to stop: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ get pods -n vcluster-platform -l app=loft --watch kubectl --context arn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1 \ get pods -n vcluster-platform -l app=loft --watch ``` ## Step 4 - Update the values files[​](#step-4---update-the-values-files "Direct link to Step 4 - Update the values files") Update the `dataSource` in both region values files to point to the new RDS endpoint: Modify the following with your specific values to generate a copyable command: NEW\_DATABASE\_URLmariadb-multi-region-restored.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com ``` config: database: dataSource: "mysql://kine@tcp(mariadb-multi-region-restored.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com:3306)/kine" ``` ## Step 5 - Upgrade both regions[​](#step-5---upgrade-both-regions "Direct link to Step 5 - Upgrade both regions") Apply the updated values files to both regions. The upgrade reapplies `replicaCount` from the values files, so this also brings both regions back up, now pointing at the restored database. * vCluster CLI * Helm ``` vcluster platform start \ --namespace vcluster-platform \ --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ --values platform-us-east-1-values.yaml \ --upgrade \ --no-tunnel vcluster platform start \ --namespace vcluster-platform \ --context arn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1 \ --values platform-eu-west-1-values.yaml \ --upgrade \ --no-tunnel ``` Modify the following with your specific values to generate a copyable command: CHART\_VERSION4.12.0 ``` helm upgrade loft vcluster-platform --install --create-namespace --repository-config='' \ --namespace vcluster-platform \ --repo "https://charts.loft.sh/" \ --version 4.12.0 \ --kube-context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ -f platform-us-east-1-values.yaml helm upgrade loft vcluster-platform --install --create-namespace --repository-config='' \ --namespace vcluster-platform \ --repo "https://charts.loft.sh/" \ --version 4.12.0 \ --kube-context arn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1 \ -f platform-eu-west-1-values.yaml ``` ## Step 6 - Wait for both regions to come back up[​](#step-6---wait-for-both-regions-to-come-back-up "Direct link to Step 6 - Wait for both regions to come back up") Wait for all pods to become ready: ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ rollout status deployment/loft -n vcluster-platform kubectl --context arn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1 \ rollout status deployment/loft -n vcluster-platform ``` ## Step 7 - Verify the restore[​](#step-7---verify-the-restore "Direct link to Step 7 - Verify the restore") Confirm the platform is healthy on both regions: ``` for CTX in arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 arn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1; do echo "=== $CTX ===" kubectl --context "$CTX" get pods -n vcluster-platform -l app=loft echo done ``` Verify the platform UI is accessible through the shared DNS domain and that both Route 53 health checks return healthy. --- # Upgrade a Multi-Region Platform Deployment info This feature is available from the **Platform** version v4.8.0 This runbook describes how to upgrade vCluster Platform across regions with minimal downtime. The process upgrades one region at a time while the other continues serving traffic through Route 53 failover. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Updated values files for each region (for example, `platform-us-east-1-values.yaml` and `platform-eu-west-1-values.yaml`) * `kubectl` contexts configured for both clusters * The target chart version available in the Helm repository ## Configure your values[​](#configure-your-values "Direct link to Configure your values") The commands in this runbook reference your cluster context ARNs. Set them below once and all commands update automatically. Expand to set page variables Modify the following with your specific values to replace on the whole page and generate copyable commands: FIRST\_REGION\_CONTEXTarn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 SECOND\_REGION\_CONTEXTarn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1 ## Step 1 - Scale down the first region[​](#step-1---scale-down-the-first-region "Direct link to Step 1 - Scale down the first region") Pick the region to upgrade first and scale its platform deployment to zero. Scaling down before upgrading prevents leader election from switching between different platform versions during the upgrade, which could cause unexpected behavior. Route 53 routes all traffic to the other region while this one is offline. Optionally, identify which region currently holds the leader lease so you can choose to upgrade the non-leader region first, minimizing leader re-elections: ``` for CTX in arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 arn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1; do echo "=== $CTX ===" kubectl --context "$CTX" get lease -n vcluster-platform -o wide echo done ``` ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ scale deployment -n vcluster-platform loft --replicas=0 ``` Wait for all pods to stop before proceeding. ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ get pods -n vcluster-platform -l app=loft --watch ``` ## Step 2 - Upgrade the first region[​](#step-2---upgrade-the-first-region "Direct link to Step 2 - Upgrade the first region") Upgrade the first region with the updated values file, overriding `replicaCount` to `0`. Helm reapplies `replicaCount` from your values on every upgrade. Without the override, this region would come back up on the new version while the other region still runs the old one. The override keeps it scaled down until Step 4. * vCluster CLI * Helm The CLI takes a single values file and has no `--set` flag. Copy the region's values file, set `replicaCount: 0` in the copy, and pass the copy: Modify the following with your specific values to generate a copyable command: FIRST\_REGION\_SCALED\_DOWN\_VALUES\_FILEplatform-us-east-1-values-scaled-down.yaml ``` vcluster platform start \ --namespace vcluster-platform \ --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ --values platform-us-east-1-values-scaled-down.yaml \ --upgrade \ --no-tunnel ``` Modify the following with your specific values to generate a copyable command: CHART\_VERSION4.12.0 FIRST\_REGION\_VALUES\_FILEplatform-us-east-1-values.yaml ``` helm upgrade loft vcluster-platform --install --create-namespace --repository-config='' \ --namespace vcluster-platform \ --repo "https://charts.loft.sh/" \ --version 4.12.0 \ --kube-context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ -f platform-us-east-1-values.yaml \ --set replicaCount=0 ``` ## Step 3 - Upgrade the second region (rolling)[​](#step-3---upgrade-the-second-region-rolling "Direct link to Step 3 - Upgrade the second region (rolling)") Upgrade the second region without scaling down. This performs a rolling update so the region continues serving traffic during the upgrade. * vCluster CLI * Helm Modify the following with your specific values to generate a copyable command: SECOND\_REGION\_VALUES\_FILEplatform-eu-west-1-values.yaml ``` vcluster platform start \ --namespace vcluster-platform \ --context arn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1 \ --values platform-eu-west-1-values.yaml \ --upgrade \ --no-tunnel ``` Modify the following with your specific values to generate a copyable command: CHART\_VERSION4.12.0 SECOND\_REGION\_VALUES\_FILEplatform-eu-west-1-values.yaml ``` helm upgrade loft vcluster-platform --install --create-namespace --repository-config='' \ --namespace vcluster-platform \ --repo "https://charts.loft.sh/" \ --version 4.12.0 \ --kube-context arn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1 \ -f platform-eu-west-1-values.yaml ``` Wait for the rollout to complete. ``` kubectl --context arn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1 \ rollout status deployment/loft -n vcluster-platform ``` ## Step 4 - Scale up the first region[​](#step-4---scale-up-the-first-region "Direct link to Step 4 - Scale up the first region") Bring the first region back online by running the upgrade again with its normal values file. This restores `replicaCount` in both the Helm release values and the deployment. You can use `kubectl scale` instead, but the Helm release values then still record `replicaCount: 0` until the next upgrade. * vCluster CLI * Helm Modify the following with your specific values to generate a copyable command: FIRST\_REGION\_VALUES\_FILEplatform-us-east-1-values.yaml ``` vcluster platform start \ --namespace vcluster-platform \ --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ --values platform-us-east-1-values.yaml \ --upgrade \ --no-tunnel ``` Modify the following with your specific values to generate a copyable command: CHART\_VERSION4.12.0 FIRST\_REGION\_VALUES\_FILEplatform-us-east-1-values.yaml ``` helm upgrade loft vcluster-platform --install --create-namespace --repository-config='' \ --namespace vcluster-platform \ --repo "https://charts.loft.sh/" \ --version 4.12.0 \ --kube-context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ -f platform-us-east-1-values.yaml ``` Wait for all pods to become ready. ``` kubectl --context arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 \ rollout status deployment/loft -n vcluster-platform ``` ## Step 5 - Verify all deployments[​](#step-5---verify-all-deployments "Direct link to Step 5 - Verify all deployments") Confirm that both platform and agent deployments are running the expected image. ``` for CTX in arn:aws:eks:us-east-1:123456789012:cluster/platform-multi-region-us-east-1 arn:aws:eks:eu-west-1:123456789012:cluster/platform-multi-region-eu-west-1; do echo "=== $CTX ===" echo -n "Platform: " && kubectl --context "$CTX" get deployment -n vcluster-platform loft \ -o jsonpath='{.spec.template.spec.containers[0].image}' && echo echo -n "Agent: " && kubectl --context "$CTX" get deployment -n vcluster-agent loft \ -o jsonpath='{.spec.template.spec.containers[0].image}' && echo echo done ``` The platform automatically upgrades connected agents after the platform upgrade completes. --- # Networking The platform uses specific network ports to enable communication between Kubernetes clusters, the platform API, and external services. These ports support API requests, webhook execution, and cost monitoring. If these ports are blocked by firewall rules, some platform features such as role-based access control (RBAC) enforcement, cost monitoring, or webhook validation might fail. Ensure these ports are open to prevent connectivity issues. ## Required open ports[​](#required-open-ports "Direct link to Required open ports") ### Connect Kubernetes to the platform[​](#connect-kubernetes-to-the-platform "Direct link to Connect Kubernetes to the platform") Kubernetes must connect to the platform’s API extensions and webhooks to process requests, enforce policies, and perform management operations related to the platform. The following ports must be open to allow for communication: | Port | Description | Purpose | | ------ | -------------------------------------------- | ----------------------------------------------------------- | | `8080` | Legacy metrics endpoint | Enables scraping metrics when `serviceMonitor.enabled=true` | | `8443` | API service extension (`v1.cluster.loft.sh`) | Enables Kubernetes API integration with the platform | | `9443` | Webhook validation (`loft webhook`) | Validates requests and enforces policies | | `9444` | Management API (`v1.management.loft.sh`) | Handles platform-wide administration and cluster management | note In private GKE clusters, Kubernetes control plane and worker nodes might reside in separate subnetworks. You might need to explicitly allow traffic on these ports using a firewall rule. ### Connect vCluster to the platform[​](#connect-vcluster-to-the-platform "Direct link to Connect vCluster to the platform") When vCluster and the platform are in the same control plane cluster, the platform exposes an in‑cluster Service that targets port `10443` on the platform pods, so vCluster in the same control plane cluster talk to the platform via that `ClusterIP` Service. You should allow egress from the vCluster namespace to platform pods on TCP port `10443`. When vCluster and the platform are in different control plane clusters, vCluster uses the `loftHost` value (your platform's load balancer or Ingress DNS) to establish a Tailscale-backed tunnel for API traffic. You shoud allow egress traffic from vCluster pods to the `lostHost` domain and the egress traffic from platform agent to `loftHost`, if it participates in the handshake in your setup. ### Enable DNS resolution in tenant clusters[​](#enable-dns-resolution-in-tenant-clusters "Direct link to Enable DNS resolution in tenant clusters") vCluster runs a CoreDNS component inside each tenant cluster to handle internal DNS queries. To avoid conflicts with the control plane cluster's DNS, CoreDNS in vCluster listens on port `1053` instead of the default port `53`. If this port is blocked, DNS queries from tenant cluster pods might not work, especially when the querying pod and the CoreDNS pod are on different nodes. This issue commonly affects EKS clusters created with Terraform, which set up separate security groups for the control plane and worker nodes. By default, the node security group does not allow inbound traffic on port `1053`. To enable proper DNS resolution within tenant clusters, allow inbound traffic on port `1053` between nodes. | Port | Description | Purpose | | ------ | -------------------- | --------------------------------------------------------------- | | `1053` | CoreDNS for vCluster | Enables internal DNS resolution across nodes in tenant clusters | note If you're using EKS with Terraform, check the default node security group and manually allow inbound traffic on TCP and UDP port `1053`. This ensures DNS queries between pods and CoreDNS can succeed even when scheduled on different nodes. ### Connect the platform to Prometheus[​](#connect-the-platform-to-prometheus "Direct link to Connect the platform to Prometheus") The platform retrieves cost metrics from Prometheus running inside connected clusters. The platform's cost control dashboard uses these metrics to provide insights into resource usage. To collect this data, the local cluster agent acts as a proxy between the platform and Prometheus. This requires an open connection over `TCP port 9090`. If cost metrics are missing or unavailable, verify that the platform can reach Prometheus on this port. | Port | Description | Purpose | | ------ | ------------------------------ | ----------------------------------------- | | `9090` | Cost control dashboard metrics | Enables cost monitoring and visualization | note If your environment has strict firewall rules or network policies, you might need to explicitly allow traffic over `TCP 9090` between the platform and Prometheus. ## Firewall rules configuration[​](#firewall-rules-configuration "Direct link to Firewall rules configuration") If your firewall blocks ports, you must create rules to allow required traffic. You can configure access using network security settings such as VPC firewall rules or AWS Security Groups. ### Allow Kubernetes to connect to the platform[​](#allow-kubernetes-to-connect-to-the-platform "Direct link to Allow Kubernetes to connect to the platform") Run the following command to allow Kubernetes to communicate with the platform: ``` gcloud compute firewall-rules create allow-k8s-api-to-vcluster-platform \ --allow tcp:8080,tcp:8443,tcp:9443,tcp:9444 \ --source-ranges= \ --target-tags=vcluster-platform ``` * Replace `` with the actual IP range of the Kubernetes API server. note In private GKE clusters, the Kubernetes control plane and nodes are not in the same subnetwork. Without this rule, Kubernetes cannot communicate with platform components. ### Allow the platform to connect to Prometheus[​](#allow-the-platform-to-connect-to-prometheus "Direct link to Allow the platform to connect to Prometheus") vCluster Platform collects cost metrics from Prometheus running inside connected clusters. To enable this, the local cluster agent proxies requests to Prometheus over `TCP port 9090`. If the platform cannot access Prometheus, cost monitoring might fail. Run the following command to allow the platform to communicate with Prometheus: ``` gcloud compute firewall-rules create allow-vcluster-platform-to-prometheus \ --allow tcp:9090 \ --source-ranges= \ --target-tags=prometheus ``` * Replace `` with the IP address of the platform instance. ### Verify firewall rules[​](#verify-firewall-rules "Direct link to Verify firewall rules") After setting up firewall rules, test the connection between the platform and Prometheus: ``` nc -zv 9090 ``` If successful, the output confirm with the following message: ``` Connection to 9090 port [tcp/http] succeeded! ``` --- # Enforcing Policies Policies allow you to customize admission control and extend vCluster Platform. As vCluster Platform internally uses Kubernetes resources for anything (including UI requests), you can customize almost everything with policies. The following use cases are possible with policies: * Custom validation of users, teams, spaces, clusters and other objects. For example, you can: * Deny creation, updating or deletion of resources only for specific users or teams * Deny security critical objects such as privileged pods in clusters * Ensure unique names, hosts or other properties within a cluster * Automatic mutation of created or updated resources. For example, you can: * Add a label or annotation which user or team created an object * Inject sidecar containers into a pod * Add groups, cluster account templates etc. automatically to a user * Automation of certain cluster tasks, such as: * Automatic creation, update or deletion of users, accounts, spaces etc. on a certain condition * Sync certain resources within a cluster * Garbage collection of not needed resources ## Installing JsPolicy[​](#installing-jspolicy "Direct link to Installing JsPolicy") Policies are based on [jsPolicy](https://jspolicy.com) and you will need to install the jsPolicy app into the cluster if you want to use policies. JsPolicy is a recommended app in vCluster Platform and can be installed via the vCluster Platform UI. ![JsPolicy](/docs/media/v1/ui/policies/install-js-policy.png) Installing jsPolicy via the vCluster Platform UI Alternatively, you can also install [jsPolicy with helm](https://www.jspolicy.com/docs/quickstart#1-install-jspolicy). ## Creating policies[​](#creating-policies "Direct link to Creating policies") Policies can be created either through the vCluster Platform UI or through directly creating `JsPolicy` objects in the cluster. For a more extensive documentation how to write policies, please check the [jsPolicy docs](https://www.jspolicy.com/docs/writing-policies/workflow). * UI * kubectl Go to the Clusters > Policies view and click on 'Add Policy'. ![Create Policy](/docs/media/v1/ui/policies/create-policy.png) Create a new JsPolicy #### Metadata[​](#metadata "Direct link to Metadata") In the metadata section of the policy, you can set the name of the policy, which has to be in domain format (e.g. my-policy.jspolicy.com or any other domain you want), the annotations and labels of the policy object. In the options section, you can choose from a variety of examples to start from or create your own from scratch. The following options exist: #### Policy Type[​](#policy-type "Direct link to Policy Type") The type of the policy. For a more extensive explanation check the [jsPolicy documentation](https://www.jspolicy.com/docs/using-policies/policy-types). The type can be one of: * `Validating`: Validating policies are executed as part of Kubernetes requests after the execution of mutating policies. The objective of validating policies is to inspect the request and then to either deny or allow it. * `Mutating`: Mutating policies are executed as part of Kubernetes requests right after the API server performs authentication and authorization (RBAC). The objective of mutating policies is to change the payload (Kubernetes object) provided in a request, e.g. add/remove a label in the `metadata.labels` of a Kubernetes object that is being created. * `Controller`: Unlike mutating and validating policies, controller policies are **not** part of the lifecycle of a request to the Kubernetes API server. Controller policies are triggered by the `Events` that Kubernetes creates for each change of the cluster state in etcd, e.g. if you create a new `Deployment` or update an `Ingress`, Kubernetes creates an `Event` for this change. The objective of controller policies is to react to changes to your cluster's state. #### Policy Scope[​](#policy-scope "Direct link to Policy Scope") The scope of the policy. This can be either `*`, `Cluster` or `Namespaced` and refers to the scope of Kubernetes resources. A `Namespaced` policy will only match resources that can be created within a namespace, while `Cluster` will only match cluster-wide resources such as `ClusterRole` etc. #### Matching Resources[​](#matching-resources "Direct link to Matching Resources") The [Kubernetes resources](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-rules) this policy should match. You can specify multiple resources here or a wildcard that matches all resources. #### Matching Operations[​](#matching-operations "Direct link to Matching Operations") Operations are the operations the policy cares about and can be one or more of: CREATE, UPDATE, DELETE, CONNECT or \*. #### Policy Javascript[​](#policy-javascript "Direct link to Policy Javascript") This is the actual JavaScript payload that is executed if a policy matches a resource and operation. Please checkout the [jsPolicy documentation](https://www.jspolicy.com/docs/why-jspolicy) and [jsPolicy examples](https://github.com/loft-sh/jspolicy/tree/main/examples) for more information about how the payload could look like. After you have configured the policy, press 'Create' to add it to the cluster. There are multiple ways to create policies for the cluster. For all options, please check the [jsPolicy documentation](https://www.jspolicy.com/docs/writing-policies/workflow). This guide will cover the easiest option by creating a simple `JsPolicy` object. Make sure your kube context targets the cluster where jsPolicy was installed in. To create a new policy, create a file called `policy.yaml` with: ``` apiVersion: policy.jspolicy.com/v1beta1 kind: JsPolicy metadata: name: "deny.resource.example" spec: operations: ["CREATE"] resources: ["pods"] javascript: | // deny default namespace if (request.namespace === "default") { deny("No new pod allowed in the default namespace") } ``` Then do a `kubectl create -f policy.yaml` to create the new policy. ## Deleting policies[​](#deleting-policies "Direct link to Deleting policies") You can delete a policy either via the vCluster Platform UI or kubectl directly. * UI * kubectl Go to the Clusters > Policies view and click on 'delete' in the actions column. ![Delete Policy](/docs/media/v1/ui/policies/delete-policy.png) Delete a JsPolicy You can a delete a policy by executing `kubectl delete jspolicy my-policy.domain.tld` in the cluster where the policy was created in. ## Policy Audit[​](#policy-audit "Direct link to Policy Audit") If a request was denied by a policy, it will write this violation into its policy log. You can view all policy violations in the Audit > Policy Violations view. ![Policy Violations](/docs/media/v1/ui/policies/policy-violations.png) View policy violations This view shows information about the request that was denied, such as which user was responsible for the violation (if there was any), the action taken, the returned message etc. ## Policy Examples[​](#policy-examples "Direct link to Policy Examples") This section holds several examples for useful policies within vCluster Platform. For more examples and explanations checkout the [jsPolicy documentation](https://www.jspolicy.com/docs/quickstart). You can apply these examples by creating a `policy.yaml` and running `kubectl apply -f policy.yaml` in the target cluster. ### Deny Space Owned By Team[​](#deny-space-owned-by-team "Direct link to Deny Space Owned By Team") This policy denies spaces that are owned by teams. ``` apiVersion: policy.jspolicy.com/v1beta1 kind: JsPolicy metadata: name: "deny-teams-spaces.resource.example" spec: operations: ["CREATE"] resources: ["spaces"] javascript: | // Do not allow creation of spaces that have a team as owner if (request.object?.spec?.account) { const account = get("Account", "config.kiosk.sh/v1alpha1", request.object.spec.account); if (account?.metadata?.labels?.["loft.sh/team"]) { deny("Teams cannot own spaces"); } } ``` ### Deny Pods of User[​](#deny-pods-of-user "Direct link to Deny Pods of User") This policy will deny pod creation only for a single vCluster Platform user called `test`. ``` apiVersion: policy.jspolicy.com/v1beta1 kind: JsPolicy metadata: name: "deny-user-pods.resource.example" spec: operations: ["CREATE"] resources: ["pods"] javascript: | // Do not allow the user "test" to create pods anywhere if (request.userInfo?.groups?.includes("loft:user:test")) { deny("Sorry test, but you are not allowed to create pods in this cluster"); } ``` ### Deny Resources in default Namespace[​](#deny-resources-in-default-namespace "Direct link to Deny Resources in default Namespace") This policy will deny all new resources that should be created in the default namespace. ``` apiVersion: policy.jspolicy.com/v1beta1 kind: JsPolicy metadata: name: "deny-default-namespace.resource.example" spec: operations: ["CREATE"] resources: ["*"] scope: Namespaced javascript: | if (request.namespace === "default") { deny("Creation of resources within the default namespace is not allowed!"); } ``` ### Deny Privileged Pods[​](#deny-privileged-pods "Direct link to Deny Privileged Pods") This policy will deny privileged pods. ``` apiVersion: policy.jspolicy.com/v1beta1 kind: JsPolicy metadata: name: "deny-privileged-pods.resource.example" spec: operations: ["CREATE"] resources: ["pods"] javascript: | const containers = request.object.spec?.containers || []; const initContainers = request.object.spec?.initContainers || []; [...containers, ...initContainers].forEach(container => { if (container.securityContext?.privileged) { deny("spec.containers[*].securityContext.privileged is not allowed") } }) ``` ### Deny Ingress Classes[​](#deny-ingress-classes "Direct link to Deny Ingress Classes") This policy will deny all ingress classes except `istio`. ``` apiVersion: policy.jspolicy.com/v1beta1 kind: JsPolicy metadata: name: "deny-ingress-classes.resource.example" spec: operations: ["CREATE", "UPDATE"] resources: ["ingresses"] javascript: | // ingress class can be set via annotation "kubernetes.io/ingress.class" or through spec.ingressClassName. const allowedIngressClasses = ["istio"]; const ingressClasses = [request.object.metadata?.annotations?.["kubernetes.io/ingress.class"], request.object.spec.ingressClassName]; const notAllowed = ingressClasses.filter(ingressClass => ingressClass && !allowedIngressClasses.includes(ingressClass)); if (notAllowed.length > 0) { deny(`ingress class ${notAllowed[0]} is not allowed`); } ``` ### Allow Space only if Part of Team[​](#allow-space-only-if-part-of-team "Direct link to Allow Space only if Part of Team") This policy will only allow space creation if the user is part of team 'admins'. ``` apiVersion: policy.jspolicy.com/v1beta1 kind: JsPolicy metadata: name: "allow-space-creation.resource.example" spec: operations: ["CREATE"] resources: ["spaces"] javascript: | // Allow space creation only if part of team "admins" if (!request.userInfo?.groups?.includes("loft:team:admins")) { deny("Sorry you cannot create spaces since you are not part of the admins team"); } ``` ### Set User as Annotation On New Object[​](#set-user-as-annotation-on-new-object "Direct link to Set User as Annotation On New Object") This policy will set the user that has created it on a new kubernetes object. ``` apiVersion: policy.jspolicy.com/v1beta1 kind: JsPolicy metadata: name: "set-user-as-annotation.resource.example" spec: operations: ["CREATE"] resources: ["*"] scope: Namespaced type: Mutating javascript: | // Set the current user as an annotation on the object const allAccounts = list("Account", "config.kiosk.sh/v1alpha1"); // Find account of the user by subject const userAccount = allAccounts.find(account => { // flatten account subjects const subjects = account.spec?.subjects?.map(subject => subject.name) || []; return subjects.includes(request.userInfo.username); }); if (userAccount) { if (!request.object.metadata) { request.object.metadata = {}; } if (!request.object.metadata.annotations) { request.object.metadata.annotations = {}; } request.object.metadata.annotations["loft.sh/created-by-user"] = userAccount.metadata?.labels?.["loft.sh/user"]; mutate(request.object); } ``` ### Inject Sidecar Container[​](#inject-sidecar-container "Direct link to Inject Sidecar Container") This policy will inject a sidecar container into pods that have an annotation "inject-side-car". ``` apiVersion: policy.jspolicy.com/v1beta1 kind: JsPolicy metadata: name: "inject-sidecar.resource.example" spec: operations: ["CREATE"] resources: ["pods"] type: Mutating javascript: | // if the annotation inject-side-car: 'true' is set on the pod we inject a sidecar if (request.object.metadata?.annotations?.["inject-side-car"] === "true") { if (!request.object.spec.containers) { request.object.spec.containers = []; } // add sidecar request.object.spec.containers.push({ name: "injected-container", image: "busybox", command: ["sleep", "3600"] }); mutate(request.object); } ``` ### Unique Ingress Hosts[​](#unique-ingress-hosts "Direct link to Unique Ingress Hosts") This policy will ensure that ingress hosts are unique in the cluster. ``` apiVersion: policy.jspolicy.com/v1beta1 kind: JsPolicy metadata: name: "unique-ingress-hosts.resource.example" spec: operations: ["CREATE", "UPDATE"] resources: ["ingresses"] type: Mutating javascript: | // iterate over all ingresses in cluster and check for same hosts list("Ingress", "networking.k8s.io/v1").forEach(ingress => { // don't check self on UPDATE if (ingress.metadata.name === request.name && ingress.metadata.namespace === request.namespace) { return; } ingress.spec.rules?.forEach(rule => { request.object.spec.rules?.forEach(objRule => { if (rule.host === objRule.host) { deny(\`ingress ` + "${ingress.metadata.namespace}/${ingress.metadata.name} already uses host ${rule.host}" + `\`); } }) }) }) ``` ### Sync Secrets[​](#sync-secrets "Direct link to Sync Secrets") This policy will sync secrets that have the labels `owner-name` and `owner-namespace` with a parent secret that has the label `sync`. On a change to a parent or child secret, the controller policy will evaluate if an update is necessary and update the child secret if it differs from the parent secret. ``` apiVersion: policy.jspolicy.com/v1beta1 kind: JsPolicy metadata: name: "sync-secrets.resource.example" spec: operations: ["CREATE"] resources: ["secrets"] type: Controller javascript: | // this policy will sync changes to a secret to all other secrets in the cluster // that have labels in the form of: {"owner-name": SECRET, "owner-namespace": NAMESPACE} // is this a parent or child secret? const labels = request.object.metadata?.labels; let syncSecret = undefined; if (labels?.["sync"] === "true") { syncSecret = request.object; } else if (labels?.["owner-name"] && labels?.["owner-namespace"]) { syncSecret = get("Secret", "v1", labels?.["owner-namespace"] + "/" + labels?.["owner-name"]); } // this is not a secret we should sync if (!syncSecret) { allow(); } // list all owned secrets list("Secret", "v1", { labelSelector: \`owner-name=`+"${syncSecret.metadata.name},owner-namespace=${syncSecret.metadata.namespace}"+`\` }).forEach(secret => { // only update if data has actually changed if (JSON.stringify(secret.data) === JSON.stringify(syncSecret.data)) { return; } // sync the secret data secret.data = syncSecret.data; const updateResult = update(secret); if (!updateResult.ok) { // just requeue if we encounter an error requeue(updateResult.message); } else { print(\`Successfully synced secret `+"${secret.metadata.namespace}/${secret.metadata.name}"+`\`); } }); ``` ## Why jsPolicy and not OPA / Kyverno?[​](#why-jspolicy-and-not-opa--kyverno "Direct link to Why jsPolicy and not OPA / Kyverno?") vCluster Platform does not require you to use jsPolicy for policies, and you can always use your own preferred solution. Every policy engine should work fine with vCluster Platform. However, if you want to use the policies view in vCluster Platform, this currently only works with [jsPolicy](https://github.com/loft-sh/jspolicy). We decided to integrate jsPolicy instead of another policy engine, because jsPolicy is the most flexible one that can cover the most use cases within vCluster Platform. Especially with [controller policies](https://www.jspolicy.com/docs/using-policies/policy-types#controller) and [mutating webhooks](https://www.jspolicy.com/docs/using-policies/policy-types#mutating) that are not at all or only partly supported in other policy engines, jsPolicy has a huge advantage in terms of flexibility and expressiveness. --- # Regional Cluster Endpoints | | | Enterprise | | | | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Multi-Region Platform](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md) | | | | | | [Regional Cluster Endpoints](https://vcluster.com/docs/platform/administer/clusters/advanced/regional-cluster-endpoints.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Regional Cluster Endpoints provide a way to avoid routing through the central vCluster Platform instance and connect directly to a connected cluster. By default, all Kubernetes contexts created through vCluster Platform route Kubernetes traffic (such as `kubectl get pods`) through the central vCluster Platform instance. This allows the platform to handle authentication, authorization, auditing, and sleep mode activity. However, if you have multiple connected clusters in a platform instance that are spread across the globe, traffic redirection through the central vCluster Platform gateway can increase request delays. A solution to this is regional cluster endpoints, which avoid rerouting traffic to the central vCluster Platform instance and instead handle authentication, authorization, and other functions directly within the connected cluster through the installed vCluster Platform agent. ## Enable regional cluster endpoints[​](#enable-regional-cluster-endpoints "Direct link to Enable regional cluster endpoints") To enable Regional Cluster Endpoints, expose the vCluster Platform agent endpoints. The automatically installed vCluster Platform agent includes these endpoints, which you can activate by adding an annotation to the connected cluster object. Expose the vCluster Platform agent through either an Ingress or a LoadBalancer/NodePort service. Create an `ingress.yaml` file and apply it using `kubectl`: ingress.yaml ``` apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: loft-agent-ingress annotations: kubernetes.io/ingress.class: nginx # ingress-nginx has been deprecated nginx.ingress.kubernetes.io/proxy-read-timeout: "43200" nginx.ingress.kubernetes.io/proxy-send-timeout: "43200" nginx.ingress.kubernetes.io/proxy-buffers-number: "8 32k" nginx.ingress.kubernetes.io/proxy-buffer-size: "32k" nginx.org/websocket-services: loft-agent cert-manager.io/cluster-issuer: my-cluster-issuer spec: rules: - host: my-cluster-endpoint.com http: paths: - path: / pathType: ImplementationSpecific backend: service: name: loft-agent port: number: 80 tls: - hosts: - my-cluster-endpoint.com secretName: name-of-tls-secret ``` Then apply the ingress using `kubectl`: Apply ingress configuration ``` kubectl apply -n vcluster-platform -f ingress.yaml ingress.networking.k8s.io/loft-agent-ingress created ``` tip As `kubectl` requires an HTTPS connection, the regional cluster endpoint must also be reachable using HTTPS. By default, the ingress uses a self-signed certificate, which is insecure. It is recommended to use either cert-manager to issue a "Let's Encrypt" certificate or to provide your own trusted certificate. After creating the ingress, ensure that the regional cluster endpoint is reachable: Test regional cluster endpoint health ``` # Omit --insecure if have a valid TLS certificate curl https://my-cluster-endpoint.com/healthz -I --insecure HTTP/2 200 date: Mon, 30 Aug 2021 08:40:20 GMT access-control-allow-headers: * access-control-allow-methods: POST, GET, OPTIONS, PUT, DELETE, PATCH access-control-allow-origin: * cache-control: no-cache, private strict-transport-security: max-age=15724800; includeSubDomains ``` ## Enable a regional cluster endpoint[​](#enable-a-regional-cluster-endpoint "Direct link to Enable a regional cluster endpoint") To configure the platform to use a regional cluster endpoint for a connected cluster, add an annotation to the cluster object: Cluster configuration with direct endpoint ``` apiVersion: management.loft.sh/v1 kind: Cluster metadata: name: test annotations: # Domain of the regional cluster endpoint loft.sh/direct-cluster-endpoint: my-direct-cluster-endpoint.com # # Optional option to tell vCluster Platform the endpoint is insecure # #loft.sh/direct-cluster-endpoint-insecure: 'true' # # Optional option to use this base64 encoded ca cert for the endpoint # #loft.sh/direct-cluster-endpoint-ca-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tC... spec: config: {} ``` The following annotations can be specified: * **`loft.sh/direct-cluster-endpoint`**: This is the domain name that should be used to connect to the cluster directly. This usually corresponds to the ingress host that you have specified earlier. As soon as this annotation is set, the regional cluster endpoint for the connected cluster is turned on and is preferred in future over the central vCluster Platform instance * **`loft.sh/direct-cluster-endpoint-insecure`** (*Optional*): If this is `true`, vCluster Platform creates Kubernetes contexts with `insecure-skip-tls-verify: true` for this endpoint * **`loft.sh/direct-cluster-endpoint-ca-data`** (*Optional*): Base64 encoded ca cert of the regional cluster endpoint ## Use a regional cluster endpoint[​](#use-a-regional-cluster-endpoint "Direct link to Use a regional cluster endpoint") tip This is supported in vCluster CLI v0.26+ If a regional cluster endpoint is enabled via the `loft.sh/direct-cluster-endpoint` annotation on a connected cluster, the vCluster Platform commands `vcluster platform connect cluster`, `vcluster platform connect space`, `vcluster platform connect vcluster` and `vcluster platform create namespace` use the regional cluster endpoint instead of the central vCluster Platform gateway. The CLI also notifies you when running one of the above commands if a regional cluster endpoint is used: Connect to cluster ``` vcluster platform connect cluster test ``` Example output ``` 11:45:26 info Using direct host endpoint: https://my-direct-cluster-endpoint.com 11:45:26 done Successfully updated kube context to use cluster agent ``` You can then check which server is contacted by running: Get namespaces with verbose logging ``` kubectl get ns -v 6 ``` Example output ``` I0624 15:53:30.287997 70943 loader.go:379] Config loaded from file: /xxx/.kube/config I0624 15:53:30.307530 70943 round_trippers.go:445] GET https://my-cluster-endpoint.com/kubernetes/cluster/api/v1/namespaces?limit=500 200 OK in 13 milliseconds NAME STATUS AGE default Active 164m kube-node-lease Active 164m kube-public Active 164m kube-system Active 164m ``` The `kubectl` commands now route traffic directly to the cluster endpoint rather than through the vCluster Platform instance. --- # Register clusters with Terraform Automate control plane cluster registration to vCluster Platform using Terraform. This guide shows you how to programmatically register clusters during infrastructure provisioning, eliminating manual registration steps. ## Overview[​](#overview "Direct link to Overview") When provisioning infrastructure with Terraform, you can automate the complete cluster registration workflow. This approach registers the cluster in vCluster Platform and installs the agent, making the cluster immediately available for tenant cluster deployment. Terraform provider deprecation The vCluster Platform Terraform provider is deprecated. This guide provides an alternative approach using the Kubernetes provider and vCluster Platform API to achieve the same automation goals. ## Why automate cluster registration[​](#why-automate-cluster-registration "Direct link to Why automate cluster registration") Manual cluster registration requires running `vcluster platform add cluster` after each cluster deployment. This creates friction in automated workflows and requires manual intervention. By automating registration through Terraform, you can: * Provision and register clusters in a single Terraform apply * Maintain infrastructure as code for the complete cluster lifecycle * Eliminate manual steps in CI/CD pipelines * Ensure consistent cluster configuration across environments ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you begin, ensure you have: * **Terraform installed**: Version 1.0 or higher * **vCluster Platform instance**: Running and accessible * **admin access key**: Required to create cluster-specific access keys * **kubectl access**: To the vCluster Platform management cluster * **Target cluster**: The cluster you want to register (or Terraform configuration to create it) ### admin access key[​](#admin-access-key "Direct link to admin access key") The automation requires an admin-level access key to create cluster-specific access keys through the API. You can create an access key through the [vCluster Platform UI](https://vcluster.com/docs/platform/administer/authentication/access-keys.md#create-an-access-key) or CLI. Secure credential storage Store the admin access key securely using a secrets management solution. Never commit access keys to version control. Consider using: * [HashiCorp Vault](https://vcluster.com/docs/platform/integrations/hashicorp-vault.md) for centralized secret management * Kubernetes Secrets with encryption at rest * Provider secret managers (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager) * Environment variables in CI/CD systems with secret masking ## How it works[​](#how-it-works "Direct link to How it works") The registration process follows three steps: 1. **Create cluster resource**: Register the cluster in vCluster Platform's management cluster 2. **Retrieve cluster-specific access key**: Get the unique key for this cluster using the API 3. **Install platform agent**: Deploy the agent using the cluster-specific key The cluster-specific access key authenticates the agent to vCluster Platform with the correct cluster identity. Using an admin key for the agent fails because the agent expects a cluster-scoped credential. ## Step-by-step guide[​](#step-by-step-guide "Direct link to Step-by-step guide") Platform hostname format The `PLATFORM_HOST` variable should contain only the hostname without the `https://` protocol prefix. For example: `platform.example.com` not `https://platform.example.com`. The Terraform configuration adds the protocol automatically. Modify the following with your specific values to replace on the whole page and generate copyable commands: PLATFORM\_HOSTplatform.example.com ADMIN\_ACCESS\_KEYyour-admin-key PLATFORM\_CONTEXTplatform-context TARGET\_CLUSTER\_CONTEXTtarget-cluster-context CLUSTER\_NAMEprod-cluster-1 CLUSTER\_DISPLAY\_NAMEProduction Cluster 1 1. Set your environment variables using the interactive form at the top of this page: Set environment variables ``` # Set variables export TF_VAR_platform_host="platform.example.com" export TF_VAR_admin_access_key="your-admin-key" export TF_VAR_platform_context="platform-context" export TF_VAR_target_cluster_context="target-cluster-context" export TF_VAR_cluster_name="prod-cluster-1" export TF_VAR_cluster_display_name="Production Cluster 1" ``` 2. Create the Cluster resource configuration. This registers the cluster with Platform and prepares it for agent connection. cluster-registration.tf ``` # Configure Kubernetes provider for vCluster Platform management cluster provider "kubernetes" { alias = "platform" config_path = "~/.kube/config" config_context = var.platform_context } # Variables variable "platform_context" { description = "kubectl context for the vCluster Platform management cluster" type = string } # Create the cluster resource resource "kubernetes_manifest" "platform_cluster" { provider = kubernetes.platform manifest = { apiVersion = "management.loft.sh/v1" kind = "Cluster" metadata = { name = var.cluster_name } spec = { displayName = var.cluster_display_name networkPeer = true managementNamespace = "vcluster-platform" } } } # Variables variable "cluster_name" { description = "Internal cluster identifier" type = string } variable "cluster_display_name" { description = "Display name shown in vCluster Platform UI" type = string } ``` **Configuration options**: * `name`: Internal identifier for the cluster (must be DNS-compatible) * `displayName`: Human-readable name shown in the UI * `networkPeer`: Enable network peering between clusters * `managementNamespace`: Namespace where the agent will be installed 3. Configure the HTTP data source to retrieve the cluster-specific access key. After creating the cluster resource, retrieve the cluster-specific access key using the vCluster Platform API. cluster-access-key.tf ``` # Data source to retrieve cluster access key data "http" "cluster_access_key" { url = "https://${var.platform_host}/kubernetes/management/apis/management.loft.sh/v1/clusters/${var.cluster_name}/accesskey" request_headers = { Authorization = "bearer ${var.admin_access_key}" } depends_on = [kubernetes_manifest.platform_cluster] } # Parse the response locals { access_key_response = jsondecode(data.http.cluster_access_key.response_body) cluster_access_key = local.access_key_response.accessKey loft_host = local.access_key_response.loftHost } # Variables variable "platform_host" { description = "vCluster Platform hostname without protocol (e.g., 'platform.example.com' not 'https://platform.example.com')" type = string } variable "admin_access_key" { description = "Admin access key for API authentication" type = string sensitive = true } ``` The API endpoint returns a JSON response containing: * `accessKey`: Cluster-specific token for agent authentication * `loftHost`: vCluster Platform host URL 4. Configure the Helm release to install the Platform agent. Install the vCluster Platform agent on your target cluster using the cluster-specific access key. agent-installation.tf ``` # Configure Kubernetes provider for target cluster provider "kubernetes" { alias = "target" config_path = "~/.kube/config" config_context = var.target_cluster_context } provider "helm" { alias = "target" kubernetes { config_path = "~/.kube/config" config_context = var.target_cluster_context } } # Get platform version for agent data "http" "platform_version" { url = "https://${var.platform_host}/version" } locals { platform_version = jsondecode(data.http.platform_version.response_body).version } # Install vCluster Platform agent resource "helm_release" "platform_agent" { provider = helm.target name = "vcluster-platform" repository = "https://charts.loft.sh/" chart = "vcluster-platform" version = local.platform_version namespace = "vcluster-platform" create_namespace = true set { name = "agentOnly" value = "true" } set { name = "url" value = "https://${var.platform_host}" } set { name = "token" value = local.cluster_access_key } depends_on = [data.http.cluster_access_key] } # Variables variable "target_cluster_context" { description = "kubectl context for the target cluster" type = string } ``` **What happens**: the agent installs in the `vcluster-platform` namespace, connects to the platform using the cluster-specific key, and establishes a secure tunnel. The cluster becomes available in the vCluster Platform UI within moments. 5. Initialize Terraform. Initialize Terraform ``` terraform init ``` This downloads the required provider plugins (Kubernetes, Helm, HTTP). 6. Review the execution plan. Preview changes ``` terraform plan ``` Verify that Terraform will: * Create the Cluster resource in Platform * Retrieve the cluster-specific access key using the API * Install the Platform agent using Helm 7. Apply the configuration. Apply configuration ``` terraform apply ``` Type `yes` when prompted. This executes all three steps: creating the cluster resource, retrieving the access key, and installing the agent. 8. Verify the cluster in Platform UI. Navigate to Infrastructure > Control Plane Clusters in the Platform interface. Your cluster should appear with "Connected" status within moments. ### Configuration files reference[​](#configuration-files-reference "Direct link to Configuration files reference") The following sections show the detailed Terraform configuration for each component. Quick reference - Complete Terraform configuration Copy and paste this complete configuration with your customized values: main.tf ``` terraform { required_providers { kubernetes = { source = "hashicorp/kubernetes" version = "~> 2.23" } helm = { source = "hashicorp/helm" version = "~> 2.11" } http = { source = "hashicorp/http" version = "~> 3.4" } } } # Provider for vCluster Platform management cluster provider "kubernetes" { alias = "platform" config_path = "~/.kube/config" config_context = "platform-context" } # Provider for target cluster provider "kubernetes" { alias = "target" config_path = "~/.kube/config" config_context = "target-cluster-context" } provider "helm" { alias = "target" kubernetes { config_path = "~/.kube/config" config_context = "target-cluster-context" } } # Step 1: Create cluster resource resource "kubernetes_manifest" "cluster" { provider = kubernetes.platform manifest = { apiVersion = "management.loft.sh/v1" kind = "Cluster" metadata = { name = "prod-cluster-1" } spec = { displayName = "Production Cluster 1" networkPeer = true managementNamespace = "vcluster-platform" } } } # Step 2: Get cluster access key data "http" "cluster_key" { url = "https://platform.example.com/kubernetes/management/apis/management.loft.sh/v1/clusters/prod-cluster-1/accesskey" request_headers = { Authorization = "bearer your-admin-key" } depends_on = [kubernetes_manifest.cluster] } locals { key_response = jsondecode(data.http.cluster_key.response_body) } # Get platform version data "http" "version" { url = "https://platform.example.com/version" } locals { version = jsondecode(data.http.version.response_body).version } # Step 3: Install agent resource "helm_release" "agent" { provider = helm.target name = "vcluster-platform" repository = "https://charts.loft.sh/" chart = "vcluster-platform" version = local.version namespace = "vcluster-platform" create_namespace = true set { name = "agentOnly" value = "true" } set { name = "url" value = "https://platform.example.com" } set { name = "token" value = local.key_response.accessKey } depends_on = [data.http.cluster_key] } # Variables variable "platform_host" { description = "vCluster Platform hostname without protocol (e.g., platform.example.com)" type = string } variable "platform_context" { description = "kubectl context for platform management cluster" type = string } variable "target_cluster_context" { description = "kubectl context for target cluster" type = string } variable "cluster_name" { description = "Cluster identifier" type = string } variable "cluster_display_name" { description = "Display name in UI" type = string } variable "admin_access_key" { description = "Admin access key" type = string sensitive = true } # Outputs output "cluster_registered" { value = "Cluster prod-cluster-1 registered successfully" } ``` ## Security considerations[​](#security-considerations "Direct link to Security considerations") ### Access key management[​](#access-key-management "Direct link to Access key management") The admin access key provides full platform access. Protect it using these practices: **Store in secrets management**: use a dedicated secrets solution rather than environment variables or files: * [HashiCorp Vault](https://vcluster.com/docs/platform/integrations/hashicorp-vault.md) with dynamic credentials * Provider secret managers with automatic rotation * Kubernetes Secrets with encryption at rest and RBAC controls **Use scoped keys when possible** - While this workflow requires an admin key for the initial setup, consider using [scoped access keys](https://vcluster.com/docs/platform/administer/authentication/access-keys.md#scoped-access-keys) for other automation tasks. **Rotate regularly** - Establish a rotation schedule for access keys: * Set expiration dates on keys * Automate rotation through your secrets management system * Revoke keys immediately when team members leave **Audit access** - Monitor access key usage: * Enable audit logging for API calls * Review access key usage patterns * Alert on unusual activity ### Network security[​](#network-security "Direct link to Network security") The agent establishes an outbound connection to vCluster Platform. Ensure: * Platform endpoint uses TLS (never set `insecure = true` in production) * Network policies allow outbound HTTPS from the agent namespace * Firewall rules permit the agent to reach the platform endpoint ## Known limitations[​](#known-limitations "Direct link to Known limitations") ### Bootstrap credential requirement[​](#bootstrap-credential-requirement "Direct link to Bootstrap credential requirement") This automation requires an admin access key to bootstrap the process. This creates a "chicken and egg" problem: you need credentials to create cluster-specific credentials. **Solutions**: 1. **Pre-provision keys**: Create the admin access key manually before running Terraform: ``` # Create key through UI or CLI vcluster platform create accesskey bootstrap-key # Store in secrets management ``` 2. **Use secrets management**: Store the bootstrap key in your secrets management system: * HashiCorp Vault: use the [Vault provider](https://registry.terraform.io/providers/hashicorp/vault/latest/docs) to retrieve keys * Provider platforms: use native Terraform data sources for secret managers * External Secrets Operator: sync keys from external systems to Kubernetes 3. **Manual initial registration**: For the very first cluster or platform instance, register it manually, then use this automation for subsequent clusters. ### Provider contexts[​](#provider-contexts "Direct link to Provider contexts") The configuration requires two kubernetes provider contexts: one for the platform management cluster and one for the target cluster. Ensure both contexts are configured in your kubeconfig before running Terraform. ## Troubleshoot common issues[​](#troubleshoot-common-issues "Direct link to Troubleshoot common issues") ### Agent fails to connect[​](#agent-fails-to-connect "Direct link to Agent fails to connect") **Symptom**: agent pod shows authentication errors in logs. **Cause**: using admin access key instead of cluster-specific key. **Solution**: verify you're using the key from the API endpoint (`/clusters//accesskey`), not a user access key. ### Cluster resource creation fails[​](#cluster-resource-creation-fails "Direct link to Cluster resource creation fails") **Symptom**: `kubernetes_manifest` resource fails with permission denied. **Cause**: kubectl context doesn't have permissions to create Cluster resources. **Solution**: ensure your platform context has admin permissions or appropriate RBAC for Cluster resources. ### Cannot retrieve access key[​](#cannot-retrieve-access-key "Direct link to Cannot retrieve access key") **Symptom**: http data source fails with 401 or 403 status code. **Cause**: admin access key invalid or lacks permissions. **Solution**: verify the access key works with curl: Verify access key with curl ``` curl -s "https://platform.example.com/kubernetes/management/apis/management.loft.sh/v1/clusters/prod-cluster-1/accesskey" \ -H "Authorization: bearer your-admin-key" ``` ## Next steps[​](#next-steps "Direct link to Next steps") After registering your cluster, you can: * [Create tenant clusters](https://vcluster.com/docs/platform/use-platform/virtual-clusters/add-virtual-clusters.md) on the registered control plane cluster * [Configure agent settings](https://vcluster.com/docs/platform/administer/clusters/advanced/agent-config.md) for advanced networking or security requirements * [Set up monitoring](https://vcluster.com/docs/platform/maintenance/monitoring/overview.md) for the agent and connected resources * [Implement policies](https://vcluster.com/docs/platform/administer/clusters/advanced/policies.md) to control resource usage across clusters --- # Cluster Roles | | | Enterprise | | | | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Cluster Access](https://vcluster.com/docs/platform/administer/clusters/cluster-roles.md) | | | | | | [Cluster Role Management](https://vcluster.com/docs/platform/administer/clusters/cluster-roles.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Cluster Roles lets you define Kubernetes RBAC Cluster roles that are synced to the specified clusters. These roles can then be assigned to individual Users or Teams within vCluster Platform by using Cluster Access objects. Cluster Roles also support aggregation rules that allow combining multiple cluster roles into one using label selectors. The vCluster Platform comes with some predefined cluster roles out-of-the-box. You can make use of them within your own spaces or clusters. However, if none of them quite fit your needs, then you can create your own Cluster Role by following the below steps. ### Create a cluster role[​](#create-a-cluster-role "Direct link to Create a cluster role") 1. Go to Infrastructure > Control Plane Clusters. 2. Click the Cluster Roles tab, then click Create Role. 3. In the configuration sheet that opens, give the role a name by replacing the 'my-role' placeholder, or by editing the manifest YAML 'metadata.name' field. 4. In the Rules section, specify RBAC rules in the RBAC Rules tab, or specify an aggregation rule in the Aggregation Rule tab. Select the cluster to sync the role to. 5. In the Management Access section, select the users or teams that should have access to this role object and set their allowed permissions. 6. Click Save Changes to create the role. --- # Connect a Cluster | | | Enterprise | | | | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Connected Clusters](https://vcluster.com/docs/platform/administer/clusters/connect-cluster.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). vCluster Platform manages tenant clusters that run across one or more connected clusters. A connected cluster is a Kubernetes cluster that serves as a Control Plane Cluster, where tenant cluster control planes run. To enable management through vCluster Platform, the vCluster Platform agent must be installed on each cluster you intend to manage. The installation can be performed through the vCluster Platform UI, CLI, or Helm. Control Plane Cluster access Direct access to a Control Plane Cluster grants broad permissions to platform resources and tenant clusters running on it. Grant this access only to privileged platform administrators. Unprivileged users should access tenant clusters directly rather than through the Control Plane Cluster or its pods. See the [Platform CLI hub](https://vcluster.com/docs/platform/reference/platform-cli.md) for UI-to-CLI equivalents across common Platform tasks, including connecting clusters. * UI * CLI * Helm 1. Go to Infrastructure > Control Plane Clusters. 2. Click Connect Cluster. 3. Give your cluster a name in Display Name. Optionally, give a name for the underlying kubernetes resource in the Cluster ID field, or leave it empty to autogenerate one. Then click Next. 4. Copy & execute the displayed vCluster CLI or Helm command. 5. Wait until vCluster Platform installs the agent in your connected cluster. 6. Once successful, use the displayed CLI command to create a space / virtual cluster in the newly connected cluster. 7. Click Finish to go to the Control Plane Clusters view. Connect a cluster UI will generate a CLI command that can be used to connect the cluster. Make sure to check the `CLI` tab for working with the command. 1. When connecting a cluster from the vCluster CLI, you will need to provide a name for the cluster. This name will be used for the Cluster resource that gets created in Kubernetes. Optionally, you can use the `display-name` flag to specify a name for this cluster, when it gets displayed under Infrastructure > Control Plane Clusters in vCluster Platform. ``` vcluster platform add cluster cluster-name --display-name [display-name] ``` The above command automatically installs the [vCluster Platform Agent](https://vcluster.com/docs/platform/administer/clusters/advanced/agent-config.md) in the cluster that corresponds to your current Kubernetes context. The control plane cluster where vCluster Platform itself is deployed is automatically connected to the platform during installation. Therefore, this command should be executed in a different cluster from the one where vCluster Platform is running. By default, the `vcluster-platform` namespace is created automatically if it does not already exist on the cluster. The vCluster Platform Agent resources are deployed into this namespace. You can override this behavior by explicitly specifying a different namespace using the `--namespace` flag. If you want to connect a cluster that is not associated with your current Kubernetes context, use the `--context` flag to specify the context of the target cluster. info For more information about available CLI flags, refer to the [CLI documentation](https://vcluster.com/docs/vcluster/cli/vcluster_platform_add_cluster). ``` vcluster platform add cluster cluster-name --display-name [display-name] --context [context] ``` 2. You can verify that the cluster is connected to the platform by running: ``` vcluster platform list clusters ``` and the expected output is: ``` CLUSTER | AGE --------------------+-------- cluster-name | 7s host-cluster | 10m4s ``` The `host-cluster` is the cluster name where the vCluster Platform is deployed. platform login **Connect to the Platform**: to use any `vcluster platform` command, you need to be connected to the platform. See [authentication documentation](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) for more details. When you started the platform, you would have automatically connected to the platform when you ran `vcluster platform start`. If you aren't sure if you are connected and already have a running platform, then you can login. Before running this command, be sure that your kubecontext is set to the cluster running the platform. Login to the platform. ``` vcluster platform login ``` If your kubectl context is not set to the cluster with running platform, it's possible to login to the platform using the CLI and access key. If you haven't already, you need to [create access key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md#create-an-access-key) to be able to login to the platform. Login to the platform using access key ``` vcluster platform login [platform-url] --access-key [access-key] ``` 1. If you prefer to use Helm, you need to make sure that you have a valid Access Key. You can either reuse the CLI Access Key or create a new one by following the [Access Key guide](https://vcluster.com/docs/platform/administer/authentication/access-keys.md#create-an-access-key). * Reusing the CLI Access Key ``` export ACCESS_KEY=$(vcluster platform access-key | jq -r .status.token) ``` * Create new Access Key by following the [Access Key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md#create-an-access-key) guide. ``` export ACCESS_KEY="[Newly Created Access Key]" ``` 2. After that, you need to provide a name for the cluster. ``` export CLUSTER_NAME="cluster-name" ``` 3. Next, you must create the `Cluster` custom resource within the platform using the following code: ``` cat <v11 ️1 | v10, v11 | 1 Needs explicit configuration to allow unencrypted transport: `require_secure_transport` set to `OFF`.
info While other, not mentioned, PostgreSQL/MySQL compatible databases may work, they have not been tested. ### How cloud database offerings are tested[​](#how-cloud-database-offerings-are-tested "Direct link to How cloud database offerings are tested") The following sections describe how each hyperscaler database offering is tested, including any notable deviations from a default deployment. AWS * Aurora PostgreSQL * Aurora MySQL * RDS PostgreSQL * RDS MySQL * RDS MariaDB Tested with major versions **v14, v15, v16, and v17**. * **Instance class:** `db.t3.medium` * **High availability:** Single-instance Aurora cluster (no read replicas) Tested with major versions **v5, v8, and v9**. * **Instance class:** `db.t3.medium` * **High availability:** Single-instance Aurora cluster (no read replicas) Tested with major versions **v14, v15, v16, v17, and v18**. * **Instance class:** `db.t3.micro` * **Storage:** 20 GB (General Purpose SSD) * **High availability:** No, Single-AZ Tested with major versions **v5, v8, and v9**. * **Instance class:** `db.t3.micro` * **Storage:** 20 GB (General Purpose SSD) * **High availability:** No, Single-AZ Tested with major versions **v10 and v11**. * **Instance class:** `db.t3.micro` * **Storage:** 20 GB (General Purpose SSD) * **High availability:** No, Single-AZ * **Secure transport:** For MariaDB **v11+**, `require_secure_transport` is enabled by default. A custom DB parameter group is used to set `require_secure_transport=OFF` (**deviation from default**). For **v10**, no changes are needed. Azure * Azure Database for PostgreSQL flexible server * Azure Database for MySQL flexible server Tested with major versions **v14, v15, v16, v17, and v18**. * **SKU / tier:** Burstable, `Standard_B1ms` * **High availability:** No (single server) Tested with major versions **v5, v8, and v9**. * **SKU / tier:** Burstable, `Standard_B1ms` * **High availability:** No (single server) * **Secure transport:** `require_secure_transport` set to `OFF` (**deviation from default**). GCP * Google Cloud SQL PostgreSQL * Google Cloud SQL MySQL Tested with major versions **v14, v15, v16, v17, and v18**. Google Cloud SQL PostgreSQL requires vCluster Platform v4.7 or later for all versions. * **Edition / tier:** Enterprise, `db-f1-micro` * **High availability:** No (single-zone) * **User setup:** A dedicated database user is created because Cloud SQL does not expose the `postgres` superuser with full privileges. Tested with major versions **v5, v8, and v9**. * **Edition / tier:** Enterprise, `db-f1-micro` * **High availability:** No (single-zone) * **User setup:** A dedicated database user is created with a `%` host wildcard to allow connections from any host. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Your current kube-context must have administrative privileges, which you can verify with `kubectl auth can-i create clusterrole -A` info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm` installed: Helm v3.10.0+ is required for deploying the platform. Refer to the [Helm Installation Guide](https://helm.sh/docs/intro/install/) if you need to install it. * `kubectl` installed: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. - A running PostgreSQL or MySQL compatible DBMS and admin credentials to the server ## Configuration[​](#configuration "Direct link to Configuration") ### Create platform admin connector secret[​](#create-platform-admin-connector-secret "Direct link to Create platform admin connector secret") A connector secret is a secret that can be used with a platform feature or integration. For the platform to start using a database server, a secret must be created that contains admin credentials for the server. The secret must: * Contain the `loft.sh/connector-type` label with the `shared-database` value * Reside in the same namespace as the platform * Contain the fields: * endpoint * password * port * user To create a database connector secret in the UI, navigate to the sidebar and select **Databases** -> **Connect Database Secrets**. Alternatively, a Secret YAML manifest can be applied to the platform's control plane cluster. Shared database secret template ``` apiVersion: v1 kind: Secret metadata: name: # this is a normal metadata name. A descriptive name is recommended. This will be used in a tenet vCluster's config to reference the connector secret namespace: # this is the namespace that vCluster Platform is installed in. "vcluster-platform" and "loft" are common values. labels: loft.sh/connector-type: "shared-database" stringData: endpoint: # this could be a service endoint or an external hostname endpoint password: port: # 3306 is a common port for database servers user: ``` Use the following `kubectl` command to create the secret: Create a shared database connector secret ``` kubectl create secret generic default-secret-name \ --namespace=vcluster-platform \ --from-literal=endpoint="your-database-endpoint" \ --from-literal=password="your-database-password" \ --from-literal=port="3306" \ --from-literal=user="your-database-user" \ -o yaml --dry-run=client > secret.yaml ``` ## Use tenant cluster with shared database[​](#use-tenant-cluster-with-shared-database "Direct link to Use tenant cluster with shared database") A new tenant cluster can now reference the created connector secret by setting the `controlPlane.backingStore.database.external.connector` field in the tenant cluster config to the `name` of the connector secret. warning Modifying a database secret to point to a different database server is not recommended and may lead to data loss. ### Example configuration[​](#example-configuration "Direct link to Example configuration") The following example shows a connector secret for the shared database feature. It connects to an internal MySQL instance exposed by a service named `mysql` in the `vcluster-platform` namespace (where the platform is assumed to be installed). You can also use any other MySQL-compatible endpoint, such as an AWS RDS instance. Shared database secret example ``` apiVersion: v1 kind: Secret metadata: name: default-data-source namespace: vcluster-platform # This must match the namespace that vCluster Platform is in. labels: loft.sh/connector-type: "shared-database" stringData: endpoint: mysql.vcluster-platform # This service can be in any namespace. password: port: "3306" user: root type: mysql # This can be mysql or postgres. If left blank, the mysql database type is assumed. ``` ## PostgreSQL TLS[​](#postgresql-tls "Direct link to PostgreSQL TLS") info This feature is available from the **Platform** version v4.10.0 Starting with platform v4.10, the PostgreSQL admin connection uses `sslmode=require` as the minimum. Previously the connection silently fell back to plaintext when the server refused TLS. Two optional fields on the connector secret tune this behavior: | Field | Effect | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `caCert` | PEM-encoded CA bundle. When set, the connection uses `sslmode=verify-full` and the server certificate is validated against this CA. | | `sslMode` | Explicit override (`disable`, `require`, `verify-full`, and so on). Wins over the default derived from `caCert`. Set `sslMode: disable` to keep an internal-network PostgreSQL without TLS working after upgrading to v4.10. | The example below enables `sslmode=verify-full` by attaching the CA bundle that signed the PostgreSQL server certificate: Connector secret with TLS verification ``` apiVersion: v1 kind: Secret metadata: name: postgres-tls namespace: vcluster-platform labels: loft.sh/connector-type: "shared-database" stringData: endpoint: postgres.internal.svc.cluster.local port: "5432" user: admin password: type: postgres caCert: | -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- ``` ## Recreate / restore tenant clusters[​](#recreate--restore-tenant-clusters "Direct link to Recreate / restore tenant clusters") To accommodate infrastructure updates or disaster recovery needs, follow this process to restore tenant clusters using a **database connector** powered backing store. 1. Annotate the target VirtualClusterInstance with `vcluster.com/database-reclaim-policy: retain` so deleting the tenant cluster does not also delete the database. 2. Record the value of the `vcluster.com/database-uid` annotation and save the `spec` from the VirtualClusterInstance. 3. Delete the tenant cluster. 4. Create the tenant cluster with the same `vcluster.com/database-uid` annotation and the saved `spec`. Scope This procedure only restores the backing store connection. Depending on your configuration, resources such as persistent volumes or other external dependencies might not be available for the recreated tenant cluster. RBAC permissions Creating VirtualClusterInstance resources with the `vcluster.com/database-uid` annotation set requires elevated permissions. Sample ClusterRole ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: vcluster-database-admin rules: - apiGroups: [ "management.loft.sh" ] resources: [ "virtualclusterinstances/externaldatabase" ] verbs: [ "bind" ] ``` ### Sample bulk migration script[​](#sample-bulk-migration-script "Direct link to Sample bulk migration script") important This script assumes that every matched tenant cluster uses a database connector. If a matched tenant cluster does not use a database connector, its data is not preserved. Sample control plane host bulk migration script ``` # requires kubectl, jq, and access to the vCluster Platform api OLD_CLUSTER="old-cluster" NEW_CLUSTER="new-cluster" # select required instances running on old cluster kubectl get virtualclusterinstance.management.loft.sh -A -o json | jq --arg c "$OLD_CLUSTER" '.items=[.items[] | select(.spec.clusterRef.cluster==$c)]' > ./old-vcis.json # annotate with retain kubectl annotate -f ./old-vcis.json vcluster.com/database-reclaim-policy=retain --overwrite # prepare new vcis specs by setting the desired cluster and remove metadata and status fields cat ./old-vcis.json | jq --arg c "$NEW_CLUSTER" '.items=[.items[] | .spec.clusterRef.cluster=$c | del(.metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .metadata.generation, .status)]' > ./new-vcis.json # delete old VCIs kubectl delete -f ./old-vcis.json --wait # create new VCIs kubectl create -f ./new-vcis.json # wait for the new VCIs to be ready kubectl wait -f ./new-vcis.json --for=condition=Ready ``` --- # IAM Database Connector info This feature is available from the **Platform** version v4.5.0 and was introduced in **vCluster** version v0.30.0 The IAM Database Connector feature allows tenant clusters to use a shared database server as their backing store and authenticate using workload identity. This page describes how to set up a shared database connector for cloud providers, for general instructions on using database connectors, see the [Database Connectors](https://vcluster.com/docs/platform/administer/connector/database.md) page. ## AWS[​](#aws "Direct link to AWS") ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") #### RDS database requirements[​](#rds-database-requirements "Direct link to RDS database requirements") * RDS instance running MySQL or PostgreSQL **with IAM Database Authentication enabled** * PostgreSQL: version 14+ * MySQL: version 5.7.16+, or 8.0+ * A database admin user with `CREATE DATABASE`, `CREATE ROLE`, and `GRANT` privileges note AWS RDS Aurora instances are currently not supported. #### EKS requirements[​](#eks-requirements "Direct link to EKS requirements") * EKS cluster with the **EKS Pod Identity Agent** add-on installed * **EBS CSI Driver** add-on with IAM permissions (required for platform persistent storage) To verify Pod Identity Agent is installed: ``` aws eks describe-addon --cluster-name YOUR_CLUSTER --addon-name eks-pod-identity-agent --query 'addon.status' ``` Expected output: `"ACTIVE"` ### Network requirements[​](#network-requirements "Direct link to Network requirements") Pods in the EKS cluster must be able to connect to the RDS instance. For clusters using private subnets, the following VPC endpoints are required: | Endpoint | Service | Purpose | | ---------------------------- | ------- | ----------------------------------- | | `com.amazonaws..rds` | RDS API | RDS service access | | `com.amazonaws..sts` | STS | IAM token generation | | `com.amazonaws..iam` | IAM API | Platform IAM role/policy management | | `com.amazonaws..eks` | EKS API | Pod Identity (if private cluster) | | `com.amazonaws..ec2` | EC2 API | ENI management | #### Configure security groups[​](#configure-security-groups "Direct link to Configure security groups") The RDS security group must allow inbound connections from EKS pods. A common mistake is only allowing traffic from the EKS cluster security group, which may not cover pod traffic. Allow PostgreSQL from entire VPC CIDR ``` aws ec2 authorize-security-group-ingress \ --group-id YOUR_RDS_SG \ --protocol tcp \ --port 5432 \ --cidr 10.0.0.0/16 # Replace with your VPC CIDR ``` warning If you see `no pg_hba.conf entry for host "10.0.x.x"` errors, this typically indicates a security group or network connectivity issue, not a PostgreSQL configuration problem. ### Setup and prepare RDS database[​](#setup-and-prepare-rds-database "Direct link to Setup and prepare RDS database") Create an RDS Database and ensure it supports **Password and IAM based Authentication**. #### Verify connectivity from EKS[​](#verify-connectivity-from-eks "Direct link to Verify connectivity from EKS") Before configuring IAM authentication, verify that pods can connect to RDS: * PostgreSQL * MySQL Test connectivity from a pod ``` kubectl run psql-test --rm -i --restart=Never --image=postgres:15 -- bash -c ' export PGPASSWORD="your-password" export PGSSLMODE=require psql -h YOUR_RDS_ENDPOINT -U postgres -c "SELECT version();" ' ``` Expected output: PostgreSQL version string (e.g., `PostgreSQL 15.x on x86_64-pc-linux-gnu...`) If the connection fails: * **Timeout**: Check security groups and VPC routing * **SSL error** (`no pg_hba.conf entry... no encryption`): RDS requires SSL. Ensure `PGSSLMODE=require` is set * **Authentication failed**: Verify password. Use environment variables for passwords with special characters Test connectivity from a pod ``` kubectl run mysql-test --rm -i --restart=Never --image=mysql:8 -- bash -c ' mysql -h YOUR_RDS_ENDPOINT -u admin -p"your-password" --ssl-mode=REQUIRED -e "SELECT version();" ' ``` Expected output: MySQL version string (e.g., `8.0.x`) If the connection fails: * **Timeout**: Check security groups and VPC routing * **SSL error**: RDS requires SSL. Ensure `--ssl-mode=REQUIRED` is set * **Authentication failed**: Verify password. Use `-p"password"` format (no space) for special characters #### Grant IAM authentication permissions[​](#grant-iam-authentication-permissions "Direct link to Grant IAM authentication permissions") Once connectivity is verified, grant the admin user permission to connect via RDS IAM authentication. * PostgreSQL * MySQL The default user for PostgreSQL RDS instances is `postgres`, adjust accordingly if a different admin user is used. Grant IAM authentication permissions for PostgreSQL ``` GRANT rds_iam TO postgres ``` Expected output: `GRANT ROLE` If you see: * `ERROR: role "rds_iam" does not exist` - IAM Database Authentication is not enabled on the RDS instance. Enable it in the RDS console or via AWS CLI. * `ERROR: permission denied` - The connected user lacks GRANT privileges. The default user for MySQL RDS instances is `admin`, adjust accordingly if a different admin user is used. Grant IAM authentication permissions for MySQL ``` ALTER USER IF EXISTS admin IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS' ``` Expected output: `Query OK, 0 rows affected` If you see: * `ERROR 1524: Plugin 'AWSAuthenticationPlugin' is not loaded` - IAM Database Authentication is not enabled on the RDS instance. * `ERROR 1227: Access denied` - The connected user lacks ALTER privileges. Important: plan for the authentication change Once IAM authentication is granted to a user, password-based login for that user stops working. Before proceeding: 1. Create a separate admin user for password-based access if needed for maintenance 2. Or be prepared to use `aws rds generate-db-auth-token` for all future connections ### Create IAM policies and role[​](#create-iam-policies-and-role "Direct link to Create IAM policies and role") Next up, you need to allow the vCluster Platform to connect to the RDS instance using IAM authentication. This is done by creating an IAM Policy and attaching it to the IAM Role used by the vCluster Platform pods. It also needs permissions to manage IAM Roles, Policies, and EKS Pod Identity Associations for the tenant clusters. info A policy for each DB Connector / RDS Instance that the platform manages is required. The vCluster Platform must be restarted for newly attached policies to take effect. #### RDS DB Connect Policy[​](#rds-db-connect-policy "Direct link to RDS DB Connect Policy") This policy is required to allow the vCluster Platform to connect as a database user. This user must be able to create and delete users and databases (AKA superuser). For convenience, the default admin user can be used. Consult the respective database documentation for the permissions required when not using the admin user. A sample policy is shown below, where `REGION`, `ACCOUNT`, and `DB_RESOURCE_ID` are values specific to the RDS instance. `DB_USER` must match the admin user that the platform deployment will connect as. The `DB_RESOURCE_ID` can be retrieved by running `aws rds describe-db-instances --output json` and looking for the `DbiResourceId` or `DbClusterResourceId` field in the output, depending on your RDS instance type. For RDS Proxy, use the `aws rds describe-db-proxies --output json` command instead and look for the last part of the `DbProxyArn` field after the final colon (`:`). For example, if the `DbProxyArn` is `arn:aws:rds:us-east-1:123456789012:db-proxy:prx-0123a01b12345c0a`, the `DB_RESOURCE_ID` is `prx-0123a01b12345c0a`. RDS DB Connect Policy ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "rds-db:connect", "Resource": "arn:aws:rds-db:${REGION}:${ACCOUNT}:dbuser:${DB_RESOURCE_ID}/${DB_USER}" } ] } ``` #### IAM Management Policy[​](#iam-management-policy "Direct link to IAM Management Policy") This policy allows the vCluster Platform to manage IAM roles for tenant cluster database access. You can set this up with or without an AWS IAM permission boundary. Using a permission boundary is strongly recommended for production environments. Permission boundaries act as a safety net by defining the maximum permissions that any IAM role created by the platform can have. Even if an inline policy grants broader permissions, the permission boundary ensures those roles can never exceed the allowed scope (for example, only `rds-db:connect` to a specific database), even though vCluster Platform is able to create new IAM roles. This prevents privilege escalation and limits the blast radius if a role is compromised. * With permission boundary (recommended) * Without permission boundary ##### Permission Boundary Policy[​](#permission-boundary-policy "Direct link to Permission Boundary Policy") First, create the permission boundary policy. This policy defines the maximum set of permissions that any IAM role created by the platform can have. It should only allow `rds-db:connect` to the specific RDS instance. Replace `${REGION}`, `${ACCOUNT}`, and `${DB_RESOURCE_ID}` with your RDS instance values. Permission Boundary Policy ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "rds-db:connect" ], "Resource": "arn:aws:rds-db:${REGION}:${ACCOUNT}:dbuser:${DB_RESOURCE_ID}/u???????????????????????????????" } ] } ``` ##### IAM Management Policy (with permission boundary)[​](#iam-management-policy-with-permission-boundary "Direct link to IAM Management Policy (with permission boundary)") Next, create the IAM Management Policy. When using a permission boundary, this policy restricts role creation to only those roles that have the specified permission boundary attached. It also includes a deny statement that prevents the platform from modifying the permission boundary policy itself. Replace `${ACCOUNT}` with your AWS account ID and `${PERMISSION_BOUNDARY_POLICY_ARN}` with the ARN of the permission boundary policy created above. IAM Management Policy (with permission boundary) ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:TagRole", "iam:ListRoleTags", "iam:GetRole" ], "Resource": "arn:aws:iam::${ACCOUNT}:role/u???????????????????????????????" }, { "Effect": "Allow", "Action": [ "iam:PutRolePermissionsBoundary", "iam:CreateRole" ], "Condition": { "StringEquals": { "iam:PermissionsBoundary": "${PERMISSION_BOUNDARY_POLICY_ARN}" } }, "Resource": "arn:aws:iam::${ACCOUNT}:role/u???????????????????????????????" }, { "Sid": "NoBoundaryPolicyEdit", "Effect": "Deny", "Action": [ "iam:CreatePolicyVersion", "iam:DeletePolicy", "iam:DeletePolicyVersion", "iam:SetDefaultPolicyVersion" ], "Resource": "${PERMISSION_BOUNDARY_POLICY_ARN}" }, { "Effect": "Allow", "Action": [ "iam:GetRolePolicy", "iam:PutRolePolicy", "iam:DeleteRolePolicy", "iam:DeleteRole" ], "Resource": "*", "Condition": { "StringEquals": { "aws:ResourceTag/vcluster.com/managed-by": "platform" } } }, { "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": "arn:aws:iam::${ACCOUNT}:role/u???????????????????????????????", "Condition": { "StringEquals": { "iam:PassedToService": "pods.eks.amazonaws.com" } } } ] } ``` Without a permission boundary, the IAM Management Policy grants the platform broader permissions to manage IAM roles. This is simpler to set up but provides fewer guardrails and allows the platform to create roles with broader permissions than intended. Replace `${ACCOUNT}` with your AWS account ID. IAM Management Policy (without permission boundary) ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:TagRole", "iam:ListRoleTags", "iam:GetRole" ], "Resource": "arn:aws:iam::${ACCOUNT}:role/u???????????????????????????????" }, { "Effect": "Allow", "Action": [ "iam:PutRolePermissionsBoundary", "iam:CreateRole" ], "Resource": "arn:aws:iam::${ACCOUNT}:role/u???????????????????????????????" }, { "Effect": "Allow", "Action": [ "iam:GetRolePolicy", "iam:PutRolePolicy", "iam:DeleteRolePolicy", "iam:DeleteRole" ], "Resource": "*", "Condition": { "StringEquals": { "aws:ResourceTag/vcluster.com/managed-by": "platform" } } }, { "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": "arn:aws:iam::${ACCOUNT}:role/u???????????????????????????????", "Condition": { "StringEquals": { "iam:PassedToService": "pods.eks.amazonaws.com" } } } ] } ``` #### RDS Resource ID Lookup Policy[​](#rds-resource-id-lookup-policy "Direct link to RDS Resource ID Lookup Policy") This read-only policy allows the vCluster Platform to look up the `DB_RESOURCE_ID` for each DB Connector. Replace `${REGION}`, `${ACCOUNT}`, `${DB_INSTANCE_ID}`, and `${DB_PROXY_ID}` with your values. `${DB_INSTANCE_ID}` is the RDS instance identifier and `${DB_PROXY_ID}` is the RDS Proxy identifier. If you are not using RDS Proxy, you can remove the second statement. RDS Resource ID Lookup Policy ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "rds:DescribeDBInstances" ], "Resource": "arn:aws:rds:${REGION}:${ACCOUNT}:db:${DB_INSTANCE_ID}" }, { "Effect": "Allow", "Action": [ "rds:DescribeDBProxies" ], "Resource": "arn:aws:rds:${REGION}:${ACCOUNT}:db-proxy:${DB_PROXY_ID}" } ] } ``` #### EKS Pod Identity Association Policy[​](#eks-pod-identity-association-policy "Direct link to EKS Pod Identity Association Policy") Manages EKS Pod Identity Associations for tenant clusters to allow them to assume the IAM Roles created for database access. Replace `${REGION}`, `${ACCOUNT}`, and `${EKS_CLUSTER_NAME}` with your values. EKS Pod Identity Association Policy ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "eks:CreatePodIdentityAssociation", "eks:ListPodIdentityAssociations" ], "Resource": "arn:aws:eks:${REGION}:${ACCOUNT}:cluster/${EKS_CLUSTER_NAME}" }, { "Effect": "Allow", "Action": [ "eks:TagResource", "eks:DescribePodIdentityAssociation", "eks:UpdatePodIdentityAssociation", "eks:DeletePodIdentityAssociation" ], "Resource": "arn:aws:eks:${REGION}:${ACCOUNT}:podidentityassociation/${EKS_CLUSTER_NAME}/*" } ] } ``` #### Assemble policies into role[​](#assemble-policies-into-role "Direct link to Assemble policies into role") Now that the required policies have been created, they need to be attached to an IAM Role used by the vCluster Platform pods. Create a new IAM role that **includes the four preceding policies** and set up a trust policy that allows the EKS Pod Identity Agent to assume the role. IAM Role Example ``` { "RoleName": "vcluster-platform-iam-database-role", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "pods.eks.amazonaws.com" }, "Action": ["sts:AssumeRole", "sts:TagSession"] } ] }, "Description": "Allows vCluster Platform running in Amazon EKS cluster to access RDS and IAM Resources." } ``` ### Create Pod Identity Association[​](#create-pod-identity-association "Direct link to Create Pod Identity Association") Now that the database user and roles are configured, it"s time to assign the role to the vCluster platform pods by running the following command. Replace the placeholders `EKS_CLUSTER_NAME` and `IAM_ROLE_ARN` with your own values. Create Pod Identity Association ``` aws eks create-pod-identity-association \ --cluster-name $EKS_CLUSTER_NAME \ --role-arn $IAM_ROLE_ARN \ --namespace vcluster-platform \ --service-account loft ``` warning After the association is created, restart the vCluster Platform pods to pick up the new role. ### Create connector secret[​](#create-connector-secret "Direct link to Create connector secret") A connector secret is a secret that can be used with a platform feature or integration. For the platform to start using a database server, a secret must be created that contains admin credentials for the server. The secret must: * Contain the `loft.sh/connector-type` label with the `shared-database` value * Reside in the same namespace as the platform * Contain the fields: * endpoint * port * user * identityProvider (only for IAM based authentication) To create a database connector secret in the UI, navigate to the sidebar and select **Databases** -> **Connect Database Secrets**. Then, select the **AWS IAM** authentication method and fill in the required fields. note The UI does not currently support setting a AWS IAM permission boundary. If you want to use such a permission boundary, you can add the annotation to the secret after it has been created by the UI, by using `kubectl annotate` for example. Alternatively, a Secret YAML manifest can be applied to the platform's control plane cluster. * With permission boundary * Without permission boundary Shared database secret template (with permission boundary) ``` apiVersion: v1 kind: Secret metadata: name: namespace: labels: loft.sh/connector-type: "shared-database" annotations: connector.platform.vcluster.com/aws-permission-boundary-policy-arn: stringData: endpoint: port: # 5432 for PostgreSQL, 3306 for MySQL user: # Database user with rds_iam granted type: # "mysql" or "postgres" identityProvider: aws ``` Shared database secret template (without permission boundary) ``` apiVersion: v1 kind: Secret metadata: name: namespace: labels: loft.sh/connector-type: "shared-database" stringData: endpoint: port: # 5432 for PostgreSQL, 3306 for MySQL user: # Database user with rds_iam granted type: # "mysql" or "postgres" identityProvider: aws ``` Use the following `kubectl` command to create the secret: * PostgreSQL * MySQL Create PostgreSQL connector secret ``` kubectl create secret generic my-rds-connector \ --namespace=vcluster-platform \ --from-literal=endpoint="your-rds-endpoint.region.rds.amazonaws.com" \ --from-literal=identityProvider="aws" \ --from-literal=port="5432" \ --from-literal=type="postgres" \ --from-literal=user="postgres" kubectl label secret my-rds-connector \ --namespace=vcluster-platform \ loft.sh/connector-type=shared-database # If using permission boundary, add the annotation: kubectl annotate secret my-rds-connector \ --namespace=vcluster-platform \ connector.platform.vcluster.com/aws-permission-boundary-policy-arn="YOUR_BOUNDARY_ARN" ``` Create MySQL connector secret ``` kubectl create secret generic my-rds-connector \ --namespace=vcluster-platform \ --from-literal=endpoint="your-rds-endpoint.region.rds.amazonaws.com" \ --from-literal=identityProvider="aws" \ --from-literal=port="3306" \ --from-literal=type="mysql" \ --from-literal=user="admin" kubectl label secret my-rds-connector \ --namespace=vcluster-platform \ loft.sh/connector-type=shared-database # If using permission boundary, add the annotation: kubectl annotate secret my-rds-connector \ --namespace=vcluster-platform \ connector.platform.vcluster.com/aws-permission-boundary-policy-arn="YOUR_BOUNDARY_ARN" ``` Verify the secret was created with the correct label: ``` kubectl get secrets -n vcluster-platform -l loft.sh/connector-type=shared-database ``` To use the secret in a tenant cluster, refer to the [Database Connectors](https://vcluster.com/docs/platform/administer/connector/database.md#use-tenant-cluster-with-shared-database) documentation. ### Troubleshoot common issues[​](#troubleshoot-common-issues "Direct link to Troubleshoot common issues") #### Connection errors[​](#connection-errors "Direct link to Connection errors") | Error | Cause | Solution | | ---------------------------------- | --------------------------------------- | ----------------------------------------------------------------------- | | `FATAL: PAM authentication failed` | IAM role not configured correctly | Verify Pod Identity Association exists and platform pods were restarted | | Connection timeout | Network issue | Check VPC endpoints (for private subnets) and security groups | | `no pg_hba.conf entry for host` | Security group blocking OR SSL required | Allow VPC CIDR in RDS security group; ensure SSL is used | | `permission denied for database` | DB user lacks privileges | Grant CREATE DATABASE, CREATE ROLE privileges to the user | #### IAM issues[​](#iam-issues "Direct link to IAM issues") Test IAM authentication manually: Generate IAM auth token ``` aws rds generate-db-auth-token \ --hostname YOUR_RDS_ENDPOINT \ --port 5432 \ --username postgres \ --region YOUR_REGION ``` If this fails with access denied, check: 1. The IAM role has the `rds-db:connect` permission 2. The resource ARN in the policy matches your RDS instance's resource ID 3. The database user matches the policy #### Pod Identity issues[​](#pod-identity-issues "Direct link to Pod Identity issues") Verify Pod Identity is working: ``` kubectl exec -it -n vcluster-platform deployment/loft -- env | grep AWS ``` Expected: `AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE` and `AWS_CONTAINER_CREDENTIALS_FULL_URI` should be set. If not set: 1. Verify Pod Identity Agent addon is installed: `aws eks describe-addon --cluster-name CLUSTER --addon-name eks-pod-identity-agent` 2. Verify association exists: `aws eks list-pod-identity-associations --cluster-name CLUSTER` 3. Restart the platform pods after creating the association --- # Inbox The Inbox is an administrator-only view for announcements that need vCluster Platform operator attention. It collects platform-generated messages such as product update notices, license warnings, and feature availability warnings in one place. ## Who can use the Inbox[​](#who-can-use-the-inbox "Direct link to Who can use the Inbox") Only platform administrators can open the Inbox. Administrators can access it from the **Inbox** entry in the left navigation, just above the username. When unread announcements are available, the entry shows an unread count. Users who aren't administrators don't see the Inbox entry in the left navigation. If they open the Inbox URL directly, the platform redirects them to the home page. ## How the Inbox is populated[​](#how-the-inbox-is-populated "Direct link to How the Inbox is populated") The platform generates announcements based on version, license, and feature state. The UI polls for new announcements every 60 seconds. New unread announcements appear in the Inbox and update the unread count in the left navigation. The Inbox can include: | Announcement type | When it appears | | -------------------------- | ------------------------------------------------------------------------------------------------- | | Product update | A newer vCluster Platform version is available. | | License announcement | The license server returns an announcement for the platform instance. | | License limit warning | An offline license has passed 90% of a resource limit, such as an active or total resource count. | | License expiration warning | An offline license is expired or expires within 90 days. | | Feature warning | The platform detects use of a feature that is not included in the current license. | ## Clear announcements[​](#clear-announcements "Direct link to Clear announcements") Announcements are generated dynamically on each poll. When the platform detects that the underlying condition is resolved, the announcement is removed from the Inbox at the next 60-second poll. Administrators don't need to manually mark resolved announcements as read. To clear a specific announcement, resolve the underlying condition: * Upgrade vCluster Platform to clear product update announcements. * Update or renew the license to clear license expiration and limit warnings. * Disable unlicensed features or use a license tier that includes those features to clear feature warnings. note The Inbox does not support custom messages. You can add custom navigation items that link to your own announcement or support pages using [custom branding](https://vcluster.com/docs/platform/configure/platform-configs/custom-branding.md), but these don't add messages to the Inbox. ## Read and unread state[​](#read-and-unread-state "Direct link to Read and unread state") The Inbox tracks whether each administrator has read an announcement. Opening an announcement marks it as read for the current administrator in the current browser. Administrators can also select **Mark all as read** from the Inbox toolbar. Read state is stored in browser local storage. If an administrator changes browsers, clears browser storage, or uses a different device, previously read announcements can appear as unread again. ## API access[​](#api-access "Direct link to API access") Administrators and roles with `list` access to `announcements` can retrieve Inbox announcement data through the platform API. Use this to integrate license and version alerts into external monitoring or alerting pipelines. `Announcement` objects belong to the `management.loft.sh/v1` API group: ``` kubectl get announcements.management.loft.sh ``` There is no built-in webhook or email notification for new announcements. For role-based access control (RBAC) details, see the [Cluster Role Template resource documentation](https://vcluster.com/docs/platform/api/resources/clusterroletemplate.md#platform-resources). --- # Overview | | | Enterprise | | | | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Node Profiles](https://vcluster.com/docs/platform/administer/node-profiles/overview.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). A NodeProfile is a named, reusable configuration for a class of nodes, for example every GPU training node. Reference the same profile from a [manual join](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md) or a static or dynamic [auto-node pool](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/auto-nodes.md), and every node gets that configuration, regardless of how it joined. The platform doesn't just apply it once. It keeps every node that references the profile in sync as the definition changes, converging edits onto already-joined nodes without re-provisioning them. Private nodes only NodeProfiles apply to private nodes only. They can't be used by tenant clusters whose control plane runs on a shared or standalone control plane cluster, or by cloud-hosted tenant clusters. ## Use a NodeProfile[​](#use-a-nodeprofile "Direct link to Use a NodeProfile") ### Define a NodeProfile[​](#define-a-nodeprofile "Direct link to Define a NodeProfile") Suppose every GPU training node needs the same label, ownership annotation, and taint, no matter how it joins the cluster. Define a NodeProfile: ``` apiVersion: management.loft.sh/v1 kind: NodeProfile metadata: name: gpu-training spec: displayName: GPU Training description: | GPU worker defaults for training workloads. nodeLabels: workload.vcluster.com/class: gpu nodeAnnotations: ops.vcluster.com/owner: ml-platform taints: - key: nvidia.com/gpu value: "true" effect: NoSchedule startupTaints: - key: node.vcluster.com/bootstrap value: "true" effect: NoSchedule ``` `nodeLabels` and `nodeAnnotations` are reconciled onto the Node object continuously, so they can change later without editing each node directly. The `taints` entry keeps non-GPU pods from scheduling on the node permanently. The `startupTaints` entry keeps the node out of scheduling only until it reaches `Ready`, then is removed automatically. ### Reference the profile[​](#reference-the-profile "Direct link to Reference the profile") For a manual join, select `gpu-training` from the profile dropdown when generating the join command. For an auto-node pool, reference it in `vcluster.yaml`: ``` # Inside your vcluster.yaml privateNodes: enabled: true autoNodes: - provider: my-node-provider static: - name: gpu-burst quantity: 2 profile: gpu-training ``` Either way, the node joins with the `workload.vcluster.com/class=gpu` label, the `ops.vcluster.com/owner=ml-platform` annotation, and the `nvidia.com/gpu=true:NoSchedule` taint already applied. The startup taint is removed once the node reaches `Ready`. From then on, the platform keeps the node's labels, annotations, and taints in sync with the profile automatically. ### Update the profile[​](#update-the-profile "Direct link to Update the profile") If compliance later requires every GPU node to carry an audit label, add it to the profile instead of editing every node: ``` apiVersion: management.loft.sh/v1 kind: NodeProfile metadata: name: gpu-training spec: displayName: GPU Training description: | GPU worker defaults for training workloads. nodeLabels: workload.vcluster.com/class: gpu compliance.vcluster.com/audited: "true" # added nodeAnnotations: ops.vcluster.com/owner: ml-platform taints: - key: nvidia.com/gpu value: "true" effect: NoSchedule startupTaints: - key: node.vcluster.com/bootstrap value: "true" effect: NoSchedule ``` Every node that references `gpu-training` picks up the new label automatically. No node needs to be re-provisioned, and no one has to edit individual nodes. ## Spec fields[​](#spec-fields "Direct link to Spec fields") All fields in `spec` are optional. An empty profile is valid but has no effect. | Field | Description | When applied | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | | `displayName` | Human-readable name shown in the platform UI. | N/A | | `description` | Free-form text shown alongside the profile in selectors. | N/A | | `nodeLabels` | Labels reconciled onto the Node object. | Continuously | | `nodeAnnotations` | Annotations reconciled onto the Node object. | Continuously | | `taints` | Steady-state taints reconciled onto the Node object. | Continuously | | `startupTaints` | Taints applied when the node joins and removed once the node reaches `Ready`. Used to hold a node from receiving traffic until join-time setup finishes. | Once, at join | ## Continuous reconciliation[​](#continuous-reconciliation "Direct link to Continuous reconciliation") Each tenant cluster runs a reconciler that watches its assigned private worker nodes and keeps them aligned with the NodeProfile they reference. The reconciler only manages the keys it owns: * Adding a label, annotation, or taint to a profile applies it to every node referencing that profile. * Removing a key from a profile removes it from those nodes. * Labels, annotations, and taints that a profile never set are left untouched, even if they were added by other means. The reconciler tracks which keys it owns using annotations on the node itself, kept separately for labels, annotations, and taints. On each reconcile, it checks that record to know what to prune if a key disappears from the profile, rather than guessing from the node's current state. A key can still collide with a value the profile doesn't own. For example, an auto-node pool might set a taint directly, using the same key as a taint the profile also defines. When that happens, the profile's value wins. The reconciler overwrites the conflicting value on every reconcile without returning an error or warning. `startupTaints` are the exception. They're applied once, at join, and removed once the node becomes `Ready`. They aren't part of the steady-state reconciliation loop, so editing `startupTaints` on a profile has no effect on nodes that already joined. Convergence is prompt, not tied to a fixed interval. The reconciler watches for profile changes and re-converges affected nodes as soon as they occur. If the tenant cluster's connection to the platform is temporarily unavailable, convergence resumes once the connection is restored. ## Assignment and precedence[​](#assignment-and-precedence "Direct link to Assignment and precedence") A node resolves to exactly one profile. A profile is reusable across many nodes, but a single node is never assigned more than one. The platform resolves the profile for a node in this order, using the first match: 1. An explicit profile set while creating manual join command, or a `profile` set on the auto-node pool that provisioned the node. 2. The tenant cluster's `privateNodes.defaultProfile`. 3. The project's default node profile, configured through [`spec.defaultNodeProfile`](https://vcluster.com/docs/platform/administer/projects/allowed/node-profiles.md). If none of these resolve, the node joins without a profile, which is a valid state. A NodeProfile is a cluster-scoped object, like NodeProvider and NodeType. Without project-level restrictions, a project could reference a profile created for a different team or purpose. The allowlist prevents that. All three sources are validated against the project's [allowed node profiles](https://vcluster.com/docs/platform/administer/projects/allowed/node-profiles.md), and a profile that isn't allowed is rejected, even if explicitly requested. ### Tamper-proof assignment[​](#tamper-proof-assignment "Direct link to Tamper-proof assignment") An explicit profile selection, made at join time or on the auto-node pool, is baked into the join command or `NodeClaim` as the `vcluster.com/node-profile` label. Because this label is set through the normal node join process, the reconciler treats it as untrusted until it validates the value against the project's allowlist. Once validated, the reconciler promotes the assignment to `node-restriction.kubernetes.io/node-profile`, a [NodeRestriction](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#noderestriction)-protected label that a node can't set or remove on itself. This prevents a compromised or misconfigured node from granting itself a different profile's labels and taints. A profile resolved through the default-profile fallback chain, rather than an explicit selection, isn't promoted this way. Defaults stay dynamic, so changing or clearing the tenant cluster's or project's default re-converges the affected nodes without requiring a new join. ## How NodeProfile relates to other node objects[​](#how-nodeprofile-relates-to-other-node-objects "Direct link to How NodeProfile relates to other node objects") [Node providers](https://vcluster.com/docs/platform/administer/node-providers/overview.md) and [node types](https://vcluster.com/docs/platform/administer/node-providers/overview.md#node-types) describe how a node is provisioned, which infrastructure it comes from and what hardware it has. NodeProfile describes what the node looks like once it's part of the tenant cluster, independent of where it came from. A manually joined node has no node provider or node type at all, but can still reference a NodeProfile. A node from an auto-node pool has both. Without a shared profile, nothing ties these nodes together, and the resulting Node objects carry no identity in common. A NodeProfile closes that gap. Every node that references the same profile carries the same `vcluster.com/node-profile=` label, no matter how it joined. For example, running `kubectl get nodes` with the label selector `vcluster.com/node-profile=gpu-training` returns every GPU training node in the tenant cluster, regardless of how each one joined. --- # Nvidia Base Command Manager | | | Enterprise | | | | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Nvidia BCM Node Provider](https://vcluster.com/docs/platform/administer/node-providers/bcm.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). In order for this provider to work you need the following prerequisites: * Access to Nvidia Base Command Manager * vCluster Platform needs to be able to reach BCM control server * `admin.key` and `admin.pem` (or similar certificates with access) which can be found at `~/.cm/admin.key` and `~/.cm/admin.pem` on the BCM head node If the above prerequisites are fulfilled, vCluster platform will be able to schedule nodes you specify within the provider as worker nodes for vCluster. BCM nodes can be either referenced directly using their hostname or using a node group. vCluster platform will restart (and wipe their contents) as soon as the node is deprovisioned to allow safe reuse of nodes between vCluster.
## Create secret with BCM credentials[​](#create-secret-with-bcm-credentials "Direct link to Create secret with BCM credentials") Create a secret in the following form to allow vCluster platform to connect to BCM: ``` apiVersion: v1 kind: Secret metadata: name: bcm-credentials namespace: vcluster-platform data: # Base64 encoded value of ~/.cm/admin.pem tls.crt: LS0t... # Base64 encoded value of ~/.cm/admin.key tls.key: LS0t... ``` ## Use BCM nodes and node groups[​](#use-bcm-nodes-and-node-groups "Direct link to Use BCM nodes and node groups") Apply the following node provider to use the BCM node group `my-node-group`: ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: bcm-provider spec: bcm: # Endpoint where the BCM api is reachable endpoint: "https://x.x.x.x:8081" # Credentials secret we created a step above secretRef: name: "bcm-credentials" namespace: "vcluster-platform" nodeTypes: - name: cpu-nodes nodeGroups: ["cpu-node-group"] # Optional cost value that helps scheduling # cost: 100 resources: cpu: 16 memory: 32Gi - name: gpu-nodes nodes: ["gpu-node-1", "gpu-node-2", "gpu-node-3"] resources: cpu: 64 memory: 64Gi nvidia.com/gpu: "1" ``` The platform will then auto-discover the amount of nodes in each node group. The node types can then be referenced in the `vcluster.yaml`: ``` privateNodes: enabled: true autoNodes: - provider: bcm-provider static: - name: cpu-pool quantity: 1 nodeTypeSelector: - property: vcluster.com/node-type value: cpu-nodes dynamic: - name: gpu-pool nodeTypeSelector: - property: vcluster.com/node-type value: gpu-nodes ``` ## Network configuration properties[​](#network-configuration-properties "Direct link to Network configuration properties") BCM node types support additional properties for network configuration: ### `bcm.vcluster.com/network-name`[​](#bcmvclustercomnetwork-name "Direct link to bcmvclustercomnetwork-name") **Type:** `string` **Optional:** **Required properties:** * [`bcm.vcluster.com/network-interface`](#bcmvclustercomnetwork-interface): Interface to configure Use this BCM network name to reconfigure the node in BCM with: * Node's management network * Node's boot interface network * Get the BCM network subnet and gateway information * Allocate IP from this network and assign to specified BCM network interface * Configure the host interface accordingly These settings are reverted to their original values when the NodeClaim is destroyed. The host network resets automatically when the node is recycled on power reset and thus boots. This property may be omitted when the `netris.vcluster.com/server-cluster` is set. ### `bcm.vcluster.com/network-interface`[​](#bcmvclustercomnetwork-interface "Direct link to bcmvclustercomnetwork-interface") **Type:** `string` **Required with:** `bcm.vcluster.com/network-name` or `netris.vcluster.com/server-cluster` The name of the node's boot network interface, for example `eth0`, `bond0`. ### `bcm.vcluster.com/network-ip-range`[​](#bcmvclustercomnetwork-ip-range "Direct link to bcmvclustercomnetwork-ip-range") **Type:** `string` (comma-separated IP ranges) Specifies IP ranges to use for allocation instead of using CIDR-based allocation. Format: `IP1-IP2,IP3-IP4` Example: `10.0.0.20-10.0.0.30` ## Netris integration[​](#netris-integration "Direct link to Netris integration") The BCM node provider can use the [Netris integration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/netris.md) for automated network configuration and IP management. The following properties require the integration to be enabled and [configured](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/netris.md). ### `netris.vcluster.com/server-cluster`[​](#netrisvclustercomserver-cluster "Direct link to netrisvclustercomserver-cluster") **Type:** `string` **Required properties:** * `bcm.vcluster.com/network-interface`: Interface to configure Specifies the [Netris server cluster](https://docs.netris.io/concepts/server-cluster) to use for this node. When set, vCluster Platform will: 1. Get the server cluster's V-net's subnet and gateway information 2. Allocate IP in this network and configure the host interface accordingly 3. Move the node to the specified Netris server cluster (and back to its original one on destroy) ### Example with Netris[​](#example-with-netris "Direct link to Example with Netris") ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: bcm-netris-provider spec: bcm: endpoint: "https://x.x.x.x:8081" secretRef: name: "bcm-credentials" namespace: "vcluster-platform" nodeTypes: - name: compute-nodes nodeGroups: ["compute-node-group"] properties: bcm.vcluster.com/network-interface: bond0 resources: cpu: 32 memory: 128Gi ``` Use in vCluster: ``` privateNodes: enabled: true autoNodes: - provider: bcm-provider static: - name: inference quantity: 2 nodeTypeSelector: - property: vcluster.com/node-type value: gpu-nodes properties: bcm.vcluster.com/network-name: dgxnet-vcluster-1 netris.vcluster.com/server-cluster: vcluster-1 ``` For more information on Netris integration, see the [Netris integration documentation](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/netris.md). --- # KubeVirt | | | Enterprise | | | | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [KubeVirt Node Provider](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). The KubeVirt provider allows you to use [KubeVirt](https://kubevirt.io/) to automatically provision virtual machines as nodes for your clusters. When a vCluster requests a new node, the platform creates a KubeVirt `VirtualMachine` based on the template defined in the `NodeProvider` and the specific `NodeType` requested. To inspect and operate KubeVirt VMs directly from vCluster Platform, see [Virtual Machines](https://vcluster.com/docs/platform/administer/virtual-machines/overview.md). For administrator guidance on VM disk imports and Multus networks, see [Data Volumes](https://vcluster.com/docs/platform/administer/virtual-machines/data-volumes.md) and [Network Attachment Definitions](https://vcluster.com/docs/platform/administer/virtual-machines/network-attachment-definitions.md). This enables you to offer different "flavors" of virtual machines, for example small, medium, and large compute shapes, as nodes for your vCluster, all managed from a central configuration. The platform turns a NodeClaim into a KubeVirt VirtualMachine on the control plane cluster, which joins the tenant cluster as a worker node through cloud-init. Deleting the NodeClaim reverses the flow. ## Overview[​](#overview "Direct link to Overview") The KubeVirt provider works by defining a base `virtualMachineTemplate` and a collection of `nodeTypes`. Each `nodeType` represents a specific kind of virtual machine you want to offer. When a NodeClaim is created for a vCluster, it references one of these `nodeTypes`. The platform then generates a VirtualMachine manifest by combining the base template with any customizations defined in the chosen `nodeType` and applies it to the target KubeVirt control plane cluster. This approach allows for customizations like: * **Resource Overrides**: Easily create node types with different amounts of CPU or memory. * **Template Merging**: Modify specific parts of the base template, like adding taints or labels. * **Template Replacement**: Define completely distinct virtual machine templates for specialized use cases. ## How it works: Cloud-init Node registration[​](#how-it-works-cloud-init-node-registration "Direct link to How it works: Cloud-init Node registration") For a virtual machine to automatically join a vCluster as a node, it needs to be configured with the correct vCluster address and join token. The KubeVirt provider automates this process using `cloud-init`. Here's the workflow: 1. When a NodeClaim is processed, the platform generates a `cloud-init` configuration containing the necessary registration scripts. 2. This configuration is stored in a Kubernetes Secret in the KubeVirt control plane cluster. 3. The provider injects this Secret into the VirtualMachine definition as a `cloudInitNoCloud` disk. 4. When the VM boots, `cloud-init` executes the script from the disk, configuring the machine and registering it as a node with the tenant cluster. Image Requirement This entire process depends on the guest OS image having **cloud-init installed and enabled**. Furthermore, the image must be compatible with KubeVirt's `cloudInitNoCloud` data source. Standard cloud images for distributions like Ubuntu are generally compatible. If you use custom images, you must ensure they meet this requirement. ## Infrastructure deployment[​](#infrastructure-deployment "Direct link to Infrastructure deployment") The KubeVirt provider can deploy the KubeVirt and CDI operators onto the control plane cluster, or use an installation that you manage yourself. By default the provider expects KubeVirt to already be installed on the target cluster. To have the provider install it, enable the deployment under `spec.kubeVirt.deploy.kubevirt`: ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: kubevirt-provider spec: kubeVirt: clusterRef: cluster: my-cluster namespace: vcluster-platform deploy: kubevirt: enabled: true helmValues: | kubevirt: operator: kubevirtVersion: v1.8.2 ``` **Deployment fields (NodeProvider `kubeVirt.deploy.kubevirt`):** | Field | Description | Default | | ------------ | -------------------------------------------------------------------------------------- | ------------------------------ | | `enabled` | Deploy the KubeVirt and CDI operators. Immutable after creation. | `false` | | `chartRepo` | Override the Helm chart repository. | `oci://ghcr.io/loft-sh/charts` | | `chart` | Override the Helm chart name. | `kubevirt` | | `version` | Override the Helm chart version. | Bundled version | | `helmValues` | Raw YAML applied to both the operators and the KubeVirt and CDI resources (see below). | — | **Configure the operators:** | Value | Description | Default | | -------------------------------------------------------------- | -------------------------------------------------------------------- | --------------------------------------------------- | | `kubevirt.operator.image.registry` / `.org` / `.name` / `.tag` | KubeVirt operator image. | `quay.io` / `kubevirt` / `virt-operator` / `v1.8.2` | | `kubevirt.operator.kubevirtVersion` | KubeVirt version the operator installs. | `v1.8.2` | | `cdi.operator.image.registry` / `.org` / `.name` / `.tag` | CDI operator image. | `quay.io` / `kubevirt` / `cdi-operator` / `v1.65.0` | | `cdi.operator.cdiVersion` | CDI version the operator installs. | `v1.65.0` | | `cdi.operator.serviceMonitorNamespace` | Namespace for the CDI `ServiceMonitor`. Empty disables its creation. | `""` | | `imagePullSecrets` | Pull secrets for the operator images. | `[]` | **Configure the KubeVirt and CDI resources:** The same `helmValues` also populate the `KubeVirt` and `CDI` resources that the provider applies. The platform renders `kubevirt.spec` verbatim into the `KubeVirt` resource `.spec`, and `cdi.spec` into the `CDI` resource `.spec`, so KubeVirt and CDI configuration such as feature gates is controllable from here. | Value | Description | Default | | --------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------ | | `kubevirt.spec` | Rendered into the `KubeVirt` resource `.spec`. | Minimal configuration, no feature gates. | | `cdi.spec` | Rendered into the `CDI` resource `.spec`. | Feature gates `HonorWaitForFirstConsumer` and `WebhookPvcRendering`, Linux node selectors. | For example, to enable a KubeVirt feature gate and add a CDI feature gate: ``` deploy: kubevirt: enabled: true helmValues: | kubevirt: spec: configuration: developerConfiguration: featureGates: - LiveMigration cdi: spec: config: featureGates: - HonorWaitForFirstConsumer - WebhookPvcRendering - ExpandDisks ``` Guarded against existing installations Before deploying, the platform checks the target cluster for the KubeVirt CRD (`kubevirts.kubevirt.io`). When KubeVirt is already present, the provider does not redeploy it. The provider removes KubeVirt on provider deletion only when it deployed it (`enabled: true`). `deploy.kubevirt.enabled`, `clusterRef.cluster` and `clusterRef.namespace` are immutable after creating the provider. To change them, create a new provider. ## Configuration[​](#configuration "Direct link to Configuration") A KubeVirt `NodeProvider` configuration consists of a reference to the control plane cluster, a base VM template, and a list of node types. ### Minimal example[​](#minimal-example "Direct link to Minimal example") Here is a minimal configuration for a KubeVirt provider. It defines a single node type that uses the base template without any modifications. ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: kubevirt-provider-minimal spec: displayName: "KubeVirt Minimal Provider" properties: kubevirt.vcluster.com/image-url: docker://quay.io/containerdisks/ubuntu:22.04 kubevirt.vcluster.com/root-disk-size: 20Gi kubeVirt: # clusterRef allows configuration of where KubeVirt is running and where VirtualMachines will be created. clusterRef: cluster: test-kubevirt-1 # name of the control plane cluster already connected to the platform namespace: vcluster-platform # namespace in this control plane cluster where VirtualMachines will be created # Base template for all Virtual Machines created by this provider virtualMachineTemplate: spec: template: spec: domain: resources: requests: cpu: "1" memory: "2Gi" # Define the types of nodes users can request nodeTypes: - name: "small-ubuntu-vm" displayName: "Small Ubuntu VM (KubeVirt)" maxCapacity: 10 ``` ### Choose where VirtualMachines are created[​](#choose-where-virtualmachines-are-created "Direct link to Choose where VirtualMachines are created") `spec.kubeVirt.namespaceStrategy` controls which namespace on the control plane cluster a node's `VirtualMachine` object is created in: * `Provider` (default): every `VirtualMachine` is created in the node provider's own namespace, `clusterRef.namespace`. * `namespaceStrategy: VirtualCluster`: each `VirtualMachine` is created in the namespace of the tenant cluster whose `NodeClaim` requested the node, instead of the provider's namespace. ``` spec: kubeVirt: namespaceStrategy: VirtualCluster ``` The VirtualCluster strategy falls back to the `Provider` behavior when it can't resolve a usable tenant cluster namespace within `clusterRef.cluster`. This happens in the following cases: * The `NodeClaim` has no tenant cluster reference. * The referenced VirtualClusterInstance doesn't exist. * The tenant cluster runs in a different connected cluster from the KubeVirt provider. * The tenant cluster is standalone and doesn't run in a namespace on the provider cluster. As a result, `namespaceStrategy: VirtualCluster` has no effect for standalone tenant clusters. ## Define NodeTypes[​](#define-nodetypes "Direct link to Define NodeTypes") The real power of the KubeVirt provider comes from its flexible `nodeTypes`. You can define multiple types, each with specific overrides or entire template replacements. ### Override resources[​](#override-resources "Direct link to Override resources") You can create a `NodeType` that inherits the base `virtualMachineTemplate` but specifies different CPU and memory resources. This is useful for offering different machine sizes. ``` nodeTypes: - name: "small-ubuntu-vm-2" displayName: "Small Ubuntu VM (KubeVirt)" maxCapacity: 10 # This NodeType uses the base template's resources (1 CPU, 2Gi Memory) - name: "mid-ubuntu-vm-2" displayName: "Mid Ubuntu VM (KubeVirt)" maxCapacity: 5 resources: cpu: 4 memory: 5Gi # This NodeType overrides the base template to provide 4 CPUs and 5Gi of memory ``` ### Merge template modifications[​](#merge-template-modifications "Direct link to Merge template modifications") The `mergeVirtualMachineTemplate` field allows you to provide a partial `VirtualMachine` template that will be strategically merged with the base template. This is ideal for changing specific attributes, like adding Kubernetes taints or labels, without redefining the entire VM. Merge Behavior The merge logic follows standard Kubernetes strategic merge patching. Arrays are typically replaced, while maps (objects) are merged. In this example, the `high-memory-node` type adds a taint and a label to the base template. ``` # ... (omitting provider spec for brevity) nodeTypes: - name: "high-memory-node" maxCapacity: 2 displayName: "High Memory Node (KubeVirt)" mergeVirtualMachineTemplate: metadata: labels: # This label will be added to the base labels workload: memory-intensive spec: template: spec: # Add a taint to ensure only specific pods schedule here taints: - key: "workload" value: "high-memory" effect: "NoSchedule" ``` ### Replace the entire template[​](#replace-the-entire-template "Direct link to Replace the entire template") For cases where a `NodeType` requires a fundamentally different configuration, you can use the `virtualMachineTemplate` field *inside* the `NodeType` definition. This completely ignores the base template and uses the specified one instead. ``` # ... (omitting provider spec for brevity) nodeTypes: - name: "different-vm-with-different-template-2" maxCapacity: 1 displayName: "Different VM with Different Template (KubeVirt)" # This template completely replaces the provider-level base template virtualMachineTemplate: metadata: labels: foo: baz # Note: the base label 'foo: bar' is not present spec: # ... (full, self-contained VirtualMachine spec) ``` ## Example: vCluster with static KubeVirt node[​](#example-vcluster-with-static-kubevirt-node "Direct link to Example: vCluster with static KubeVirt node") This example demonstrates a realistic multi-cluster scenario. We will configure a `NodeProvider` on the management cluster that provisions specialized VMs onto a separate, KubeVirt-enabled target cluster. 1. Verify prerequisites. Before you begin, ensure you have a KubeVirt-enabled Kubernetes cluster connected to your vCluster Platform management plane. You can verify this by listing the connected clusters. ``` kubectl get clusters ``` You should see your target cluster in the list. For this example, we'll assume it's named `test-kubevirt-1`. ``` NAME CREATED AT loft-cluster 2025-08-21T10:43:14Z test-kubevirt-1 2025-08-21T11:51:36Z test-no-kubevirt-2 2025-08-21T12:34:25Z ``` 2. Create the NodeProvider Apply the `NodeProvider` configuration to your management cluster. This configuration references the target cluster, `test-kubevirt-1`, and specifies that new VMs should be created in the `vcluster-platform` namespace on that cluster. ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: kubevirt-provider-advanced spec: displayName: "KubeVirt Advanced Provider" properties: kubevirt.vcluster.com/image-url: docker://quay.io/containerdisks/ubuntu:22.04 kubevirt.vcluster.com/root-disk-size: 20Gi kubeVirt: clusterRef: cluster: test-kubevirt-1 namespace: vcluster-platform virtualMachineTemplate: metadata: labels: provider: kubevirt spec: template: spec: domain: devices: interfaces: - name: default masquerade: {} resources: requests: cpu: "2" memory: "4Gi" networks: - name: default pod: {} nodeTypes: - name: "standard-node" displayName: "Standard Ubuntu VM (KubeVirt)" maxCapacity: 10 - name: "high-memory-node" maxCapacity: 5 displayName: "High Memory Node (KubeVirt)" resources: cpu: 4 memory: 16Gi mergeVirtualMachineTemplate: metadata: labels: workload: memory-intensive spec: template: spec: taints: - key: "workload" value: "high-memory" effect: "NoSchedule" ``` 3. Verify NodeTypes are Available After applying the `NodeProvider`, the platform processes it and creates corresponding `NodeType` resources. You can list these to confirm they are ready to be claimed. ``` kubectl get nodetype ``` The command will show that both the `standard-node` and the `high-memory-node` are in the `Available` phase. ``` NAME AVAILABLE TOTAL PROVIDER COST PHASE CREATED AT high-memory-node 5 5 kubevirt-provider-advanced 72 Available 2025-08-25T13:17:30Z standard-node 10 10 kubevirt-provider-advanced 28 Available 2025-08-25T13:17:30Z ``` 4. Create a vCluster with Auto-Provisioning Now, create a vCluster instance that is configured to automatically request a node upon creation. We'll do this by providing a values file that enables the `autoNodes` feature. First, create a file named `test-kubevirt-vc.yaml` with the following content. This tells the vCluster to automatically claim one node matching the `standard-node` type. ``` privateNodes: enabled: true autoNodes: - provider: my-provider static: - name: static-standard-nodes quantity: 1 nodeTypeSelector: - property: vcluster.com/node-type value: standard-node ``` Next, create the vCluster using this file. ``` vcluster create --driver platform --connect=false -f test-kubevirt-vc.yaml --cluster loft-cluster test-kubevirt-node-1 ``` 5. Verify NodeClaim. Once the vCluster is created, its `autoNodes` configuration will automatically generate a `NodeClaim`. You can verify this on the management cluster. ``` kubectl get nodeclaim -A ``` You will see a `NodeClaim` for the `standard-node` type, which eventually will reach `Available` state, indicating that the request has been fulfilled. ``` NAMESPACE NAME STATUS VCLUSTER NODETYPE CREATED AT p-default test-kubevirt-node-1-x49gd Available test-kubevirt-node-1 standard-node 2025-08-25T13:41:07Z ``` 6. Verify the VirtualMachine on the Target Cluster. The `NodeClaim` triggers the `NodeProvider` to create a `VirtualMachine`. Switch your `kubectl` context to the target KubeVirt cluster (`test-kubevirt-1`) and inspect the VM object. ``` # Switch context to the KubeVirt control plane cluster kubectx test-kubevirt-1 # Check for the running Virtual Machine kubectl -n vcluster-platform get vm ``` The output confirms that the `VirtualMachine` corresponding to the `NodeClaim` is running. ``` NAME AGE STATUS READY test-kubevirt-node-1-x49gd 3m1s Running True ``` 7. Verify the Node in the vCluster. Finally, connect to your vCluster and verify that the new node has successfully joined the cluster. ``` vcluster --driver platform connect test-kubevirt-node-1 ``` After connecting, check the nodes within the vCluster. ``` kubectl get nodes ``` The output will show the new node in the `Ready` state, confirming the node is up and ready for workloads. ``` NAME STATUS ROLES AGE VERSION test-kubevirt-node-1-9xm54 Ready <none> 27s v1.33.4 ``` ## Configuration properties[​](#configuration-properties "Direct link to Configuration properties") Properties are key-value pairs on the NodeProvider, NodeType, or NodeClaim that control provisioning behavior. NodeClaim properties take precedence over NodeType. NodeType properties take precedence over NodeProvider. ### Inline template[​](#inline-template "Direct link to Inline template") #### `kubevirt.vcluster.com/vm-template`[​](#kubevirtvclustercomvm-template "Direct link to kubevirtvclustercomvm-template") **Type:** `string` (inline YAML of a `VirtualMachine` resource) Allows a NodeClaim to fully specify its `VirtualMachine` template inline, bypassing the NodeProvider's `virtualMachineTemplate` and any NodeType. When this property is set on a NodeClaim, the provider skips the default NodeType-based scheduling and uses the supplied template directly. The claim does not need a `typeRef` in this case. This is useful when a tenant wants to create a one-off VM whose shape does not match any predefined NodeType. Standalone Machines only This property is currently only compatible with standalone Machines. It cannot be used with NodeClaims created by a tenant cluster's node pool. This is not enforced. Setting it on a node-pool claim will produce undefined behavior. ### Image configuration[​](#image-configuration "Direct link to Image configuration") The image configuration properties synthesize a root-disk `DataVolumeTemplate`, volume, and disk and prepend them to the VM template. The synthesized disk is prepended to the VM's disk list, so it takes boot precedence over any other OS disks defined in the template. For administrator workflows, ownership, and import troubleshooting, see [Data Volumes](https://vcluster.com/docs/platform/administer/virtual-machines/data-volumes.md). #### `vcluster.com/os-image`[​](#vclustercomos-image "Direct link to vclustercomos-image") **Type:** `string` References a [KubeVirt OSImage](https://vcluster.com/docs/platform/use-platform/machines/os-images.md#kubevirt) resource by name. The OSImage's properties merge into the NodeClaim's properties. If the OSImage sets `kubevirt.vcluster.com/image-url` or `kubevirt.vcluster.com/image-datasource`, the NodeClaim's source-related fields (`image-url`, `image-checksum`, `image-datasource`) are discarded before the overlay to avoid mismatched sources. #### `kubevirt.vcluster.com/image-url`[​](#kubevirtvclustercomimage-url "Direct link to kubevirtvclustercomimage-url") **Type:** `string` (HTTP(S) URL or `docker://` reference) URL of the OS image. HTTP and HTTPS URLs produce an `http` DataVolume source; `docker://` URLs produce a `registry` source. Configure either this property or `kubevirt.vcluster.com/image-datasource`. If both are present, the provider uses the `DataSource`. #### `kubevirt.vcluster.com/image-checksum`[​](#kubevirtvclustercomimage-checksum "Direct link to kubevirtvclustercomimage-checksum") **Type:** `string` Checksum of the OS image, used only when `image-url` is an HTTP(S) URL. #### `kubevirt.vcluster.com/image-datasource`[​](#kubevirtvclustercomimage-datasource "Direct link to kubevirtvclustercomimage-datasource") **Type:** `string` (`` or `/`) References a CDI `DataSource`. The provider sets the DataVolume's `sourceRef` to this DataSource. If you only supply ``, the provider looks up the DataSource in the node provider namespace from `spec.kubeVirt.clusterRef.namespace`. Use `/` to select a different namespace explicitly. Configure either this property or `kubevirt.vcluster.com/image-url`. If both are present, the provider uses the `DataSource` and ignores the URL and checksum. For a cross-namespace clone, the connected-cluster credentials the KubeVirt node provider uses must be able to create `datavolumes/source` or pods in the DataSource namespace. #### `kubevirt.vcluster.com/root-disk-size`[​](#kubevirtvclustercomroot-disk-size "Direct link to kubevirtvclustercomroot-disk-size") **Type:** `string` (Kubernetes quantity, for example `40Gi`) **Required:** Yes, whenever image configuration is used. Storage request for the synthesized root-disk DataVolume. Provisioning fails if image configuration is set without a root-disk size. #### `kubevirt.vcluster.com/image-datavolume-template`[​](#kubevirtvclustercomimage-datavolume-template "Direct link to kubevirtvclustercomimage-datavolume-template") **Type:** `string` (inline YAML) A partial `DataVolumeTemplate` that is deep-merged onto the defaults before the platform applies the image source and disk size. Use this to override fields like `storage.accessModes` or `storage.storageClassName`. The platform always sets the DataVolume's `source` and `sourceRef` based on the image properties. You cannot override them here. #### `kubevirt.vcluster.com/root-disk-template`[​](#kubevirtvclustercomroot-disk-template "Direct link to kubevirtvclustercomroot-disk-template") **Type:** `string` (inline YAML) A partial disk entry that is deep-merged onto the default root disk (`bus: virtio`) before being prepended to the VM's `domain.devices.disks` list. Use this to set the bus, boot order, or other disk fields. ### Network configuration[​](#network-configuration "Direct link to Network configuration") KubeVirt provisioning supports additional properties for network configuration with cloud-init: #### `kubevirt.vcluster.com/network-cidr`[​](#kubevirtvclustercomnetwork-cidr "Direct link to kubevirtvclustercomnetwork-cidr") **Type:** `string` (CIDR notation) Specifies a CIDR range for allocating IP addresses to the VM. Example: `10.100.0.0/24` **Required properties:** * `kubevirt.vcluster.com/network-nameservers`: DNS nameservers **Optional properties:** * `kubevirt.vcluster.com/network-interface`: Interface name in the VM (defaults to `enp1s0`) #### `kubevirt.vcluster.com/network-interface`[​](#kubevirtvclustercomnetwork-interface "Direct link to kubevirtvclustercomnetwork-interface") **Type:** `string` The name of the network interface in the VM to configure, for example `eth0`, `eth1`, or `enp1s0`. It should normally match the default name from the operating system's device enumeration. Optional for CIDR-based configuration. Defaults to `enp1s0` when unset. #### `kubevirt.vcluster.com/network-ip-range`[​](#kubevirtvclustercomnetwork-ip-range "Direct link to kubevirtvclustercomnetwork-ip-range") **Type:** `string` (comma-separated IP ranges) Specifies IP ranges to use for allocation instead of using CIDR-based allocation. Format: `IP1-IP2,IP3-IP4` Example: `10.0.0.20-10.0.0.30,10.0.0.40-10.0.0.50` #### `kubevirt.vcluster.com/network-nameservers`[​](#kubevirtvclustercomnetwork-nameservers "Direct link to kubevirtvclustercomnetwork-nameservers") **Type:** `string` (comma-separated) DNS nameservers to configure in the VM. Example: `8.8.8.8,8.8.4.4` #### `kubevirt.vcluster.com/network-data`[​](#kubevirtvclustercomnetwork-data "Direct link to kubevirtvclustercomnetwork-data") **Type:** `string` (base64-encoded cloud-init network-config) Allows providing a complete custom network configuration for cloud-init. When set, this overrides automatic network configuration. #### `kubevirt.vcluster.com/network-attachment-definition`[​](#kubevirt-vcluster-com-network-attachment-definition "Direct link to kubevirt-vcluster-com-network-attachment-definition") **Type:** `string` (format: `[/]`) References a Multus `NetworkAttachmentDefinition` that already exists in the target namespace on the control plane cluster. The provider injects a `multus` network and a corresponding interface into the VM template using this NAD. If a network or interface with the same name already exists in the template, it is overwritten. This property is the supported way to attach a VM to an additional network such as a bridge. Set it on the NodeType for VMs that should share a network, or on the NodeClaim for a one-off attachment. For UI workflows, examples, ownership, and NAD troubleshooting, see [Network Attachment Definitions](https://vcluster.com/docs/platform/administer/virtual-machines/network-attachment-definitions.md). The following properties tune the injected interface. They only take effect when `kubevirt.vcluster.com/network-attachment-definition` is set. #### `kubevirt.vcluster.com/interface-binding`[​](#kubevirtvclustercominterface-binding "Direct link to kubevirtvclustercominterface-binding") **Type:** `string` Sets the binding method for the injected interface. One of `bridge`, `masquerade`, or `sriov`. Defaults to `bridge`. KubeVirt removed several network bindings, including macvtap and slirp, from its in-tree API in v1.3, so this property now accepts only the remaining in-tree bindings and rejects any other value. Plugin-based bindings such as macvtap, passt, slirp, and custom bindings are selected through `kubevirt.vcluster.com/interface-template` instead. #### `kubevirt.vcluster.com/interface-template`[​](#kubevirtvclustercominterface-template "Direct link to kubevirtvclustercominterface-template") **Type:** `string` (YAML) A partial KubeVirt interface definition merged over the default injected interface (`{name: multus, bridge: {}}`). Use it for binding-specific options that `kubevirt.vcluster.com/interface-binding` does not cover. For a network binding plugin that is already [installed and registered with KubeVirt](https://kubevirt.io/user-guide/network/network_binding_plugins/), set `binding: {name: }` in this template. The interface name is always forced to `multus`, because KubeVirt requires the interface name to match the network name. #### `kubevirt.vcluster.com/multus-default`[​](#kubevirtvclustercommultus-default "Direct link to kubevirtvclustercommultus-default") **Type:** `string` (`"true"` or `"false"`) When `"true"`, the Multus network becomes the VM's default network, and the pod network and its interface are removed, since KubeVirt rejects a VM that has both. Defaults to `"false"`. ### Network environment[​](#network-environment "Direct link to Network environment") #### `vcluster.com/network-environment`[​](#vclustercomnetwork-environment "Direct link to vclustercomnetwork-environment") **Type:** `string` Names the [NodeEnvironment](https://vcluster.com/docs/platform/administer/node-providers/overview.md#node-environments) whose properties merge into the NodeClaim's effective property set. The platform resolves this network environment from the merged properties of NodeProvider, NodeType, and NodeClaim. When set, this property takes precedence over the typed `NodeClaim.spec.environmentRef` field. Use it to declare network environment defaults at the provider or type level, or to keep network configuration separate from provider and type definitions. ### SSH and user data[​](#ssh-and-user-data "Direct link to SSH and user data") The `vcluster.com/ssh-keys` and `vcluster.com/user-data` properties behave identically across providers. See the [metal3 reference](https://vcluster.com/docs/platform/administer/node-providers/metal3.md#ssh-and-user-data) for details. To render cloud-init from a reusable template, reference a [MachineConfigTemplate](https://vcluster.com/docs/platform/use-platform/machines/machine-config-templates.md) with the `vcluster.com/user-data-template-config` property. See [Machine Config Templates](https://vcluster.com/docs/platform/use-platform/machines/machine-config-templates.md) for the template syntax and available variables. ## Integrate with kube-ovn[​](#integrate-with-kube-ovn "Direct link to Integrate with kube-ovn") KubeVirt VMs can attach to [kube-ovn](https://kubeovn.github.io/docs/stable/en/) networks, so each set of auto-nodes runs on its own isolated, routable overlay network with OVN-native DHCP and optional external egress. ### Attach VMs to a kube-ovn network[​](#attach-vms-to-a-kube-ovn-network "Direct link to Attach VMs to a kube-ovn network") A VM joins a kube-ovn network the same way it joins any other network, through a Multus `NetworkAttachmentDefinition` that's set as the VM's default network with an interface binding. This works whether you created the kube-ovn network yourself, another system manages it, or the platform provisioned it for you. To have the platform create the network, see [Provision a kube-ovn network](#provision-a-kube-ovn-network). By default, the integration uses the `managedtap` interface binding instead of `bridge`. The `bridge` binding makes KubeVirt run its own DHCP inside the launcher pod, which answers the guest before kube-ovn can, shadowing OVN-native DHCP so the VM never receives the address and gateway the subnet serves. `managedtap` avoids this. The binding is overridable, so you can set a different one where it fits your network. The attachment is driven by these KubeVirt properties (see [Network configuration](#network-configuration)): * `kubevirt.vcluster.com/network-attachment-definition`: the kube-ovn Multus NAD to attach to. * `kubevirt.vcluster.com/multus-default`: `"true"` so the kube-ovn network is the VM's default network. * `kubevirt.vcluster.com/interface-template`: `{"binding":{"name":"managedtap"}}` to use the managedtap binding. When the NodeEnvironment carries the `kube-ovn.vcluster.com/subnet` property, the platform fills these in automatically, each only when you have not set it yourself. Set any of them explicitly on the NodeType or NodeClaim to override the default. * [`kubevirt.vcluster.com/network-attachment-definition`](#kubevirt-vcluster-com-network-attachment-definition) points at the `NetworkAttachmentDefinition` named after the subnet in the provider namespace (`/`). * `kubevirt.vcluster.com/multus-default` becomes `"true"`. * `kubevirt.vcluster.com/interface-template` selects the `managedtap` binding, for the reason described above. #### Register the managedtap binding[​](#register-the-managedtap-binding "Direct link to Register the managedtap binding") The integration uses `managedtap` by default, and registering it is required whenever you use it. The binding must be registered on the `KubeVirt` resource on the control plane cluster, and it requires KubeVirt v1.4 or later. If the default `managedtap` binding is not registered, provisioning fails. Override the binding on the NodeType or NodeClaim if you do not want to register it. Register the binding through the KubeVirt provider's `deploy.kubevirt.helmValues`. The platform renders `kubevirt.spec` verbatim into the `KubeVirt` resource `.spec`, so add the network binding under it: ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: kubevirt-provider spec: kubeVirt: deploy: kubevirt: enabled: true helmValues: | kubevirt: spec: configuration: network: binding: managedtap: domainAttachmentType: managedTap ``` If KubeVirt is managed outside the provider, register the same binding on your existing `KubeVirt` resource instead. Verify the binding registration The network binding registration follows the KubeVirt API, not a vCluster schema. Confirm the exact fields for your KubeVirt version against the [KubeVirt network binding plugin documentation](https://kubevirt.io/user-guide/network/network_binding_plugins/). ### Provision a kube-ovn network[​](#provision-a-kube-ovn-network "Direct link to Provision a kube-ovn network") Instead of pointing VMs at an existing kube-ovn network, you can have the platform create one for you. Set `kube-ovn.vcluster.com/vpc` on the [NodeEnvironment](https://vcluster.com/docs/platform/administer/node-providers/overview.md#node-environments) to provision the kube-ovn resources on the control plane cluster. This property gates provisioning. Without it, the platform creates nothing. Provisioning runs as part of the NodeEnvironment's infrastructure lifecycle, so the network exists before any VM uses the environment and is removed when the environment is deleted. A VM attaches to a network provisioned this way through the same [attachment properties](#attach-vms-to-a-kube-ovn-network). When the NodeEnvironment carries `kube-ovn.vcluster.com/subnet`, those attachment defaults are filled in automatically. #### `kube-ovn.vcluster.com/vpc`[​](#kube-ovnvclustercomvpc "Direct link to kube-ovnvclustercomvpc") **Type:** `string` Name of the kube-ovn `Vpc`. Setting it enables provisioning and creates the VPC, subnet, and `NetworkAttachmentDefinition`. When the VPC does not exist, the platform creates it. A pre-existing VPC is only modified when this NodeEnvironment manages it. #### `kube-ovn.vcluster.com/subnet`[​](#kube-ovnvclustercomsubnet "Direct link to kube-ovnvclustercomsubnet") **Type:** `string` **Required:** Yes, when `kube-ovn.vcluster.com/vpc` is set. Name of the kube-ovn `Subnet` provisioned inside the VPC. The platform also creates a `NetworkAttachmentDefinition` of the same name in the provider namespace that VMs attach to. #### `kube-ovn.vcluster.com/subnet-cidr`[​](#kube-ovnvclustercomsubnet-cidr "Direct link to kube-ovnvclustercomsubnet-cidr") **Type:** `string` (IPv4 CIDR) **Required:** Yes, when `kube-ovn.vcluster.com/vpc` is set. CIDR block for the subnet. Only IPv4 is supported; an IPv6 or dual-stack CIDR is rejected. The subnet's CIDR is immutable, so changing it on a live environment requires recreating the subnet. #### `kube-ovn.vcluster.com/external-subnet`[​](#kube-ovnvclustercomexternal-subnet "Direct link to kube-ovnvclustercomexternal-subnet") **Type:** `string` Name of a pre-existing external `Subnet`. Setting it wires the VPC's external connectivity and default route, using the external subnet's gateway as the next hop. The platform never creates or deletes this external subnet. Required for egress. #### `kube-ovn.vcluster.com/egress-enabled`[​](#kube-ovnvclustercomegress-enabled "Direct link to kube-ovnvclustercomegress-enabled") **Type:** `string` (`"true"` or `"false"`) When `"true"`, the platform creates the egress source NAT objects (`OvnEip` and `OvnSnatRule`) so VMs on the subnet can reach external networks. Requires `kube-ovn.vcluster.com/external-subnet`. Defaults to `"false"`. Disabling egress after it was enabled is not supported. The platform labels every object it provisions with `kube-ovn.vcluster.com/node-environment`, set to the NodeEnvironment's UID, and removes those objects by that label when the NodeEnvironment is deleted. Objects it did not create, such as a pre-existing VPC or the external subnet, are left untouched. ### Expose services with the kube-ovn cloud-controller-manager[​](#expose-services-with-the-kube-ovn-cloud-controller-manager "Direct link to Expose services with the kube-ovn cloud-controller-manager") Provisioning a kube-ovn network and attaching VMs to it gives them a VPC and overlay connectivity. To expose `LoadBalancer` services and direct pod routing on that same VPC, deploy the kube-ovn cloud-controller-manager in the tenant cluster. See [kube-ovn cloud-controller-manager](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/kube-ovn.md) for setup. The integration here builds the VPC and wires the VMs onto it; the cloud-controller-manager then provides LoadBalancer services and pod routing on top of it. ## Netris integration[​](#netris-integration "Direct link to Netris integration") You can integrate KubeVirt VMs with [Netris](https://www.netris.io/) for automated network configuration and IP management. ### `netris.vcluster.com/server-cluster`[​](#netrisvclustercomserver-cluster "Direct link to netrisvclustercomserver-cluster") **Type:** `string` Specifies the Netris server cluster to use for this VM. When set, vCluster Platform: 1. Queries the Netris server cluster to collect subnet, gateway and vxlan id information. 2. Allocates an IP address from the subnet. 3. Creates Multus NetworkAttachment definition `netris--` using the bridge plugin with the `nbr-` bridge. 4. Injects cloud-init network configuration with the allocated IP. 5. Injects a network and interface into the VM template through the [`kubevirt.vcluster.com/network-attachment-definition`](#kubevirt-vcluster-com-network-attachment-definition) property. The interface injection itself is a generic mechanism. Netris is one way to drive it, but any NodeType or NodeClaim can set `kubevirt.vcluster.com/network-attachment-definition` directly to attach a VM to an existing NAD without involving Netris. **Required properties:** * `kubevirt.vcluster.com/network-interface`: Interface to configure * `kubevirt.vcluster.com/network-nameservers`: DNS servers **Prerequisites:** * [Netris integration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/netris.md) must be enabled and configured in the vCluster configuration. * vCluster Platform license must include the Netris feature. * Bridge with the `nbr-` must exist on the control plane cluster nodes. **Optional properties:** * `kubevirt.vcluster.com/network-attachment-definition` If specified, the integration does not create another Multus NetworkAttachmentDefinition, but instead uses this one. ### Example with Netris and Multus[​](#example-with-netris-and-multus "Direct link to Example with Netris and Multus") ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: kubevirt-netris-provider spec: properties: kubevirt.vcluster.com/image-url: docker://quay.io/containerdisks/ubuntu:22.04 kubevirt.vcluster.com/root-disk-size: 20Gi kubeVirt: clusterRef: cluster: kubevirt-host namespace: vcluster-platform virtualMachineTemplate: spec: template: spec: domain: devices: interfaces: - name: multus bridge: {} resources: requests: cpu: "2" memory: "4Gi" networks: - name: multus multus: networkName: netris-bridge nodeTypes: - name: netris-vms maxCapacity: 2 properties: # Netris server cluster for network configuration netris.vcluster.com/server-cluster: vcluster-1 # Interface to configure in the VM kubevirt.vcluster.com/network-interface: enp1s0 # DNS servers kubevirt.vcluster.com/network-nameservers: "8.8.8.8,8.8.4.4" resources: cpu: 2 memory: 4Gi ``` Create a NetworkAttachmentDefinition in the KubeVirt control plane cluster: ``` apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: netris-bridge namespace: vcluster-platform spec: config: | { "cniVersion": "0.3.1", "type": "bridge", "bridge": "br-netris" } ``` Use in vCluster: ``` privateNodes: enabled: true autoNodes: - provider: kubevirt-netris-provider static: - name: netris-vms quantity: 2 nodeTypeSelector: - property: vcluster.com/node-type value: netris-vms # Enable Netris integration integrations: netris: enabled: true connector: netris-credentials ``` For more information on Netris integration, see the [Netris integration documentation](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/netris.md). --- # Metal3 | | | Enterprise | | | | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Metal3 Node Provider](https://vcluster.com/docs/platform/administer/node-providers/metal3.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). The Metal3 provider allows you to provision bare metal servers as Machines using [Metal3](https://metal3.io/) and [Ironic](https://ironicbaremetal.org/). When a Machine is requested, the platform claims an available `BareMetalHost` resource and configures it with the requested OS image and user data. Ironic then handles the PXE boot and OS installation on the physical server. This enables you to offer different configurations of bare metal servers, all managed through BareMetalHost resources on a control plane cluster. Machines can be used as private nodes for tenant clusters or provisioned independently. The platform writes the requested state to an available BareMetalHost. Metal3 and Ironic provision the external server, which joins a tenant cluster only when requested as a private node. Deleting the Machine returns the host to the available pool. ## Overview[​](#overview "Direct link to Overview") The Metal3 provider works by selecting available BareMetalHost resources on a control plane cluster and provisioning them with an OS image and user data configuration. Node types let you organize bare metal servers based on type, location, or other criteria. When the Machine is deleted, the platform powers the server off and clears its image, user data, and network data references. It restores pre-existing managed DHCP annotations and makes the BareMetalHost available for reuse. ## How it works: Provisioning[​](#how-it-works-provisioning "Direct link to How it works: Provisioning") After the platform selects a BareMetalHost, it prepares the server for provisioning: 1. The platform generates a user data configuration and stores it in a Kubernetes Secret on the control plane cluster. 2. The provider sets the BareMetalHost's `userData` reference to this Secret and the image information from the configured [OSImage](https://vcluster.com/docs/platform/use-platform/machines/os-images.md) or direct image properties. Ironic then provisions the server through power management, PXE boot, and an in-memory installer that writes the OS to disk. The server boots into the provisioned OS with the user data configuration applied. When a Machine is used as a private node for a tenant cluster, the user data includes registration scripts that automatically join the server to the tenant cluster. ## Infrastructure deployment[​](#infrastructure-deployment "Direct link to Infrastructure deployment") The Metal3 provider can deploy the required infrastructure components on the control plane cluster. Each component can be individually enabled and customized with Helm values. You can also manage Metal3 and Ironic separately. When you enable the bundled DHCP server with external Ironic, configure `proxy.ironicHttpUrl` and either `proxy.ironicApiUrl` or `proxy.apiAdvertiseUrl`. * Metal3 & Ironic * DHCP Server * Multus Bare metal provisioning and lifecycle management. Deploys the Bare Metal Operator, Ironic, an IPA image server, and MariaDB as separate workloads. Install [cert-manager](https://cert-manager.io/) on the control plane cluster before enabling this component. The Bare Metal Operator uses it for admission webhooks, and internal components use it for mutual TLS (mTLS). The Platform UI checks this prerequisite and warns when it is missing. Upgrading from a Metal3 chart earlier than 0.6.0 The default Metal3 chart version is `0.6.0`. This version replaces the legacy single-pod deployment with separate workloads and a MariaDB backend. Review this architecture and database change before upgrading an existing Metal3 deployment. To retain the legacy topology temporarily, pin `spec.metal3.deploy.metal3.version` to the version that is already deployed. The previous vCluster Platform default was `0.4.1`. The components use the following runtime topology: * Ironic conductors run active/active. They shard host ownership over a consistent hash ring and coordinate through JSON-RPC. * The managed MariaDB is a single-replica StatefulSet. For high availability, run an external MariaDB and set `mariadb.mode: external`. * The IPA image server serves public static boot content outside the mTLS mesh. **Helm values:** | Value | Description | Default | | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | | `ironic.replicas` | Number of Ironic conductors. Values greater than `1` run them active/active. | `2` | | `ironic.ipaAdvertiseIP` | Ironic LoadBalancer VIP advertised to the in-memory agent (IPA). Set it to enable real provisioning against physical hosts; empty keeps the Ironic API in-cluster only (for scale-out testing without a real host cycle). | `""` | | `ironic.service` | Applied as the Ironic Service spec, so any field can be set (for example `type: LoadBalancer`, `loadBalancerClass`, and a nested `metadata` map for annotations and labels). `name`, `namespace`, and `selector` are managed by the chart. Ports use chart defaults but can be replaced with a custom `ports` list. Empty leaves a default ClusterIP Service. | `{}` | | `ironic.image.registry` | Ironic container image registry. | `quay.io` | | `ironic.image.repository` | Ironic container image repository. | `metal3-io/ironic` | | `ironic.image.tag` | Ironic image tag | `release-35.0` | | `ipaImages.replicas` | Replicas of the IPA image server that serves the IPA kernel and initramfs. | `2` | | `mariadb.mode` | Database backend for Ironic: `statefulset` (single-replica managed MariaDB StatefulSet with its own PVC and cert-manager TLS) or `external` (connect to your own MariaDB). | `statefulset` | | `mariadb.externalHost` | Host Ironic connects to in `external` mode. Ignored otherwise. | `""` | | `mariadb.database` | Database name. | `ironic` | | `mariadb.username` | Database username Ironic uses. An empty value resolves to `ironic`. | `""` | | `mariadb.password` | Database password Ironic uses. In `statefulset` mode, an empty value generates a password on first install and preserves it across upgrades. In `external` mode, an empty value resolves to `ironic`. | Mode-dependent | | `mariadb.caCertSecretName` / `mariadb.caCertSecretKey` | `external` mode only. Secret and key holding the CA certificate used to verify the external MariaDB over TLS. When set, Ironic connects over TLS. | `""` / `ca.crt` | | `mariadb.persistence.size` | PVC size for the managed MariaDB in `statefulset` mode. | `10Gi` | | `bareMetalOperator.image.registry` | Bare Metal Operator image registry. | `quay.io` | | `bareMetalOperator.image.repository` | Bare Metal Operator image repository. | `metal3-io/baremetal-operator` | | `bareMetalOperator.image.tag` | Bare Metal Operator tag | `v0.13.1` | | `ipaDownloader.image.registry` | IPA image downloader registry. | `ghcr.io` | | `ipaDownloader.image.repository` | IPA image downloader repository. | `loft-sh/vcluster-platform-ipa-images` | | `ipaDownloader.image.tag` | IPA image downloader tag | `v0.2.2` | Set `ironic.ipaAdvertiseIP` to the VIP of the Ironic LoadBalancer Service so bare metal hosts can reach the Ironic agent endpoint during provisioning: ``` deploy: metal3: enabled: true helmValues: | ironic: replicas: 2 ipaAdvertiseIP: 192.168.100.10 service: type: LoadBalancer # The assigned address must match ipaAdvertiseIP. loadBalancerIP: 192.168.100.10 ``` For highly available MariaDB, run your own database and point Ironic at it: ``` deploy: metal3: enabled: true helmValues: | mariadb: mode: external externalHost: mariadb.example.internal database: ironic username: ironic password: ``` Handles PXE and HTTP Boot by acting as a proxy between bare metal servers and Ironic, which may reside in a different network. When the provider deploys Metal3, the platform configures the boot-image endpoint. Configure `proxy.apiAdvertiseUrl` to send IPA directly to the bundled Ironic agent endpoint. The values below target the vcluster-platform-dhcp-server chart `v0.17.0`. **Deployment fields (NodeProvider `metal3.deploy.dhcp`):** | Field | Description | Default | | ------------ | --------------------------------------- | ------------------------------- | | `enabled` | Deploy the DHCP server. | `false` | | `chartRepo` | Override the Helm chart repository. | `oci://ghcr.io/loft-sh/charts` | | `chart` | Override the Helm chart name. | `vcluster-platform-dhcp-server` | | `version` | Override the Helm chart version. | `0.17.0` | | `helmValues` | Raw YAML passed as values to the chart. | — | **Helm values:** | Value | Description | Default | | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | | `image.registry` | Container image registry. | `ghcr.io` | | `image.repository` | Container image repository. | `loft-sh/vcluster-platform-dhcp-server` | | `image.tag` | Container image tag. | Chart app version | | `image.pullPolicy` | Image pull policy. | `Always` | | `imagePullSecrets` | Pull secrets for the image. | `[]` | | `replicas` | Number of DHCP server replicas. Only supported with a DHCP relay in front of the server, since without one every replica answers the same broadcast. Replies are stateless (addresses come from the BareMetalHost, not a lease database), so any replica can serve any client. Incompatible with `networkAttachmentDefinition.enabled`; run under `hostNetwork` or plain pod networking. | `1` | | `hostNetwork` | Run the pod on the host network. When `true`, the pod also runs as root and the Multus NetworkAttachmentDefinition isn't attached. | `false` | | `podAnnotations` / `podLabels` | Extra pod metadata. | `{}` | | `resources` | Container resource requests and limits. | `{}` | | `nodeSelector` / `tolerations` / `affinity` | Pod scheduling constraints. | `{}` / `[]` / `{}` | | `securityContext` | Container security context. Overridden to `runAsUser: 0` when `hostNetwork` is `true`. | `NET_BIND_SERVICE` capability, `runAsNonRoot: true`, `runAsUser: 1000` | | `extraArgs` | Extra arguments appended to the server binary. Use to set the IPA inspector and installer kernel parameters. | `[]` | | `extraEnv` | Extra environment variables on the server container. | `[]` | | `networkAttachmentDefinition.enabled` | Attach a Multus secondary interface so the pod sits on the provisioning L2 and can answer broadcast DHCP. Ignored under `hostNetwork`. Set to `false` for ordinary pod networking with no Multus dependency; this only works behind a DHCP relay, and you must set `dhcp.serverIP` and `http.advertiseUrl` to an address clients can reach (typically the `provisioningService` load balancer). | `true` | | `networkAttachmentDefinition.vip` | Virtual IP with prefix for the DHCP server. Injected into the NetworkAttachmentDefinition under `ipam.static`. Ignored when `hostNetwork: true`. | `192.168.100.3/24` | | `networkAttachmentDefinition.resourceName` | Device-plugin resource the NetworkAttachmentDefinition binds to, stamped as the `k8s.v1.cni.cncf.io/resourceName` annotation so Multus passes the allocated device ID to the CNI plugin (for SR-IOV or other device-plugin interfaces). Empty leaves the annotation off. | `""` | | `networkAttachmentDefinition.config` | CNI configuration JSON for the NetworkAttachmentDefinition. Use `bridge` if control plane cluster nodes have a bridge attached to the provisioning network. Use `macvlan` if the bare metal servers are on the same network as the control plane cluster nodes. | bridge `br0` | | `dhcp.serverIP` | IP the DHCP server advertises as itself. | `$(SERVER_IP)` (pod env) | | `dhcp.port` | DHCP listen port. | `67` | | `dhcp.listenAddr` | DHCP listen address. Set to `$(SERVER_IP):{{ .Values.dhcp.port }}` under `hostNetwork` to avoid answering on shared bridges. Rendered with `tpl`. | `0.0.0.0:67` | | `tftp.port` | TFTP listen port. | `69` | | `tftp.listenAddr` | TFTP listen address. Set to `""` to disable TFTP entirely (HTTP Boot still works). Rendered with `tpl`. | `$(SERVER_IP):69` | | `http.port` | HTTP boot file server port. | `8080` | | `http.listenAddr` | HTTP boot file server listen address. Rendered with `tpl`. | `$(SERVER_IP):8080` | | `http.advertiseUrl` | Base URL clients use to fetch boot files. Also the default for `proxy.apiAdvertiseUrl`. Rendered with `tpl`. | `http://$(SERVER_IP):8080` | | `https.listenAddr` | HTTPS boot file server listen address. Set it to enable HTTPS for Redfish virtual media. Bind to `0.0.0.0` when the BMC reaches the server through an interface other than `SERVER_IP`. Rendered with `tpl`. | `""` | | `https.port` | HTTPS boot file server port. | `8443` | | `https.extraSans` | Additional IP addresses or DNS names for the proxy TLS certificate. Include the address that BMCs use for Redfish virtual media. | `[]` | | `provisioningService.type` | Service type exposing the DHCP, TFTP, and HTTP ports so a DHCP relay can unicast to one address instead of relying on broadcast. Usually `LoadBalancer`. Empty leaves the Service out entirely. Needed for `replicas > 1`. | `""` | | `provisioningService` (spec) | Interpreted as the Service spec, so any field can be set (`loadBalancerClass`, `externalTrafficPolicy`, and so on). A nested `metadata` map is lifted out and applied to the object's metadata. `ports` defaults to the DHCP, TFTP, and HTTP ports and may be overridden, but an override replaces the defaults wholesale, so it must repeat every port; `name`, `namespace`, and `selector` are always the chart's. | `{}` | | `proxy.callbackPort` | IPA callback proxy port. | `8081` | | `proxy.callbackListenAddr` | IPA callback proxy listen address. Rendered with `tpl`. | `$(POD_IP):8081` | | `proxy.callbackAdvertiseUrl` | URL Ironic uses to reach the IPA callback proxy. Rendered with `tpl`. | `https://ipa-callback..svc.cluster.local` | | `proxy.apiAdvertiseUrl` | Ironic API base URL written into the IPA kernel parameters. Empty defaults to `http.advertiseUrl`, which routes IPA through this proxy. To bypass the proxy with bundled Metal3, first configure `ironic.ipaAdvertiseIP`, then set this value to its agent endpoint, such as `https://192.168.100.10:6388`. Port `6385` is the in-cluster mTLS endpoint and isn't directly accessible to IPA. For externally managed Ironic, use an API root reachable from the provisioning network. Setting this value makes `proxy.ironicApiUrl` unnecessary. Rendered with `tpl`. | `""` | | `proxy.ironicHttpUrl` | Ironic HTTP endpoint the proxy forwards image and boot-file requests to. Always required. | Set to the bundled IPA image service when Metal3 is enabled | | `proxy.ironicApiUrl` | Ironic API endpoint the proxy forwards to. Optional when `proxy.apiAdvertiseUrl` is set. The generated URL for the bundled charts targets a port that the Ironic Service doesn't expose, so configure the direct `proxy.apiAdvertiseUrl` path for physical host provisioning. | Generated when Metal3 is enabled | **Server flags (via `extraArgs`):** | Flag | Description | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `--inspector-collectors` | Comma-separated list overriding the `ipa-inspection-collectors` kernel parameter in the inspector iPXE script. | | `--inspector-extra-kernel-param` | Extra kernel parameter appended to the inspector iPXE script. May be specified multiple times. | | `--installer-extra-kernel-param` | Extra kernel parameter appended to the installer iPXE script. May be specified multiple times. | | `--callback-insecure-skip-verify` | Skip TLS verification when the callback proxy forwards to IPA. | | `--api-advertise-url` | Ironic API base URL written into the IPA kernel parameters. Corresponds to `proxy.apiAdvertiseUrl`; defaults to the HTTP advertise URL. | | `--leader-election-id` | Name of the Lease electing which replica manages the proxy certificate Secret. The chart sets this flag automatically when `replicas` is greater than `1`; don't add it through `extraArgs`. The DHCP, TFTP, HTTP, and callback servers keep serving in every replica. | | `--tftp-single-port` | EXPERIMENTAL. Serve every TFTP transfer from the listen port instead of a per-transfer ephemeral port, so replies survive Service NAT. Required when TFTP traffic is reached through a Service. Some client firmware may reject replies from port 69, so it is off by default. | **Per-BareMetalHost DHCP annotations:** The DHCP server reads these annotations from the matched `BareMetalHost` to build the DHCP reply. | Annotation | Description | | --------------------------------- | ----------------------------------------------------------------------------------- | | `metal3.vcluster.com/ip-address` | IPv4 address with prefix (CIDR) to lease. Required. | | `metal3.vcluster.com/gateway` | Default gateway. If unset, the first host address in the `ip-address` CIDR is used. | | `metal3.vcluster.com/dns-servers` | Comma-separated DNS servers. | | `metal3.vcluster.com/ntp-servers` | Comma-separated NTP servers. | Example with a bridge: ``` deploy: dhcp: enabled: true helmValues: | networkAttachmentDefinition: vip: 192.168.100.2/24 config: | { "cniVersion": "0.3.1", "type": "bridge", "bridge": "br0", "isDefaultGateway": false } ``` Example with macvlan (bare metal servers on the same network as the control plane cluster nodes): ``` deploy: dhcp: enabled: true helmValues: | networkAttachmentDefinition: vip: 10.0.0.2/24 config: | { "cniVersion": "0.3.1", "type": "macvlan", "master": "eth0", "mode": "bridge" } ``` Example with `hostNetwork` and custom IPA kernel parameters: ``` deploy: dhcp: enabled: true helmValues: | hostNetwork: true dhcp: listenAddr: "$(SERVER_IP):67" extraArgs: - --inspector-extra-kernel-param=console=ttyS0,115200n8 - --installer-extra-kernel-param=console=ttyS0,115200n8 ``` #### High availability and DHCP relay[​](#high-availability-and-dhcp-relay "Direct link to High availability and DHCP relay") By default the DHCP server runs a single replica attached to the provisioning L2 through Multus, where it answers broadcast DHCP directly. This mode doesn't scale past one active answerer. To run more than one replica, place a DHCP relay in front of the server and expose it through a Service: * Set `replicas` greater than `1`. * Set `provisioningService.type: LoadBalancer` so the relay has a single unicast target that also serves boot files. * Point the relay at the `provisioningService` address, and set `dhcp.serverIP` and `http.advertiseUrl` to that same address so clients boot through the Service instead of from whichever replica answered. * Add the experimental `--tftp-single-port` so TFTP replies survive the Service NAT. The chart configures leader election automatically. * Disable Multus with `networkAttachmentDefinition.enabled: false`, since its static IPAM would hand the same VIP to every pod. ``` deploy: dhcp: enabled: true helmValues: | replicas: 2 networkAttachmentDefinition: enabled: false dhcp: serverIP: 192.168.100.20 http: advertiseUrl: http://192.168.100.20:8080 provisioningService: type: LoadBalancer metadata: annotations: metallb.universe.tf/loadBalancerIPs: 192.168.100.20 extraArgs: - --tftp-single-port ``` CNI plugin that enables attaching the DHCP server to a separate provisioning network. **Helm values:** | Value | Description | Default | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------- | | `image.registry` | Container image registry | `ghcr.io` | | `image.repository` | Container image repository | `k8snetworkplumbingwg/multus-cni` | | `image.tag` | Container image tag | `v4.2.4-thick` | | `resources` | CPU and memory requests and limits for the main Multus container. The installer init container has its own fixed resources and isn't affected by this value. | `100m` CPU and `50Mi` memory | The platform installs Multus into `metal3.clusterRef.namespace`. ``` deploy: multus: enabled: true ``` The DHCP server reads its lease data from BareMetalHost annotations. When network properties enable platform IPAM, the provider writes the allocated address and related network settings to those annotations. ## Configuration[​](#configuration "Direct link to Configuration") A Metal3 `NodeProvider` configuration consists of a cluster reference, optional infrastructure deployment settings, and a list of node types. ### Cluster reference[​](#cluster-reference "Direct link to Cluster reference") The `clusterRef` specifies the control plane cluster where the platform installs Metal3 components and where the BareMetalHost resources live. This cluster must be connected to vCluster Platform. | Field | Description | Required | | ---------------------- | ---------------------------------------------------------------------------------------- | -------- | | `clusterRef.cluster` | Name of the connected control plane cluster | Yes | | `clusterRef.namespace` | Namespace on the control plane cluster for Metal3 components and BareMetalHost resources | Yes | The cluster and namespace references are immutable after you create the provider. A connected control plane cluster can be referenced by only one Metal3 NodeProvider. Ironic network access Ironic must have network access to the BMC addresses of the bare metal servers. Ensure the control plane cluster where Ironic is deployed can reach the BMC network (Redfish/IPMI endpoints). ### Example[​](#example "Direct link to Example") This configuration deploys Metal3, Ironic, the DHCP server, and Multus, and defines a single node type that selects BareMetalHosts with the label `role: compute`. ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: metal3-provider spec: displayName: "Metal3 Bare Metal Provider" metal3: clusterRef: cluster: bare-metal-cluster namespace: metal3-system deploy: metal3: enabled: true helmValues: | ironic: # Expose the IPA API on a provisioning-network LoadBalancer VIP. ipaAdvertiseIP: 192.168.100.10 service: type: LoadBalancer loadBalancerIP: 192.168.100.10 dhcp: enabled: true helmValues: | networkAttachmentDefinition: # Attach the DHCP server to the provisioning network. vip: 192.168.100.2/24 proxy: # Send IPA directly to the bundled non-mTLS agent endpoint. apiAdvertiseUrl: https://192.168.100.10:6388 multus: enabled: true nodeTypes: - name: "compute-node" displayName: "Compute Node" resources: cpu: "32" memory: 128Gi bareMetalHosts: selector: matchLabels: role: compute properties: vcluster.com/os-image: ubuntu-noble ``` ## Get started[​](#get-started "Direct link to Get started") This walkthrough covers the essential steps to go from a connected control plane cluster to a provisioned bare metal server. Before you begin: * Install cert-manager on the control plane cluster. * Connect each control plane cluster node to the provisioning network through a `br0` bridge, or replace the DHCP NetworkAttachmentDefinition configuration. * Configure a load balancer implementation that can assign `192.168.100.10` to the Ironic Service. Replace all example network addresses with addresses from your provisioning network. * Create an [OSImage](https://vcluster.com/docs/platform/use-platform/machines/os-images.md#create-an-osimage) named `ubuntu-noble`, or change the node type to reference an existing Metal3-compatible OSImage. 1. Apply the NodeProvider. Create a NodeProvider that references your control plane cluster and defines at least one node type. The node type's label selector determines which BareMetalHosts it can claim. ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: metal3-provider spec: displayName: "Metal3 Bare Metal Provider" metal3: clusterRef: cluster: bare-metal-cluster namespace: metal3-system deploy: metal3: enabled: true helmValues: | ironic: # Expose the IPA API on a provisioning-network LoadBalancer VIP. ipaAdvertiseIP: 192.168.100.10 service: type: LoadBalancer loadBalancerIP: 192.168.100.10 dhcp: enabled: true helmValues: | networkAttachmentDefinition: # Attach the DHCP server to the provisioning network. vip: 192.168.100.2/24 proxy: # Send IPA directly to the bundled non-mTLS agent endpoint. apiAdvertiseUrl: https://192.168.100.10:6388 multus: enabled: true nodeTypes: - name: "compute-node" displayName: "Compute Node" resources: cpu: "32" memory: 128Gi bareMetalHosts: selector: matchLabels: role: compute properties: vcluster.com/os-image: ubuntu-noble ``` ``` kubectl apply -f metal3-provider.yaml ``` Wait for Metal3 and Ironic to be running on the control plane cluster before creating BareMetalHost resources. The Metal3 webhook must be ready to validate them. 2. Create BMC credentials. Create a Secret with the BMC username and password for your server. This Secret is referenced by the BareMetalHost resource. ``` apiVersion: v1 kind: Secret metadata: name: server-01-bmc namespace: metal3-system type: Opaque stringData: username: admin password: ``` ``` kubectl apply -f server-01-bmc-secret.yaml ``` 3. Create a BareMetalHost. Register the physical server by creating a BareMetalHost resource. The `bmc.address` scheme determines which driver Metal3 uses, such as Redfish or IPMI. The `bootMACAddress` identifies the NIC used for PXE boot. ``` apiVersion: metal3.io/v1alpha1 kind: BareMetalHost metadata: name: server-01 namespace: metal3-system labels: role: compute spec: bmc: address: redfish://192.168.1.100 credentialsName: server-01-bmc disableCertificateVerification: true bootMACAddress: "aa:bb:cc:dd:ee:01" ``` ``` kubectl apply -f server-01-bmh.yaml ``` The server moves through `registering` and `inspecting` states as Metal3 verifies BMC access and collects hardware inventory. 4. Verify the server reaches available state. Once the BareMetalHost passes inspection, it transitions to `available`. This means the server is registered, its hardware inventory is collected, and it is ready for provisioning. ``` kubectl get baremetalhost -n metal3-system ``` ``` NAME STATE CONSUMER ONLINE ERROR AGE server-01 available false 5m ``` 5. Create a tenant cluster that claims the server. Create a tenant cluster with private nodes configured to use the Metal3 provider. The platform provisions a Machine, which claims the BareMetalHost and installs the OS through Ironic. ``` privateNodes: enabled: true autoNodes: - provider: metal3-provider static: - name: compute-nodes quantity: 1 nodeTypeSelector: - property: vcluster.com/node-type value: compute-node ``` After provisioning completes, the server boots into the configured OS and joins the tenant cluster as a worker node. ## Define node types[​](#define-node-types "Direct link to Define node types") Each node type specifies which BareMetalHosts it can claim and what properties to apply during provisioning. ### Select BareMetalHosts by label[​](#select-baremetalhosts-by-label "Direct link to Select BareMetalHosts by label") Use `bareMetalHosts.selector` to match BareMetalHosts by labels. The provider checks requested CPU and memory against the inspected BareMetalHost inventory. Other resource keys don't affect BareMetalHost selection, so use labels to select hardware such as GPUs. ``` nodeTypes: - name: "gpu-server" displayName: "GPU Server" resources: cpu: "64" memory: 256Gi bareMetalHosts: selector: matchLabels: role: gpu datacenter: us-east - name: "general-compute" displayName: "General Compute" resources: cpu: "32" memory: 128Gi bareMetalHosts: selector: matchLabels: role: compute ``` ## Configuration properties[​](#configuration-properties "Direct link to Configuration properties") Properties are key-value pairs on node types or Machines that control provisioning behavior. ### Image configuration[​](#image-configuration "Direct link to Image configuration") #### `vcluster.com/os-image`[​](#vclustercomos-image "Direct link to vclustercomos-image") **Type:** `string` References an [OSImage](https://vcluster.com/docs/platform/use-platform/machines/os-images.md) resource by name. The OSImage's properties are used to configure the image URL, checksum, and checksum type. This is the recommended way to configure OS images. #### `metal3.vcluster.com/image-url`[​](#metal3vclustercomimage-url "Direct link to metal3vclustercomimage-url") **Type:** `string` **Required:** Yes (if not using `vcluster.com/os-image`) Direct URL to the OS image. Example: `https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img` #### `metal3.vcluster.com/image-checksum`[​](#metal3vclustercomimage-checksum "Direct link to metal3vclustercomimage-checksum") **Type:** `string` **Required:** Yes (if not using `vcluster.com/os-image`) Checksum of the OS image for verification. #### `metal3.vcluster.com/image-checksum-type`[​](#metal3vclustercomimage-checksum-type "Direct link to metal3vclustercomimage-checksum-type") **Type:** `string` Checksum algorithm. Supported values are `md5`, `sha256`, `sha512`, and `auto`. If omitted, Ironic defaults to `md5`. Set it to `auto` to have Ironic detect the algorithm from the checksum. ### Network configuration[​](#network-configuration "Direct link to Network configuration") #### `metal3.vcluster.com/network-cidr`[​](#metal3vclustercomnetwork-cidr "Direct link to metal3vclustercomnetwork-cidr") **Type:** `string` (standard CIDR notation) Specifies the gateway IP and subnet for IP allocation. Use standard CIDR notation where the host portion is the gateway address. The platform allocates IPs from the resulting subnet using its built-in IPAM. Example: `10.0.0.1/24` #### `metal3.vcluster.com/network-ip-range`[​](#metal3vclustercomnetwork-ip-range "Direct link to metal3vclustercomnetwork-ip-range") **Type:** `string` (comma-separated IP ranges) Restricts allocation to explicit IP ranges. The `metal3.vcluster.com/network-cidr` property remains required because it supplies the gateway and prefix. Keep every range within that subnet. Format: `IP1-IP2,IP3-IP4` Example: `10.0.0.20-10.0.0.30,10.0.0.40-10.0.0.50` #### `metal3.vcluster.com/dns-servers`[​](#metal3vclustercomdns-servers "Direct link to metal3vclustercomdns-servers") **Type:** `string` (comma-separated) DNS servers to configure on the provisioned server. Example: `8.8.8.8,8.8.4.4` #### `metal3.vcluster.com/network-data`[​](#metal3vclustercomnetwork-data "Direct link to metal3vclustercomnetwork-data") **Type:** `string` Complete custom network-data configuration for the installed operating system. When set, it replaces the network-data document that the provider would generate. CIDR and IP range properties still control address allocation and provisioning DHCP. #### `vcluster.com/network-data-template-config`[​](#vclustercomnetwork-data-template-config "Direct link to vclustercomnetwork-data-template-config") **Type:** `string` References a [MachineConfigTemplate](https://vcluster.com/docs/platform/use-platform/machines/machine-config-templates.md) by name. The provider renders the template's `networkDataTemplate` into a cloud-init `network-config` document when `metal3.vcluster.com/network-data` isn't set. See [Machine Config Templates](https://vcluster.com/docs/platform/use-platform/machines/machine-config-templates.md) for the template syntax and available variables. ### Network environment[​](#network-environment "Direct link to Network environment") #### `vcluster.com/network-environment`[​](#vclustercomnetwork-environment "Direct link to vclustercomnetwork-environment") **Type:** `string` Names the network environment, a [NodeEnvironment](https://vcluster.com/docs/platform/administer/node-providers/overview.md#node-environments) resource whose properties are merged into the NodeClaim's effective property set. The platform resolves this from the merged properties of NodeProvider, NodeType, and NodeClaim. When set, this property takes precedence over the typed `NodeClaim.spec.environmentRef` field. ### Server pinning[​](#server-pinning "Direct link to Server pinning") #### `metal3.vcluster.com/server-name`[​](#metal3vclustercomserver-name "Direct link to metal3vclustercomserver-name") **Type:** `string` Pins a Machine to a specific BareMetalHost by name. This is useful when creating a Machine directly and you want to target a particular server. Set this property on the Machine, not on the node type. ### SSH and user data[​](#ssh-and-user-data "Direct link to SSH and user data") #### `vcluster.com/ssh-keys`[​](#vclustercomssh-keys "Direct link to vclustercomssh-keys") **Type:** `string` (comma-separated) References [SSHKey](https://vcluster.com/docs/platform/use-platform/machines/ssh-keys.md) resources by name. The public keys are included in the user data during provisioning. #### `vcluster.com/user-data`[​](#vclustercomuser-data "Direct link to vclustercomuser-data") **Type:** `string` Custom cloud-init configuration merged with the generated user data. Accepts any valid cloud-config directives, including `packages`, `write_files`, and `runcmd`. The platform's provisioning commands are appended after any user-supplied `runcmd` entries. Takes precedence over `vcluster.com/user-data-template-config`. #### `vcluster.com/user-data-template-config`[​](#vclustercomuser-data-template-config "Direct link to vclustercomuser-data-template-config") **Type:** `string` References a [MachineConfigTemplate](https://vcluster.com/docs/platform/use-platform/machines/machine-config-templates.md) by name. The provider renders the template's `cloudInitTemplate` into a cloud-config document when `vcluster.com/user-data` isn't set. See [Machine Config Templates](https://vcluster.com/docs/platform/use-platform/machines/machine-config-templates.md) for the template syntax and available variables. The platform always merges resolved SSH keys into the cloud-config. For a tenant cluster worker, it also appends the join command. For a tenant cluster control plane machine, it appends the control plane installation commands. Independently provisioned Machines receive no tenant cluster registration command. ## Try it yourself[​](#try-it-yourself "Direct link to Try it yourself") The community and experimental [vCluster Bare Metal with KubeVirt](https://github.com/loft-sh/vcluster-bare-metal-with-kubevirt) guide runs the Metal3 provisioning flow locally with KubeVirt VMs as fake bare metal servers. It sets up a [vind](https://vcluster.com/docs/vcluster/deploy/control-plane/docker-container/basics.md) (vCluster in Docker) cluster, a Metal3 NodeProvider, and simulated BareMetalHosts with Redfish BMC endpoints. No physical hardware is required. --- # Metal3 Network Integration The Metal3 provider provisions bare metal servers, but on its own it doesn't isolate the network those servers run on. Network integration determines how strongly each tenant's traffic is separated on the metal fabric and how the platform manages addresses for provisioned servers. The right choice depends on how much you trust the tenants that share your hardware. A single trusted team can run on a flat provisioning network, while untrusted or external tenants need isolation enforced in the physical network. The `metal3.neutronEnabled` NodeProvider field enables Neutron integration. Within Neutron mode, the `metal3.vcluster.com/network-provider` property selects an optional fabric provider on a network environment (a [`NodeEnvironment`](https://vcluster.com/docs/platform/administer/node-providers/overview.md#node-environments) resource). Set the property to `netris` for Netris, or leave it empty for a flat network. Requires a network environment Neutron and Netris require a network environment. They support Machines provisioned independently or used as [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for a tenant cluster. Use private nodes and a separate Netris server cluster for each network environment when tenants are untrusted. ## Choose a network integration mode[​](#choose-a-network-integration-mode "Direct link to Choose a network integration mode") There are three modes, from a flat shared network to full isolation in hardware. | Mode | Isolation level | IPAM | When to use | | -------------------------------- | -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | No network integration (default) | None. The provisioning network is the tenant network. | Optional. Platform IPAM allocates addresses when a network CIDR is configured. | A single trusted tenant, or a lab where the provisioning network is the only network. | | Neutron (flat network) | None at the fabric. The network stays flat and shared; Ironic owns the port lifecycle. | Platform IPAM, per tenant network. | You want Ironic to drive ports through its neutron interface, but don't need hardware isolation. | | Netris | Hardware. Environment-level L2 isolation through Netris. | Platform IPAM; the subnet is derived from the Netris server cluster unless a CIDR is pinned. | Untrusted, external, or regulated tenants that must be isolated at the physical network layer. | The two flat modes keep tenant cluster control planes and APIs separate and manage their addresses independently. Every server still shares the same layer 2 network, so these modes don't provide network isolation. Only Netris moves that isolation boundary into the hardware fabric. ## No network integration (default)[​](#no-network-integration-default "Direct link to No network integration (default)") With `metal3.neutronEnabled` set to `false`, the provisioning network is also the tenant network. When `metal3.vcluster.com/network-cidr` is configured, the platform's built-in IPAM allocates an address on that shared network. It writes the address, gateway, and DNS servers directly onto the `BareMetalHost` as DHCP annotations. Without a network CIDR, the platform doesn't allocate an address or write managed DHCP annotations. The bundled DHCP server can't serve a PXE client unless another system supplies the required IP address annotation on its `BareMetalHost`. Alternatively, use external DHCP. You can provide custom network data to configure the installed operating system, but it doesn't replace provisioning DHCP. Addresses come from the existing network properties documented on the [Metal3 provider page](https://vcluster.com/docs/platform/administer/node-providers/metal3.md#network-configuration): * `metal3.vcluster.com/network-cidr` sets the gateway and subnet to allocate from. * `metal3.vcluster.com/network-ip-range` restricts allocation to explicit ranges. * `metal3.vcluster.com/network-data` supplies a complete custom network-data document. Because all servers share one network, use this mode only where the tenants sharing the hardware are trusted. ## Neutron (flat network)[​](#neutron-flat-network "Direct link to Neutron (flat network)") In this mode Ironic drives its `neutron` network interface instead of writing addresses straight onto the `BareMetalHost`. The platform allocates the address for each port and Ironic manages the port lifecycle across provisioning, cleaning, and inspection. Neutron doesn't isolate the network Neutron changes how ports are managed, not how the network is segmented. The underlying network is still flat and shared across tenants. For isolation enforced in hardware, use Netris. Enable Neutron and supply the externally provisioned network names that Ironic requests. The platform derives the chart's `neutron.enabled` value from `metal3.neutronEnabled`. Flat Neutron also requires `metal3.vcluster.com/network-cidr` in the effective properties of every NodeClaim. The platform can't allocate the tenant port without this CIDR. The following example defines the CIDR on a node type and binds its NodeClaims to a network environment in their management-plane namespace: ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: metal3-provider spec: metal3: clusterRef: cluster: bare-metal-cluster namespace: metal3-system # Enables the platform network controller and the chart's Neutron support. neutronEnabled: true deploy: metal3: enabled: true helmValues: | neutron: networks: provisioning: provisioning cleaning: cleaning inspection: inspection nodeTypes: - name: compute properties: # Flat Neutron can't derive a subnet, so configure one explicitly. metal3.vcluster.com/network-cidr: 192.168.100.1/24 # Resolve this NodeEnvironment from each NodeClaim's namespace. vcluster.com/network-environment: tenant-a-network --- apiVersion: management.loft.sh/v1 kind: NodeEnvironment metadata: name: tenant-a-network namespace: tenant-a spec: providerRef: metal3-provider ``` When `cleaning` or `inspection` are left empty, they fall back to the provisioning network. Servers provisioned in this mode select their network through a network environment, so a network environment is required. ## Netris (hardware network isolation)[​](#netris-hardware-network-isolation "Direct link to Netris (hardware network isolation)") Netris provides network isolation in hardware. On provisioning, the platform attaches each server in a network environment to that environment's Netris server cluster. Give each tenant network environment its own server cluster to isolate tenant traffic at the physical network layer. Requires [Neutron mode](#neutron-flat-network). Netris builds on Neutron and adds environment-level L2 isolation, so it also requires a network environment. ### Provide Netris API credentials[​](#provide-netris-api-credentials "Direct link to Provide Netris API credentials") Create a Secret on the management cluster that contains the Netris API `url`, `username`, and `password`. Reference its management-plane namespace on the NodeProvider. This namespace is independent of the namespace in `metal3.clusterRef`, which identifies where Metal3 resources run on the connected control plane cluster. Add the following Netris settings to the provider configuration from the preceding Neutron section. The fragment configures the credentials and binds the node type to the network environment shown later on this page: ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: metal3-provider spec: metal3: neutronEnabled: true netris: secretRef: name: netris-credentials namespace: vcluster-platform nodeTypes: - name: compute properties: # Resolve this NodeEnvironment from each NodeClaim's namespace. vcluster.com/network-environment: tenant-a-network ``` ### Select and configure the Netris provider[​](#select-and-configure-the-netris-provider "Direct link to Select and configure the Netris provider") The Netris provider is selected and configured through properties on the network environment. Set them under `spec.properties`. | Property | Required | Description | | -------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `metal3.vcluster.com/network-provider` | Yes | Set to `netris` to select the Netris network provider. | | `netris.vcluster.com/server-cluster` | Yes | Name of the Netris server cluster to attach each server to. Also the subnet source when no CIDR is pinned. | | `netris.vcluster.com/server-cluster-template` | Conditional | Name of a Netris server-cluster template. The platform creates the server cluster from it when the cluster doesn't already exist. | | `netris.vcluster.com/server-cluster-template-json` | Conditional | JSON array used as the template's `vnets` array when a template of that name doesn't already exist. | | `netris.vcluster.com/admin` | Conditional | Netris tenant that owns the created server cluster. Required when the platform creates the server cluster. | | `netris.vcluster.com/site` | Conditional | Netris site the created server cluster belongs to. Required when the platform creates the server cluster. | | `netris.vcluster.com/vpc` | No | Name of an existing VPC for the created server cluster. When empty, Netris creates a VPC implicitly as part of server-cluster creation. | When the named server cluster already exists, the platform adopts it and never marks it managed. In that case, `server-cluster-template`, `admin`, and `site` are unneeded. When the platform creates a server cluster or template, it records what it created on the network environment so it can remove exactly those resources on teardown: * `netris.vcluster.com/managed-server-cluster` * `netris.vcluster.com/managed-server-cluster-template` These annotations are managed by the platform. Don't set them by hand. Don't share managed Netris resources The platform deletes only the server cluster and template recorded as managed by a network environment. It doesn't delete a VPC specified by `netris.vcluster.com/vpc`. If other network environments reference a managed server cluster or template, deleting its creator can remove the shared resource or fail while the resource is still in use. Give each network environment its own server cluster and template. Share only pre-existing resources that the platform adopts instead of manages. The subnet for each server is derived from the Netris server cluster's network unless `metal3.vcluster.com/network-cidr` pins it, in which case the pinned CIDR always wins. ### Override the Netris server name[​](#override-the-netris-server-name "Direct link to Override the Netris server name") By default a server is registered in Netris under its `BareMetalHost` name. To register it under a different name, set the following annotation on the `BareMetalHost`: ``` apiVersion: metal3.io/v1alpha1 kind: BareMetalHost metadata: name: server-01 annotations: netris.vcluster.com/server-name: rack1-server-01 ``` ### Example[​](#example "Direct link to Example") This network environment selects the Netris provider and points at a server cluster that the platform creates from a template if it doesn't already exist: ``` apiVersion: management.loft.sh/v1 kind: NodeEnvironment metadata: name: tenant-a-network # NodeEnvironment must use the same management-plane namespace as its NodeClaims. namespace: tenant-a spec: providerRef: metal3-provider properties: metal3.vcluster.com/network-provider: netris netris.vcluster.com/server-cluster: tenant-a netris.vcluster.com/server-cluster-template: default-template netris.vcluster.com/admin: tenant-a-admin netris.vcluster.com/site: datacenter-1 ``` Servers select this network environment through the `vcluster.com/network-environment` property, described on the [Metal3 provider page](https://vcluster.com/docs/platform/administer/node-providers/metal3.md#network-environment). Any server provisioned into it is attached to the `tenant-a` Netris server cluster. The server cluster forms the hardware-backed network boundary for this environment. --- # Overview Node providers let you configure sources for private nodes for tenant clusters automatically. Platform supports the following node providers: * [Terraform](https://vcluster.com/docs/platform/administer/node-providers/terraform.md) (powered by [OpenTofu](https://opentofu.org) under the hood) * [KubeVirt](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md) for VM-backed private nodes. To inspect and operate KubeVirt VMs directly, see [Virtual Machines](https://vcluster.com/docs/platform/administer/virtual-machines/overview.md). * [Nvidia Base Command Manager](https://vcluster.com/docs/platform/administer/node-providers/bcm.md) * [vMetal](https://vcluster.com/docs/platform/administer/node-providers/metal3.md) (bare metal provisioning using Metal3/Ironic) Each node provider defines provider-specific configuration and node types that tenant clusters select from. When a tenant cluster requests a new node, it creates a node claim, telling the platform to provision a node with the given requirements. The platform then matches these requirements to the cheapest available node type and creates a node from that type. An example node provider with two distinct node types: ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: terraform-example spec: terraform: nodeTemplate: git: repository: https://github.com/my-org/my-repo.git nodeTypes: - name: medium resources: cpu: "2" memory: 4Gi properties: instance-type: medium - name: large resources: cpu: "4" memory: 8Gi properties: instance-type: large ``` This creates a node provider with a Terraform node template. For each node claim, the platform executes the Terraform script. Reference this node provider in your tenant cluster using: ``` # Use this in your vcluster.yaml privateNodes: enabled: true autoNodes: provider: terraform-example dynamic: # This node pool automatically chooses the node type # to right size the amount of pods. - name: my-node-pool ``` Single node provider, many tenant clusters Do not create a node provider per tenant cluster. Create a single node provider and share it across tenant clusters. ## Node types[​](#node-types "Direct link to Node types") Node types are machine types that vCluster selects based on properties, resources, and cost. Create a node type for each instance type you want vCluster to choose from, such as different VM sizes or cloud instance types. vCluster automatically selects the best fitting node type based on requirements. If there are no requirements or multiple node types match, vCluster selects the cheapest available node type. ### Properties[​](#properties "Direct link to Properties") Properties on a node type are similar to labels and let users specify traits to include or exclude certain node types when tenant clusters are created. Users can define custom properties in addition to the always-available built-in properties. #### Use properties[​](#use-properties "Direct link to Use properties") Use properties through the `requirements` field under `autoNodes` node pools to include or exclude node types: ``` # Inside your vcluster.yaml privateNodes: enabled: true autoNodes: provider: my-provider properties: # Exact match my-propert: my-value # One of my-property/value-1: true my-property/value-2: true my-property/value-3: true # Not in my-property/value-1: false my-property/value-2: false my-property/value-3: false # Exists my-property: true # Not exists my-property: false static: - name: my-static-pool ``` The following operators are available (and similar to [Kubernetes set-based requirements](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#set-based-requirement)): * `In` (default): Matches one or multiple values on the node type * `NotIn`: Matches if the given values aren't part of the properties * `Exists`: Matches if the property is defined on the node type * `NotExists`: Matches if the property is not defined on the node type #### Custom properties[​](#custom-properties "Direct link to Custom properties") Custom properties are defined using the `properties` field and map a key to one or multiple values. For example: ``` ... kind: NodeProvider ... spec: terraform: nodeTypes: - name: medium [...] properties: # Single value my-property: my-value # Multiple values my-property-multi: my-value-1,my-value-2 # All values my-property-all: "*" ``` Property values must adhere to the [syntax and character set of Kubernetes labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set), as they are set as labels on created nodes. #### In-built properties[​](#in-built-properties "Direct link to In-built properties") The platform automatically adds these properties: * `vcluster.com/node-type`: The name of the node type. Maps vCluster node pools to a specific node type. Since node type names are globally unique, they always map to a single node provider. * `node.kubernetes.io/instance-type`: Same as `vcluster.com/node-type`, but the official Kubernetes label. * `kubernetes.io/os`: Always `linux`. * `topology.kubernetes.io/zone`: Maps to the `spec.zone` field of the node type. Defaults to `global` if unspecified. * `karpenter.sh/capacity-type`: Always `on-demand`. Capacity type distinctions are not currently supported. ### Resources[​](#resources "Direct link to Resources") The `resources` field is required on each node type and specifies the computing resources nodes of that type provide. Certain node providers (such as BCM or KubeVirt) auto-discover resources, while others require you to specify at least `cpu` and `memory` for each node type. Resources are specified according to the [Kubernetes resource units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes). You can also specify custom resources such as `nvidia.com/gpu`, used for scheduling inside the tenant cluster. The platform does not schedule a node claim to a node type with custom resources unless those resources are explicitly requested. A tenant cluster that does not request a GPU is never assigned a GPU node type, even if it is the only available type. ### Cost[​](#cost "Direct link to Cost") Each node type has an associated cost, either set manually using `spec.cost` or calculated from `resources`. The cost unit is intentionally unspecified. It is not a currency but a relative value vCluster and the platform use to select the cheapest matching node type. Default costs when not specified manually: * Each `cpu` costs `10` * Each GB of `memory` costs `2` * Each other resource costs `10000`, discouraging scheduling on specialized resources. Karpenter always tries to find the cheapest node and automatically provisions or deprovisions nodes as costs change. ### Capacity[​](#capacity "Direct link to Capacity") Capacity is the number of available nodes for a given node type. The appropriate limit depends on the scenario. For bare metal, for example, you may want to cap the number of usable machines. Depending on the provider, capacity is set manually or discovered automatically: * *Terraform*: set using `maxCapacity`, otherwise unlimited * *KubeVirt*: set using `maxCapacity`, otherwise unlimited * *BCM*: is automatically discovered * *Metal3*: determined by available BareMetalHosts matching the node type's label selector If specified, the platform tracks how many nodes of this type have been created and marks the node type as unavailable once the limit is reached. vCluster stops using that node type until nodes are deprovisioned. ## Node claims[​](#node-claims "Direct link to Node claims") When a tenant cluster requests a new node, either dynamically using [Karpenter](https://karpenter.sh/) or through static configuration, vCluster creates a node claim in the platform with requirements. The platform then selects the node type that best fulfills those requirements. If multiple node types match, the platform chooses the cheapest available one. You can restrict the node types a project's NodeClaims may use with [`spec.allowedNodeTypes`](https://vcluster.com/docs/platform/administer/projects/allowed/node-types.md). Deleting a NodeClaim doesn't drain the node When a NodeClaim is deleted, the platform starts destroying the underlying machine without cordoning the tenant cluster Node or evicting its workloads. Any pods still running on that Node stop without eviction. Drain the Node from the tenant cluster before you delete the NodeClaim: Modify the following with your specific values to generate a copyable command: NODE\_NAMEmy-node ``` kubectl drain my-node --ignore-daemonsets --delete-emptydir-data ``` `kubectl drain` cordons the Node before it evicts pods, so no new pods get scheduled there during the drain. ## Node environments[​](#node-environments "Direct link to Node environments") Node environments are optional and scoped to a single tenant cluster and node provider. Some node providers require one-time setup before they can create nodes. For example, a tenant cluster in AWS using the Terraform provider needs a VPC, routing tables, and subnets before deploying any nodes. The platform creates node environments lazily, only when a tenant cluster first selects that provider. Once the environment is ready, node creation proceeds. --- # Terraform | | | Enterprise | | | | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Terraform Node Provider](https://vcluster.com/docs/platform/administer/node-providers/terraform.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). The terraform provider allows you to use Terraform scripts to automatically create [nodes](https://vcluster.com/docs/platform/administer/node-providers/overview.md) and [node environments](https://vcluster.com/docs/platform/administer/node-providers/overview.md). When a vCluster requests a new node, the platform will execute the specified Terraform script to create and join a node automatically into the vCluster. If the node or vCluster is later deleted, the platform will destroy the node using Terraform.
OpenTofu We use OpenTofu to execute the terraform scripts. We are currently working on a way to inject your own binary of regular terraform into the platform to use services like [Terraform Cloud](https://developer.hashicorp.com/terraform/cloud-docs) ## Node Templates[​](#node-templates "Direct link to Node Templates") Node templates are terraform scripts either defined inside the node provider object directly or referenced via git. Git We recommend to use git for storing your Terraform node and node environment templates. This way you have a complete audit trail of who has changed what and you can even create multiple node providers based off different branches, tags or commits. ### Use Git[​](#use-git "Direct link to Use Git") When specified via git, the platform will fetch the repository and cache it locally until it is not needed anymore. It will check for changes based on the specified interval (default is 10 minutes). The following example specifies how to use a repository at `https://github.com/my-org/my-user.git` to create a node: ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: terraform-example spec: terraform: nodeTemplate: git: repository: https://github.com/my-org/my-repo.git ``` Besides `repository` you can use the following options for git: * `branch`: branch to checkout * `commit`: commit sha256 to checkout * `subPath`: the path within the repository * `username.secretName`, `username.secretNamespace` and `username.key` to specify the username for a private git repository * `password.secretName`, `password.secretNamespace` and `password.key` to specify the password (or token) for a private git repository * `fetchInterval`: specifies the interval to fetch the repository. Refetching only checks for remote changes but does not do a complete repull ### Use inline[​](#use-inline "Direct link to Use inline") As an alternative to git you can also specify the template inline in the node provider object. The following example specifies a very simple inline template to create an AWS ec2 instance for a vCluster node: ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: terraform-example spec: terraform: nodeTemplate: inline: |- terraform { required_version = ">= 1.5" required_providers { aws = { source = "hashicorp/aws" version = ">= 5.0" } } } provider "aws" { region = var.vcluster.requirements["region"] } resource "aws_instance" "node" { ami = "my-ami" instance_type = "my-instance-type" subnet_id = "my-subnet-id" vpc_security_group_ids = ["my-security-group-id"] user_data = var.vcluster.userData } ``` ### Use quick start templates[​](#use-quick-start-templates "Direct link to Use quick start templates") Terraform quick-start templates are available to automatically provision the required cloud infrastructure and deploy Kubernetes components such as the [Cloud Controller Manager (CCM)](https://kubernetes.io/docs/concepts/architecture/cloud-controller/) and [Container Storage Interface (CSI)](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/).
These components enable full cloud provider integration, ensuring that control-plane operations, storage provisioning, and networking features work seamlessly. With these templates, you can set up a fully functional environment with minimal manual effort. Currently supported providers: * [AWS](https://github.com/loft-sh/vcluster-auto-nodes-aws/tree/v0.1.0) * [Azure](https://github.com/loft-sh/vcluster-auto-nodes-azure/tree/v0.1.0) * [GCP](https://github.com/loft-sh/vcluster-auto-nodes-gcp/tree/v0.1.0) See the full [quick start deployment guide](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/auto-nodes/quick-start-templates.md) for more details. ### Available terraform vCluster variables[​](#available-terraform-vcluster-variables "Direct link to Available terraform vCluster variables") Besides the user-defined credentials specified below, the platform passes a `var.vcluster` object to each node script. The following values are available: * `var.vcluster.name`: Name of the vCluster instance * `var.vcluster.namespace`: Project namespace of the vCluster instance * `var.vcluster.userData`: The [cloud-init script](https://cloud-init.io/) generated that should be passed to the instance to join the node into the vCluster * `var.vcluster.requirements`: A key-value map of the specified and resolved requirements of the node claim * `var.vcluster.instance`: Full tenant cluster instance object * `var.vcluster.nodeClaim`: Full node claim object * `var.vcluster.nodeType`: Full node type object * `var.vcluster.nodeEnvironment.outputs`: If a node environment was specified for this provider, the defined outputs of the node environment ## Node types[​](#node-types "Direct link to Node types") [Node types](https://vcluster.com/docs/platform/administer/node-providers/overview.md) are used by vCluster and the platform to decide what exact machine type should get provisioned. Each node type can define their own node template or reuse the central node type under `spec.nodeType`. Properties of node types will be passed via the `var.vcluster.requirements` field to the Terraform script. Properties Use properties of node types to define custom terraform variables which can be used for info like instance type or region. An example could look like this: ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: terraform-example spec: terraform: nodeTemplate: git: repository: https://github.com/my-org/my-repo.git subPath: generic nodeTypes: - name: medium resources: cpu: "2" memory: "4Gi" properties: instance-type: medium # can be accessed via 'var.vcluster.requirements["instance-type"]' inside terraform - name: large resources: cpu: "4" memory: "8Gi" properties: region: "*" # allow all values for region cpu-generation: "old,new" # allow old or new as cpu-generation nodeTemplate: # override the spec.terraform.nodeTemplate git: repository: https://github.com/my-org/my-repo.git subPath: large ``` ### Max capacity[​](#max-capacity "Direct link to Max capacity") The node type field `maxCapacity` allows you to specify an upper limit of how many nodes should be created for the node type. ## Node environment template[​](#node-environment-template "Direct link to Node environment template") [Node environments](https://vcluster.com/docs/platform/administer/node-providers/overview.md) can be configured similar to node templates either inline or using git. Each node environment will be created once per vCluster before the first node claim is provisioned. Only if a vCluster is using a node the provider a new node environment is provisioned. The following example specifies a very simple inline template to create an AWS VPC instance for a vCluster: ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: terraform-example spec: terraform: nodeEnvironmentTemplate: inline: |- terraform { required_version = ">= 1.5" required_providers { aws = { source = "hashicorp/aws" version = ">= 5.0" } } } provider "aws" { region = var.vcluster.requirements["region"] } resource "aws_vpc" "vcluster" { cidr_block = "10.0.0.0/16" enable_dns_support = true enable_dns_hostnames = true tags = { Name = "${var.vcluster.name}-vpc" } } output "vpc_id" { value = aws_vpc.vcluster.id } nodeTemplate: inline: |- ... ``` The defined output `vpc_id` within the node environment can be accessed in the node template using `var.vcluster.nodeEnvironment.outputs["vpc_id"]`. ### Available terraform vCluster variables[​](#available-terraform-vcluster-variables-1 "Direct link to Available terraform vCluster variables") Besides the user-defined credentials specified below, the platform passes a `var.vcluster` object to each node environment script. The following values are available: * `var.vcluster.name`: Name of the vCluster * `var.vcluster.namespace`: Project namespace of the vCluster * `var.vcluster.requirements`: A key-value map of the specified and resolved requirements of the first node claim used to create this node environment * `var.vcluster.instance`: Full tenant cluster instance object * `var.vcluster.nodeEnvironment`: Full node environment object ## Credentials[​](#credentials "Direct link to Credentials") Credentials to the Terraform scripts can be defined using Kubernetes secrets. The key and values of these secrets can then be passed as either environment variables or regular variables to the Terraform commands. Within the vCluster you can select the credentials using the `terraform.vcluster.com/credentials` property. This allows you to specify different credentials for different teams or users and select the dynmically. Dynamic Credentials If possible we recommend to use [dynamic credentials](https://developer.hashicorp.com/terraform/tutorials/cloud/dynamic-credentials) for providers instead of long-living hardcoded credentials using Kubernetes secrets. Almost all Terraform providers support this and marks the additional configuration using Kubernetes secrets obsolete. ### Environment variables secret[​](#environment-variables-secret "Direct link to Environment variables secret") You can create a secret in the following form: ``` apiVersion: v1 kind: Secret metadata: name: aws-credentials namespace: any-namespace labels: terraform.vcluster.com/provider: my-node-provider data: # Base64 encoded value AWS_ACCESS_KEY_ID: ... # Base64 AWS_SECRET_ACCESS_KEY: ... ``` This will lead to the platform passing the environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` to the `tofu init` and `tofu apply` commands. ### Terraform variables secret[​](#terraform-variables-secret "Direct link to Terraform variables secret") You can create a secret in the following form: ``` apiVersion: v1 kind: Secret metadata: name: aws-credentials namespace: any-namespace annotations: terraform.vcluster.com/type: vars labels: terraform.vcluster.com/provider: my-node-provider data: # Base64 encoded value aws_access_key_id: ... # Base64 aws_secret_access_key: ... ``` This will lead to the platform passing the Terraform variables `aws_access_key_id` and `aws_secret_access_key` to the `tofu init` and `tofu apply` commands using `-var aws_access_key_id=VALUE`. ## Terraform backend[​](#terraform-backend "Direct link to Terraform backend") By default, the platform will use the [Kubernetes backend](https://developer.hashicorp.com/terraform/language/backend/kubernetes) to store the state of the nodes and node environments in Kubernetes secrets in the `vcluster-platform` namespace. The platform will create a separate workspace per node and node environment in the following form: * *Node*: `NODE_CLAIM_NAME.PROJECT_NAME.node` * *Node Environment*: `NODE_ENVIRONMENT_NAME.PROJECT_NAME.env` ### Use a custom backend[​](#use-a-custom-backend "Direct link to Use a custom backend") The platform will parse the Terraform script files before initializing Terraform and search for `backend` blocks. If none is found, it will use the above described default backend. If it finds a user-defined `backend` it will not use the default backend and instead try to use the specified backend. --- # Manage Allowed Clusters Allowed clusters control the clusters which project members are granted access. ### Add allowed clusters[​](#add-allowed-clusters "Direct link to Add allowed clusters") 1. Select the project you'd like to configure using the project selector dropdown. Click on Settings. 2. Click on Control Plane Clusters. 3. Click on the input and select the cluster names to allow. If you wish to change this manually, you may edit the YAML directly. 4. Once all project options have been specified, click the Save Changes. All Clusters Instead of selecting every cluster, a special selection named `All Clusters` can be used to give access to all clusters to the project. Use this option with caution. --- # Manage Node Profiles A project can restrict which [node profiles](https://vcluster.com/docs/platform/administer/node-profiles/overview.md) its private nodes may use, and separately set a default profile for nodes that don't select one explicitly. Use these settings to keep tenants on a curated set of profiles and to apply a baseline configuration to nodes that join without an explicit profile. ## Define allowed and default profiles[​](#define-allowed-and-default-profiles "Direct link to Define allowed and default profiles") Suppose the `ml-platform` project should only allow the `gpu-training` and `general-compute` profiles, and should default to `general-compute` when a node doesn't select a profile explicitly. Edit the project YAML and set `spec.allowedNodeProfiles` and, optionally, `spec.defaultNodeProfile`: ``` apiVersion: management.loft.sh/v1 kind: Project metadata: name: ml-platform spec: allowedNodeProfiles: - name: gpu-training - name: general-compute defaultNodeProfile: general-compute ``` With this spec, nodes in this project can use either `gpu-training` or `general-compute`. A node that joins without an explicit profile, and whose tenant cluster has no `privateNodes.defaultProfile`, falls back to the project's `general-compute` default. If a project shouldn't use node profiles at all, disallow them: ``` spec: allowedNodeProfiles: [] ``` Also clear the default An empty `allowedNodeProfiles` list is rejected while `defaultNodeProfile` is still set, because the default itself would no longer be allowed. Clear `defaultNodeProfile` in the same update. ## Behavior[​](#behavior "Direct link to Behavior") The project fields are `spec.allowedNodeProfiles` and `spec.defaultNodeProfile`. Changing either field requires permission to update the corresponding project subresource. ### Allowed node profiles[​](#allowed-node-profiles "Direct link to Allowed node profiles") The value of `allowedNodeProfiles` determines which profiles a project's nodes may use: | Value | Effect | | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | Unset (field omitted) | All node profiles are allowed. This is the default. | | Empty list (`allowedNodeProfiles: []`) | No node profiles are allowed. Any explicit `profileRef` or `profile` is rejected. `defaultNodeProfile` must also be unset. | | One or more entries | Only allows node profiles matching any specified entry. | Each entry in `allowedNodeProfiles` has a `name`, which can be either an exact `NodeProfile` name or an owner-prefix wildcard ending in `.*`. Matching rules: * Names are case sensitive. * A wildcard entry must end in `.*`. Only that literal suffix triggers prefix matching, so `platform.*` matches any profile whose name starts with `platform.`, such as `platform.gpu-training`. A pattern like `gpu-*` is treated as an exact name. * The platform rejects a manual join, `NodeClaim`, or auto-node pool that explicitly references a disallowed profile. ### Default node profile[​](#default-node-profile "Direct link to Default node profile") `defaultNodeProfile` sets the project fallback profile. A node uses this value only when it doesn't select a profile explicitly and the tenant cluster's `privateNodes.defaultProfile` is also unset. The default must always be allowed by `allowedNodeProfiles`. That means `defaultNodeProfile` can't be set while `allowedNodeProfiles` is an empty list, because nothing matches an empty allowlist. ## Precedence[​](#precedence "Direct link to Precedence") A profile is resolved in the order described in [Assignment and precedence](https://vcluster.com/docs/platform/administer/node-profiles/overview.md#assignment-and-precedence). An explicit selection wins first, then the tenant cluster's default, then the project's default. Whichever source wins, the result is validated against `allowedNodeProfiles` before the profile is applied. --- # Manage Allowed Node Types Allowed node types control which [node types](https://vcluster.com/docs/platform/administer/node-providers/overview.md) can be referenced by NodeClaims created in a project. Use this to restrict tenants to a subset of the node types available on the platform. For example, you can use this to limit a project to a specific provider or instance class. ## Behavior[​](#behavior "Direct link to Behavior") The project field is `spec.allowedNodeTypes`. Each entry has a `name` that is either an exact node type name (for example, `aws.large`) or a provider wildcard (such as `aws.*`, which matches every node type belonging to provider `aws`). | Value | Effect | | ----------------------------------- | ------------------------------------------------------------------ | | Unset (field omitted) | All node types are allowed. This is the default. | | Empty list (`allowedNodeTypes: []`) | No node types are allowed. NodeClaims in the project are rejected. | | One or more entries | Only allows node types matching any specified entry. | Matching rules: * Names are case sensitive. * The wildcard is only recognized as the suffix `.*`. `aws.*` matches `aws.large` but does not match `awsx.large`. * The platform rejects a NodeClaim that explicitly references a disallowed node type at creation time. * For unbound NodeClaims (no `typeRef`), the reconciler filters candidate node types against `allowedNodeTypes` at scheduling time and never assigns a disallowed type to the claim. Changing `allowedNodeTypes` requires permission to update the project's `allowedNodeTypes` subresource. ## Configure allowed node types[​](#configure-allowed-node-types "Direct link to Configure allowed node types") Edit the project and set `spec.allowedNodeTypes`: ``` apiVersion: management.loft.sh/v1 kind: Project metadata: name: team-a spec: allowedNodeTypes: - name: aws.* - name: metal3.gpu-h100 ``` This project can use any node type from the `aws` provider, plus the specific `metal3.gpu-h100` type. To disallow all node types in a project: ``` spec: allowedNodeTypes: [] ``` --- # Manage Allowed Templates Allowed templates are the templates that project members are allowed to use when creating resources in a project. ## Security best practices[​](#security-best-practices "Direct link to Security best practices") By default, vCluster Platform requires templates for non-admin users to create tenant clusters. This default exists because creating tenant clusters without templates can pose security risks. Why templates matter for security When users create tenant clusters without a template, they have full control over the vCluster configuration. Misconfigured settings can allow users to: * Access control plane cluster resources they should not have access to through sync settings * Sync sensitive resources from the control plane cluster into the tenant cluster * Create resources in the host namespace that affect other workloads * Deploy arbitrary resources to the host namespace using [`experimental.deploy.host`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/deploy.md), which inherits the platform's permissions and can create role bindings or cluster role bindings * Access the host namespace through [Namespace Objects and Namespace Apps](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/isolation.md#configure-namespace-objects), which deploy resources directly to the control plane cluster ### Recommendations[​](#recommendations "Direct link to Recommendations") 1. **Keep templates required**: The default `requireTemplate` setting ensures non-admin users can only create tenant clusters from pre-approved templates. Do not disable this setting unless you trust all project members with full vCluster configuration access. 2. **Use hardened templates**: Configure templates with restricted sync settings. See [security hardening](https://vcluster.com/docs/platform/administer/templates/create-templates.md#security-hardening) for guidance on creating secure templates. 3. **Review allowed templates carefully**: Only add templates to a project that have been reviewed for security implications. Avoid using "All Templates" unless all available templates are appropriately secured. 4. **Limit admin access**: Only grant project admin or platform admin roles to users who need the ability to create tenant clusters without templates. ## Add templates[​](#add-templates "Direct link to Add templates") Only project admins or global admins can add more templates to a project. 1. Select the project you'd like to configure using the project selector dropdown. Click on Settings. 2. Click on Template Options. 3. Select which resource you want to add more templates to. 4. In the "Allowed" section, click the input and select the name of the template to add. If you wish to change this manually, you may edit the YAML directly. 5. Once all project options have been specified, click the Save Changes. All Templates Instead of adding every template, a special selection named `All Templates` can be used to give all templates of the platform to the project. Use this option with caution. ## Set a default template[​](#set-a-default-template "Direct link to Set a default template") A default template can be selected for a project, so that the template is automatically selected when creating the resource. The default template is not enforced during resource creation. 1. Select the project you'd like to configure using the project selector dropdown. Click on Settings. 2. Click on Template Options. 3. Select which resource you want to set a default template. 4. Select the name of the template to set the default template. If you wish to change this manually, you may edit the YAML directly. 5. Once all project options have been specified, click the Save Changes. ## Advanced: Setting a Namespace Pattern[​](#advanced-setting-a-namespace-pattern "Direct link to Advanced: Setting a Namespace Pattern") When creating a resource, it can be repetitive to specify a `spec.clusterRef. namespace` in each resource. Namespace patterns provide a template to use when creating namespaces for resources. These can be used to save time or enforce a naming convention. For example, the namespace pattern `{{.Values.loft.project}}-{{.Values.loft.name}}` would result in a namespace with name `my-project-space-1` being created for the space instance named `space-1` for a project named `my-project`. 1. Select the project you'd like to configure using the project selector dropdown. Click on Settings. 2. Click on Template Options. 3. Expand the Advanced Options section. 4. Click on the Namespace Pattern input and enter a namespace pattern. If you wish to change this manually, you may edit the YAML directly. 5. Once all project options have been specified, click the Save Changes. ### Available variables for tenant clusters[​](#available-variables-for-tenant-clusters "Direct link to Available variables for tenant clusters") The variables available for templating `namespacePattern.virtualCluster` are: * `.Values.loft.user.name`: The name of the owner, if the owner is a user * `.Values.loft.team.name`: The name of the owner, if the owner is a team * `.Values.loft.name`: The name of the space or tenant cluster instance * `.Values.loft.project`: The name of the project * `.Values.loft.space`: The name of the space containing the tenant cluster * `.Values.loft.cluster`: The name of the connected cluster the space is scheduled to * `.Values.loft.virtualClusterName`: The name of the tenant cluster * `.Values.loft.virtualClusterNamespace`: The name of the space containing the tenant cluster ### Available variables for spaces[​](#available-variables-for-spaces "Direct link to Available variables for spaces") The variables available for templating `namespacePattern.space` are: * `.Values.loft.user.name`: The name of the owner, if the owner is a user * `.Values.loft.team.name`: The name of the owner, if the owner is a team * `.Values.loft.name`: The name of the space or tenant cluster instance * `.Values.loft.project`: The name of the project * `.Values.loft.cluster`: The name of the connected cluster the space is scheduled to --- # Create a Project To get started creating a Project, in the vCluster Platform UI: Security recommendation Keep templates required for projects where non-admin users create tenant clusters. This prevents users from configuring sync settings that could grant elevated access to control plane cluster resources. See [security best practices](https://vcluster.com/docs/platform/administer/projects/allowed/templates.md#security-best-practices) for more information. 1. All available projects are in the top left dropdown. Click the toggle to find the Create New Project option. 2. Click on Members and add your users or teams and their project roles. See the [Manage Project Members](https://vcluster.com/docs/platform/administer/users-permissions/permissions/project.md) section for more information. 3. Click on Template Options to set the templates allowed for the project. By default, templates are required and can be changed in this step. See [Manage Allowed Templates](https://vcluster.com/docs/platform/administer/projects/allowed/templates.md) for more information. 4. Click on Control Plane Clusters to set the clusters that members should have access to. See the [Manage Allowed Clusters](https://vcluster.com/docs/platform/administer/projects/allowed/clusters.md) section for further information. 5. Click on Quotas to set the quotas for the project or specific user/team. See the [Quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md) section for further information. 6. Optionally, configure integrations for the project. See the [Argo CD integration](https://vcluster.com/docs/platform/integrations/argocd.md) and [HashiCorp Vault integration](https://vcluster.com/docs/platform/integrations/hashicorp-vault.md) documentation for further information. 7. Once all project options have been specified, click the Save Changes. --- # Manage Quotas | | | Enterprise | | | | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Project Quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Quotas allow you to set resource usage and count limits for the project. This works similarly to Kubernetes `ResourcesQuotas`, but applies across multiple clusters. Quotas can be defined across the whole project or on a per-user/per-team basis. Project quotas with multiple tenant clusters When you create multiple tenant cluster instances, the individually used resources count toward the shared project quota. ## Active resource quotas[​](#active-resource-quotas "Direct link to Active resource quotas") You can restrict the number of active instances for custom instance types. These quotas are suffixed with `.active`. They are enforced during instance creation, and also when a request is received to wake up a sleeping instance. ## Template quotas[​](#template-quotas "Direct link to Template quotas") You can restrict the number of instances using a given template at the same time. By selecting the instance (for example, `virtualclusterinstances`), a template dropdown appears to select what template to restrict. These are enforced during instance creation. ## Generic resource quotas[​](#generic-resource-quotas "Direct link to Generic resource quotas") Quotas can also specify limits on generic resources such as `count/pods` and `limits.cpu`, similar to a standard Kubernetes ResourceQuota, but the limits apply to the scope of the project. On cluster resource limits Custom resources such as `virtualclusterinstances` can be instantly enforced as soon as consumption exceeds the limit, but on-cluster resources such as `limits.cpu` cannot be immediately enforced to avoid blocking requests for a long time. Users may be able to exceed the quota by small amounts by simultaneously creating resources on multiple clusters. ## Machine quotas[​](#machine-quotas "Direct link to Machine quotas") You can limit the number of Machines (`NodeClaim` resources) a project can provision. Use the `nodeclaims` resource for a project-wide limit, and the per-user/team bucket to cap how many Machines a single user or team can own. To restrict Machines from a specific node provider, suffix the resource with the provider reference: `nodeclaims.provider=`. This counts only the Machines provisioned through that provider, so you can set separate limits per provider. A Machine without a matching limit is unrestricted. Machine quotas are enforced when a Machine is created. As with the instance quotas above, a Machine owned by a user or team counts against both the project quota and that owner's per-user/team quota. ## Add quotas for the project[​](#add-quotas-for-the-project "Direct link to Add quotas for the project") 1. Select the project you'd like to configure using the project selector dropdown. Click Settings. 2. Click Quotas. 3. Select the Per Project tab. 4. Click Add Quota. An entry in the quota table will appear. 5. For the Resource column, select the desired resource to add a quota. If you wish to change this manually, you may edit the YAML directly. 6. For the Quota column, enter a number or use the step selector to increase or decrease to the desired count. 7. Once all project options have been specified, click the Save Changes. ## Add quotas for a user or team[​](#add-quotas-for-a-user-or-team "Direct link to Add quotas for a user or team") When using a quota for a user or team, it applies to **all** users or teams in the project and cannot be applied to specific users and teams. For example, this project defines a total tenant cluster instance quota of 20, but prevents any user or team from using more than 5 tenant cluster. By adjusting these quotas, you can ensure that no user or team uses more than their fair share of cluster resources. 1. Select the project you'd like to configure using the project selector dropdown. Click Settings. 2. Click Quotas. 3. Select the Per User / Team tab. 4. Click Add Quota. An entry in the quota table will appear. 5. For the Resource column, select the desired resource to add a quota. If you wish to change this manually, you may edit the YAML directly. 6. For the Quota column, enter a number or use the step selector to increase or decrease to the desired count. 7. Once all project options have been specified, click the Save Changes. Team and Team Member Quotas If a team is the owner of a space or tenant cluster, the usage will be counted by the team and not count against each member of the team, nor will the usage of a team member count towards the team's quota. ## Quota exceeded behavior[​](#quota-exceeded-behavior "Direct link to Quota exceeded behavior") Once quotas are defined, users, and teams can create resources up to the defined limits. When usage meets the limit, attempts to create new resources are rejected with an error message indicating the resource type and limit that would be exceeded. Examples of possible error messages: ``` Error: metadata.name: Forbidden: cannot create spaceinstance, because project spaceinstances limit of 5 would be exceeded Error: metadata.name: Forbidden: cannot create spaceinstance, because per-user spaceinstances limit of 5 would be exceeded Error: metadata.name: Forbidden: cannot create virtualclusterinstance, because project virtualclusterinstances limit of 5 would be exceeded Error: metadata.name: Forbidden: cannot create virtualclusterinstance, because per-user virtualclusterinstances limit of 5 would be exceeded Error: metadata.name: Forbidden: cannot create nodeclaim, because project nodeclaims limit of 5 would be exceeded Error: metadata.name: Forbidden: cannot create nodeclaim, because per-user nodeclaims limit of 5 would be exceeded ``` Changing Quotas In order to avoid these errors in practice, quotas may be increased. Decreasing quotas, on the other hand, will not result in resources being automatically deleted. If you decrease quotas, resources will most likely exceed limits unless specifically cleaned up. For help identifying which quota denied a request, see [Troubleshoot quota and admission denials](https://vcluster.com/docs/platform/troubleshoot/quota-admission.md). ## Enforce limits inside a tenant cluster[​](#enforce-limits-inside-a-tenant-cluster "Direct link to Enforce limits inside a tenant cluster") Project quotas track aggregate usage across the project. They don't enforce per-namespace storage caps or per-PVC size limits inside individual tenant clusters. To close that gap, deploy ResourceQuota objects or Kyverno policies through a tenant cluster template. [Enforce storage quotas per tenant cluster →](https://vcluster.com/docs/platform/how-to/enforce-storage-quotas.md) --- # Create global secrets Global secrets let you define and share secrets across all registered clusters. vCluster Platform synchronizes the secret data into any native Kubernetes secret that references it. Synchronization applies only to secrets in spaces, not to secrets inside tenant clusters. To make global secret data available inside a tenant cluster, [create a project secret](https://vcluster.com/docs/platform/administer/secrets/project/create.md) that references the global secret. The **Global Secrets** menu item is only visible when you have `create` permission on the SharedSecret resource and the `Secrets` feature flag is enabled. If the menu item is missing, ask your platform administrator to verify both. To create a global secret follow these steps: 1. Go to Access & Secrets > Global Secrets. 2. Click Add Global Secret. 3. Enter a name in my-secret. 4. Enter a description in Enter a description for this object.... 5. Click Data. 6. Add your secret key/value pairs in Secret Data. 7. Click Management Access. 8. Click Add Rule to add an access rule. 9. In Users & Teams, select the user or team to grant access. 10. In Permissions, select the verb to enable. 11. Click Save Changes. To use this global secret, create a secret in the namespace where you need the secret data, taking care to add the labels: ``` apiVersion: v1 kind: Secret metadata: name: my-secret namespace: my-space labels: loft.sh/sharedsecret-namespace: vcluster-platform loft.sh/sharedsecret-name: my-secret type: Opaque ``` --- # Delete global secrets To delete a global secret follow these steps: 1. Go to Access & Secrets > Global Secrets. 2. Click Delete in the drop-down menu next to your secret. 3. Click Delete in the confirmation dialog. --- # Create project secrets Project secrets let you define and share secrets across the clusters where your project's namespaces and tenant clusters run. There are two ways to populate a project secret: 1. Enter secret data directly in the project secret. 2. Sync from a global secret, which lets you propagate the same secret data across many projects from one place. In both cases you consume the secret by creating a regular Kubernetes secret with labels that reference the project secret. vCluster Platform then synchronizes the data into that secret automatically. Data or Shared Secrets, but not both It is not possible to define data custom in a project secret and synchronize a shared secret. If you need to define additional secret data that is specific to your project, it is best to create a separate secret, and combine them in your pod specifications. ## Enter secret data directly[​](#enter-secret-data-directly "Direct link to Enter secret data directly") 1. Go to Projects, select your project, then click Secrets. 2. Click Create Secret. 3. Enter a display name such as "My Secret" and an optional description. The UI fills in `.metadata.name` automatically; edit the YAML directly to override it. 4. Add your secret key/value pairs in Secret Data. 5. Click Create Project Secret. See [Secret Sync](https://vcluster.com/docs/platform/use-platform/secrets/secret-sync.md) to learn how to use project secret data from a Pod. ## Sync from a global secret[​](#sync-from-a-global-secret "Direct link to Sync from a global secret") You must have read permission on the global secret, otherwise the request will be denied. 1. Go to Projects, select your project, then click Secrets. 2. Click Create Secret. 3. Enter a display name such as "My Secret" and an optional description. The UI fills in `.metadata.name` automatically; edit the YAML directly to override it. 4. In Data, enable Synchronize data from a global secret, then select the global secret in Select a Global Secret. 5. In Management Access, add which users and teams can access and modify the project secret. 6. Click Create Project Secret. See [Secret Sync](https://vcluster.com/docs/platform/use-platform/secrets/secret-sync.md) to learn how to use project secret data from a Pod. --- # Delete project secrets To delete a project secret follow these steps: 1. Go to Projects, select your project, then click Secrets. 2. Click Delete in the drop-down menu next to your secret. 3. Click Delete in the confirmation dialog. --- # Retrieve project secrets Use the vCluster CLI when working within the platform's project abstraction, as it lets you refer to secrets by name and project without knowing the underlying namespace. Use `kubectl` when you need the raw Kubernetes secret object, work in a CI/CD pipeline that already has a kube context, or are in an environment where the vCluster CLI is not available. ## Retrieve using vCluster CLI[​](#retrieve-using-vcluster-cli "Direct link to Retrieve using vCluster CLI") To retrieve a specific project secret, run: ``` vcluster platform get secret SECRET_NAME [flags] ``` To list all secrets in a project, run: ``` vcluster platform list secrets --project default ``` ## Retrieve using `kubectl`[​](#retrieve-using-kubectl "Direct link to retrieve-using-kubectl") To get a list of project secrets using `kubectl`, run the following command: ``` kubectl get secrets --all-namespaces --selector loft.sh/project-secret ``` If you want to only list project secrets for a project, add the project's namespace to the command: ``` kubectl get secrets --namespace loft-project-default --selector loft.sh/project-secret ``` --- # Manage Certified Stacks Certified Stacks are StackTemplates that vCluster Platform bundles and identifies as tested integrations. Platform 4.12 includes NVIDIA Run:ai templates for dedicated and central control-plane deployment models. Certified templates are read-only, and Platform updates can replace bundled resources during an upgrade. To change one, see [Customize a certified stack](#customize-a-certified-stack) for how to build your own StackTemplate from a certified one. ## Review bundled NVIDIA Run:ai resources[​](#review-bundled-nvidia-run-resources "Direct link to review-bundled-nvidia-run-resources") Platform 4.12 includes these certified resources: | Resource | Purpose | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `run-ai-dedicated-control-plane` StackTemplate | Installs the NVIDIA Run:ai control plane, GPU components, and cluster components in one tenant cluster. | | `run-ai-central-control-plane-host` StackTemplate | Installs the shared NVIDIA Run:ai control plane and GPU components once on a control plane cluster. | | `run-ai-central-control-plane-registration` StackTemplate | Registers one tenant with the shared NVIDIA Run:ai control plane. | | `run-ai-central-control-plane` StackTemplate | Installs the NVIDIA Run:ai tenant components in one tenant cluster. | | `runai-tenant` VirtualClusterTemplate | Creates a tenant cluster with the dedicated Stack. | | `runai-tenant-central-control-plane` VirtualClusterTemplate | Creates a tenant cluster, registration Stack, and central-model tenant Stack. | The bundle also includes the Apps referenced by these StackTemplates. All four Run:ai StackTemplates use App tasks, so they require the Apps feature, included on every plan, but not the Argo CD Integration feature. ## Choose a deployment model[​](#choose-a-deployment-model "Direct link to Choose a deployment model") | Model | Infrastructure | Choose this model when | | ----------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------ | | Dedicated control plane | Each tenant cluster runs its own NVIDIA Run:ai control plane and GPU Operator. | Each tenant needs independent NVIDIA Run:ai and GPU Operator lifecycles. | | Central control plane | Tenants share one NVIDIA Run:ai control plane and one GPU Operator on the control plane cluster. | Trusted tenants can share labeled nodes and a control-plane lifecycle. | Both bundled tenant templates use shared control plane cluster nodes by default. The dedicated model separates the lifecycle of the NVIDIA Run:ai control plane, but doesn't create private worker nodes. The central model partitions visible GPU nodes with labels. Node labels are scheduling policy, not a security boundary. Shared nodes are for trusted tenants Shared nodes are a supported, common model for trusted tenants. They give each tenant control-plane, API, and namespace isolation, but tenant workloads share the same kernel and physical nodes. They aren't a security boundary for untrusted tenants with Kubernetes access or arbitrary workload execution. Good fits are development, testing, CI/CD, and internal engineering teams. Use [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for external, resale, regulated, or otherwise untrusted tenant offerings, optionally with [vNode](https://www.vnode.com/docs/) for runtime isolation. NetworkPolicy is an added isolation layer worth enabling even for trusted tenants. vCluster can create the policies for you through `policies.networkPolicy`, and your control plane cluster's CNI enforces them. Confirm your CNI supports enforcement, since some accept NetworkPolicy resources without acting on them. See the [security baseline](https://vcluster.com/docs/vcluster/security/.md). If you accept shared nodes for a trusted-tenant use case, follow the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) as the required next step. ## Before you begin[​](#before-you-begin "Direct link to Before you begin") Both deployment models require: * vCluster Platform 4.12 and vCluster 0.37 * GPU-capable nodes with an NVIDIA runtime configuration appropriate for the provider * an ingress LoadBalancer and working DNS resolution * a StorageClass for NVIDIA Run:ai stateful components * NVIDIA Run:ai JFrog registry credentials * an administrator email address and password for the NVIDIA Run:ai control plane On AWS, the ingress-provider setting must match the LoadBalancer type, or the deploy stalls with a confusing error. See [Troubleshoot Stacks](https://vcluster.com/docs/platform/troubleshoot/stacks.md) for other ingress and certificate failure modes. The certified templates default to NVIDIA Run:ai 2.26. Chart versions are pinned in the bundled Apps. Changing only the `runaiVersion` parameter doesn't upgrade the charts. warning Choose one owner for the node-level NVIDIA components. For example, default Google Kubernetes Engine GPU pools already manage the driver and device plugin. Select the matching provider option instead of installing a second GPU stack. ## Deploy a dedicated control plane[​](#deploy-a-dedicated-control-plane "Direct link to Deploy a dedicated control plane") The `runai-tenant` VirtualClusterTemplate creates a tenant cluster and declares a `deploy.stacks` entry named `runai`. This entry references the `run-ai-dedicated-control-plane` StackTemplate. Platform generates the StackInstance name from the tenant cluster name and Stack configuration name. Before creating the tenant cluster, collect these values: * the StorageClass * the ingress provider and optional domain * NVIDIA Run:ai registry server, username, password, and email * the control-plane administrator email and password * an optional image pull Secret for the vCluster Platform image The bundled tenant template uses the standard GPU provider settings and self-signed TLS. Copy the VirtualClusterTemplate to expose other GPU or TLS parameters. For production, use a certificate for a domain you control. The default mode creates and retains an internal certificate authority. It derives a `nip.io` hostname when the LoadBalancer has an IPv4 address. After creation, monitor the generated StackInstance for the tenant cluster. Then follow [Verify a deployment](#verify-a-deployment) below. A healthy Helm task alone doesn't prove that the NVIDIA Run:ai scheduler is ready. ## Deploy a central control plane[​](#deploy-a-central-control-plane "Direct link to Deploy a central control plane") The central model has three lifecycles, covering the shared host foundation, one registration per tenant, and the components inside each tenant cluster. ### Install the shared host foundation[​](#install-the-shared-host-foundation "Direct link to Install the shared host foundation") Install `run-ai-central-control-plane-host` once on the control plane cluster. Use `runai` as the StackInstance name. Configure the shared StorageClass, ingress address, NVIDIA Run:ai credentials, TLS, and GPU provider. Wait for every host task to become healthy before creating tenants. Record the published control-plane FQDN and confirm the NVIDIA Run:ai UI is reachable. ### Label tenant GPU nodes[​](#label-tenant-gpu-nodes "Direct link to Label tenant GPU nodes") Assign each tenant a distinct, persistent node-label value. The bundled tenant template uses this key by default: ``` runai.vcluster.com/tenant ``` ``` # Assign this GPU node to one trusted tenant. kubectl label node gpu-node-a runai.vcluster.com/tenant=team-a # Confirm the tenant's visible node set. kubectl get nodes -l runai.vcluster.com/tenant=team-a ``` Configure the label through the node pool when possible. Manually applied labels don't appear on new nodes created by an autoscaler. Each node must match at most one tenant value. A tenant with no matching nodes can't schedule workloads. Overlapping assignments can cause independent tenant schedulers to allocate the same GPU capacity. ### Create a tenant[​](#create-a-tenant "Direct link to Create a tenant") Create the tenant cluster from `runai-tenant-central-control-plane`. Set `nodeSelectorValue` to the tenant's unique node-label value. The template performs these operations: 1. It creates a `run-ai-central-control-plane-registration` StackInstance in the project namespace. 2. The registration writes tenant facts to a Secret on the control plane cluster. 3. vCluster syncs the facts, registry credentials, and optional certificate authority into the tenant cluster. 4. The tenant's `run-ai-central-control-plane` Stack waits for those Secrets before installing its agent. Leave `tenantSlug` empty for the normal path. Set it only when an administrator created a distinct registration in advance. Duplicate slugs can adopt and deregister the same NVIDIA Run:ai cluster. The bundled tenant template expects the host to use a private certificate authority. If the shared endpoint uses a publicly trusted certificate, copy the tenant template. Set `requireControlPlaneCA: false` on its `deploy.stacks` entry. The host Stack and tenant Stack must use matching values. ## Verify a deployment[​](#verify-a-deployment "Direct link to Verify a deployment") Connect to the vCluster Platform management API and inspect aggregate and task status: ``` vcluster platform connect management kubectl get stackinstances -A kubectl get stackinstance -n \ -o jsonpath='{.status.phase}{"\n"}{range .status.tasks[*]}{.name}{"\t"}{.phase}{"\t"}{.message}{"\n"}{end}' ``` For each tenant cluster, also verify: * the ingress address is reachable and serving the expected certificate * the expected GPU nodes and `nvidia.com/gpu` capacity are visible * the NVIDIA Run:ai operator finishes its dependency checks * the scheduler is running * the cluster appears in the NVIDIA Run:ai UI In the central model, GPU allocation works from synced node capacity. Per-tenant GPU utilization views aren't available because DCGM Exporter runs outside each tenant cluster. ## Remove a dedicated deployment[​](#remove-a-dedicated-deployment "Direct link to Remove a dedicated deployment") Delete the tenant cluster or its generated dedicated StackInstance. Helm uninstall doesn't remove every namespace, persistent volume claim, or custom resource definition. Inspect the destination before reinstalling or deleting remaining data. ## Remove a central deployment[​](#remove-a-central-deployment "Direct link to Remove a central deployment") Order matters because the shared host foundation doesn't own tenant registrations: 1. Delete every tenant cluster. This removes its in-cluster agent but doesn't deregister the tenant. 2. Delete that tenant's `runai-reg-` registration StackInstance. Its delete hook deregisters the tenant. 3. Confirm that the NVIDIA Run:ai UI lists no clusters. 4. Delete the shared host StackInstance. danger Deleting the shared host StackInstance removes the central NVIDIA Run:ai control plane and its PostgreSQL, NATS, and Thanos Receive data volumes. Every remaining tenant loses its control plane. Removal can leave namespaces, completed hook Jobs, Prometheus volumes, and NVIDIA custom resource definitions. Review [Troubleshoot Stacks](https://vcluster.com/docs/platform/troubleshoot/stacks.md) before a clean reinstall. ## Customize a certified stack[​](#customize-a-certified-stack "Direct link to Customize a certified stack") Certified objects are read-only. Use the following guide to build your own template from the certified YAML. * Platform UI * Management API 1. Go to Management > Stacks & Apps, and select the **Stacks** tab. 2. Find the template you want to start from. Certified templates show a **Certified** badge next to the name. Click View, and the **CR** tab of the editor pane shows the full resource. 3. Copy the YAML from the editor pane. 4. Return to the **Stacks** tab, and click Create Stack Template. 5. Paste the YAML into the editor pane, replacing what's already there. 6. Set `metadata.name` to a new name, and remove the `vcluster.com/certified` annotation. Also remove the cluster-managed `metadata` fields that the certified object carries, such as `creationTimestamp`, `resourceVersion`, and `uid`. 7. Make any other changes, then click Create Stack Template. For task, parameter, and output syntax, see [Create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md). For every field, see the [StackTemplate reference](https://vcluster.com/docs/platform/api/resources/stacktemplate.md). 1) Connect `kubectl` to the vCluster Platform management API: ``` vcluster platform connect management ``` 2) List the Stack templates, and find the one to start from. The default table doesn't show annotations, so add a column for the certified marker: ``` kubectl get stacktemplates.management.loft.sh \ -o custom-columns='NAME:.metadata.name,CERTIFIED:.metadata.annotations.vcluster\.com/certified' ``` Certified templates report `true` in the `CERTIFIED` column, and your own templates report ``. 3) Export that template to a file: Modify the following with your specific values to generate a copyable command: TEMPLATE\_NAMErun-ai-dedicated-control-plane ``` kubectl get stacktemplates.management.loft.sh run-ai-dedicated-control-plane -o yaml > my-stack.yaml ``` 4) Open `my-stack.yaml` and set `metadata.name` to a new name. Delete the `vcluster.com/certified` annotation and the `status` block. Also delete the cluster-managed fields that the export carries, such as `metadata.creationTimestamp`, `metadata.generation`, `metadata.resourceVersion`, and `metadata.uid`. The metadata that remains looks like this: my-stack.yaml ``` apiVersion: management.loft.sh/v1 kind: StackTemplate metadata: name: my-run-ai-dedicated-control-plane spec: # Unchanged, apart from your edits. ``` 5) Make any other changes to `spec`. For task, parameter, and output syntax, see [Create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md). For every field, see the [StackTemplate reference](https://vcluster.com/docs/platform/api/resources/stacktemplate.md). 6) Create your template: ``` kubectl apply -f my-stack.yaml ``` Your customized template is a separate resource. It doesn't receive changes from later certified bundles, and a Platform upgrade that ships a new bundle doesn't replace it. A certified bundle is versioned and tested as a whole, and its parts reference each other by name. Task names appear in `dependsOn` edges and in `{{ .Outputs.. }}` references. An output can only read from a namespace the Stack itself deployed into. Chart versions are pinned to a combination that was tested together. Change any of these only after validating the whole workflow. The upstream [dedicated control-plane](https://github.com/loft-sh/vcluster-stacks/blob/main/certified-stacks/run-ai/dedicated-control-plane/README.md) and [central control-plane](https://github.com/loft-sh/vcluster-stacks/blob/main/certified-stacks/run-ai/central-control-plane/README.md) guides cover exhaustive parameters and implementation details for the NVIDIA Run:ai bundle. --- # Create a Stack template A Stack template defines applications that vCluster Platform deploys as one unit. It also defines the order in which those applications become ready. Use a Stack when applications have dependencies, need health gates, or pass values to later tasks. Each application is a task, tasks declare which other tasks they wait for, and the platform rolls them out in that order. A `StackTemplate` is a reusable, cluster-scoped blueprint. A `StackInstance` applies that blueprint to one tenant cluster or control plane cluster. Each task creates either an AppInstance or an ArgoCDApplication. ## Understand licensing[​](#understand-licensing "Direct link to Understand licensing") Stacks don't have a separate license feature. Each task requires the feature for the resource it creates: * An App task requires the Apps feature. * An Argo CD Application task requires the Argo CD Integration feature. * A Stack that mixes task types requires both features. Both features are available on every plan, including the free plan. If a task's feature is ever removed from the active plan, the StackInstance remains blocked with the `FeatureNotAllowed` reason. Learn more about [Apps](https://vcluster.com/docs/platform/understand/what-are-apps.md) and the [Argo CD integration](https://vcluster.com/docs/platform/integrations/argocd.md). ## Create a Stack template in the UI[​](#create-a-stack-template-in-the-ui "Direct link to Create a Stack template in the UI") 1. Go to Management > Stacks & Apps, and select the **Stacks** tab. 2. Click Create Stack Template. 3. Enter a name, display name, and description for the template. 4. Add App or Argo CD Application tasks. Configure dependencies between tasks in the graph. 5. Add parameters and published outputs when the Stack needs configurable or returned values. 6. Click Create Stack Template. You can also apply the resource through the management API using the **YAML** editor. ## Deploy a starter stack[​](#deploy-a-starter-stack "Direct link to Deploy a starter stack") Use this complete example as a starting point for your own Stack. It has no external chart or infrastructure dependencies. The first inline App task creates a ConfigMap from a Stack parameter. The second task waits for the first task to become healthy, then writes the captured greeting to another ConfigMap. 1. Connect `kubectl` to the vCluster Platform management API: ``` vcluster platform connect management ``` 2. Save the following resources as `starter-stack.yaml`. Set the project namespace and tenant cluster name for your destination, and set `spec.owner` to a user or team that can reach it. App tasks deploy as their owner, so a Stack with an App task needs one. Modify the following with your specific values to generate a copyable command: PROJECT\_NAMESPACEp-my-project OWNER\_USERadmin TENANT\_CLUSTER\_NAMEdevelopment starter-stack.yaml ``` # Define a reusable Stack template with two inline App tasks. apiVersion: management.loft.sh/v1 kind: StackTemplate metadata: name: starter-stack spec: displayName: Starter Stack description: Deploys two ConfigMaps to demonstrate task ordering and output handoff. parameters: - variable: greeting label: Greeting description: Message passed from the first task to the second task. type: string defaultValue: Hello from a Stack tasks: - name: backend app: template: spec: displayName: Starter backend description: Creates the greeting consumed by the frontend task. defaultNamespace: stack-starter wait: true config: manifests: |- apiVersion: v1 kind: ConfigMap metadata: name: starter-backend data: greeting: {{ .Values.greeting | quote }} outputs: - name: greeting fromResource: apiVersion: v1 kind: ConfigMap namespace: stack-starter name: starter-backend jsonPath: "{.data.greeting}" - name: frontend # Don't create this AppInstance until the backend task is healthy. dependsOn: - backend app: template: spec: displayName: Starter frontend description: Records the greeting captured from the backend task. defaultNamespace: stack-starter wait: true config: manifests: |- apiVersion: v1 kind: ConfigMap metadata: name: starter-frontend data: backendGreeting: {{ .Outputs.backend.greeting | quote }} publishedOutputs: - name: greeting fromTask: task: backend output: greeting --- # Apply the template to an existing tenant cluster. apiVersion: management.loft.sh/v1 kind: StackInstance metadata: name: starter-stack namespace: p-my-project spec: displayName: Starter Stack # App tasks deploy as this user or team, so an owner is required. owner: user: admin destination: virtualCluster: name: development templateRef: name: starter-stack parameters: greeting: Hello from my first Stack # Delete child AppInstances if their tasks are removed from the template. prunePolicy: Prune ``` 3. Create the StackTemplate and StackInstance: ``` kubectl apply -f starter-stack.yaml ``` 4. Wait for both tasks to report `Healthy`: ``` kubectl get stackinstance starter-stack -n \ -o jsonpath='{.status.phase}{"\n"}{range .status.tasks[*]}{.name}{"\t"}{.phase}{"\n"}{end}' ``` The StackInstance reports `Healthy`. The destination contains `starter-backend` and `starter-frontend` ConfigMaps in the `stack-starter` namespace. Change `spec.parameters.greeting` on the StackInstance to test reconciliation. Replace either inline App template with your own manifests, or use `templateRef` to reference a reusable App. ## Define a stack template[​](#define-a-stack-template "Direct link to Define a stack template") The following example deploys a database App before an API App. Create the referenced `postgresql` and `example-api` Apps before you create this template. stack-template.yaml ``` apiVersion: management.loft.sh/v1 kind: StackTemplate metadata: name: example-platform spec: displayName: Example application platform description: Deploys a database before the API that uses it. parameters: - variable: databaseStorage label: Database storage description: Persistent storage allocated to the database. type: string defaultValue: 10Gi tasks: - name: database app: templateRef: name: postgresql parameters: storage: "{{ .Values.databaseStorage }}" - name: api # Wait until the database AppInstance reports Healthy. dependsOn: - database timeout: 20m app: templateRef: name: example-api ``` Apply the template through the vCluster Platform management API: ``` vcluster platform connect management kubectl apply -f stack-template.yaml ``` Tasks without dependency edges can run concurrently. A dependent task starts only after every task in `dependsOn` becomes healthy. The following sections explain the task, parameter, and output syntax used in this example in detail. ## Configure tasks[​](#configure-tasks "Direct link to Configure tasks") Give each task a unique DNS-label-safe `name`. A duplicate name is rejected when the Stack is applied. A Stack supports up to 20 tasks. Each task can set its own `timeout`. A task without one uses the Stack's `defaults.taskTimeout`, or 10 minutes if that isn't set either. Set exactly one task type: * `app` creates an AppInstance. * `argoCDApplication` creates an ArgoCDApplication. Within the selected type, set either an inline `template` or a reusable `templateRef`. Don't set both. Parameters are valid only with `templateRef`. They pass through to the referenced template. caution For an `app` task, keep the combined StackInstance and task name short. The generated child name only enforces the 63-character Kubernetes object-name limit, but Helm's own release-name limit is 53 characters. A longer combination fails to deploy with a `release name is invalid` error that doesn't mention length. This task uses a reusable ArgoCDApplicationTemplate: ``` tasks: - name: workloads argoCDApplication: templateRef: name: application-workloads parameters: environment: production ``` note The Argo CD connector must be enabled before an Argo CD Application task can become healthy. See [Connect Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md). There's no field to skip a task based on a parameter. A Stack always resolves its full task set. To make a task a no-op under some condition, interpolate the `templateRef` name so it points at an App or ArgoCDApplicationTemplate that does nothing: ``` - name: gpu app: templateRef: name: 'gpu-operator{{ if eq .Values.gpuProvider "gke" }}-skip{{ end }}' ``` ## Pass parameters to tasks[​](#pass-parameters-to-tasks "Direct link to Pass parameters to tasks") Declare optional parameters under `spec.parameters`. Tasks read their values with `{{ .Values. }}`. A parameter declaration renders as a labeled form field in the UI, with a default value, and can enforce validation such as `required`, a `min`/`max` range, or a `validation` regular expression. Values that don't have declarations are still available to tasks. However, vCluster Platform can't validate those values or supply defaults. ## Pass outputs between tasks[​](#pass-outputs-between-tasks "Direct link to Pass outputs between tasks") A task can capture one scalar value from a Secret or namespaced Kubernetes resource. A later task reads the value with `{{ .Outputs.. }}` and must list the source task in `dependsOn`. See [Template values and outputs in Stacks](https://vcluster.com/docs/platform/how-to/use-go-template.md#template-values-and-outputs-in-stacks) for more Go template examples. ``` tasks: - name: database app: templateRef: name: postgresql outputs: - name: password fromSecret: namespace: database name: database-credentials key: password - name: serviceip fromResource: apiVersion: v1 kind: Service namespace: database name: postgresql # Select one scalar field with kubectl JSONPath syntax. jsonPath: "{.spec.clusterIP}" - name: api dependsOn: - database app: templateRef: name: example-api parameters: databasePassword: "{{ .Outputs.database.password }}" databaseAddress: "{{ .Outputs.database.serviceip }}" publishedOutputs: # Expose the service address without exposing the database password. - name: databaseAddress fromTask: task: database output: serviceip ``` Output names and the names of tasks that declare outputs can contain only lowercase letters and digits. Other task names can also contain hyphens. Resource outputs can't read cluster-scoped resources, and `fromResource` rejects `kind: Secret` outright. Read a Secret value with `fromSecret`, which also keeps it flagged as sensitive. The source namespace must be a namespace this Stack has already deployed an App or Argo CD Application into. vCluster Platform rejects a resource output whose source is in any other namespace before it reads the resource, even if the controller could otherwise reach it. Output references support template functions and pipelines. For example, `{{ .Outputs.database.password | quote }}` is valid. Keep the reference in the canonical `.Outputs..` form. An output reference can't control an `if`, `range`, or `with` action, declare a variable, or share one action with a `.Values` reference because parameters and outputs are rendered at different times. Parameter-driven control flow can contain output references in its body: ``` endpoint: '{{ if .Values.tls }}https://{{ .Outputs.api.hostname }}{{ else }}http://{{ .Outputs.api.hostname }}{{ end }}' ``` The text `<%` is reserved for internal output rendering and isn't valid in a task payload. Published outputs are available through the StackInstance outputs subresource. They aren't written to `status`. See the [StackInstanceOutputs reference](https://vcluster.com/docs/platform/api/resources/stackinstanceoutputs.md) for the response schema. Access to sensitive values requires separate permission to retrieve StackInstance outputs. See [Stack permissions](https://vcluster.com/docs/platform/administer/users-permissions/permissions/stacks.md). ## Test a stack template[​](#test-a-stack-template "Direct link to Test a stack template") Create a StackInstance that references the template. A destination and, once set, an owner are both immutable after creation. A tenant cluster destination must be in the same project as the StackInstance. There's no cross-project form of this field, so deploying the same Stack into tenant clusters across several projects takes one StackInstance per project, all referencing the same template. stack-instance.yaml ``` apiVersion: management.loft.sh/v1 kind: StackInstance metadata: name: example-platform namespace: p-example spec: displayName: Example application platform # App tasks deploy as this user or team, so an owner is required. owner: user: admin destination: virtualCluster: name: development templateRef: name: example-platform parameters: databaseStorage: 20Gi defaults: # Apply this timeout to tasks that don't set their own timeout. taskTimeout: 15m # Delete children removed from the resolved task graph. prunePolicy: Prune ``` ``` kubectl apply -f stack-instance.yaml kubectl get stackinstance example-platform -n p-example ``` The default `prunePolicy` is `Retain`. If a task is removed from the template, its existing child remains and appears in `status.orphanedApplications`. Set `Prune` to delete removed children in reverse dependency order. warning A referenced StackTemplate is resolved during every reconciliation. Stack templates don't have Stack revisions. Review template changes before applying them, to avoid an unwanted rollout to existing StackInstances. ## Next steps[​](#next-steps "Direct link to Next steps") * [Use a Stack](https://vcluster.com/docs/platform/use-platform/apps/use-stacks.md) to deploy a StackTemplate. * [Manage Certified Stacks](https://vcluster.com/docs/platform/administer/templates/certified-stacks.md) to use templates bundled with vCluster Platform. * [Manage Stacks declaratively](https://vcluster.com/docs/platform/how-to/gitops-end-to-end.md#manage-stacks-declaratively) to store StackTemplates and StackInstances in Git. * [Stack permissions](https://vcluster.com/docs/platform/administer/users-permissions/permissions/stacks.md) to configure RBAC for Stack resources. * [StackTemplate reference](https://vcluster.com/docs/platform/api/resources/stacktemplate.md) for the full field schema. * [Troubleshoot Stacks](https://vcluster.com/docs/platform/troubleshoot/stacks.md) to inspect blocked or failed tasks. --- # Create a template Templates standardize tenant cluster, namespace, App, and Stack configurations. Create templates through the vCluster Platform UI or by applying manifests directly to your cluster with the management API.
## Create a template using the UI[​](#create-a-template-using-the-ui "Direct link to Create a template using the UI") You can create templates directly in the UI:
* Tenant cluster template * App template * Stack template * Namespace template 1. Go to Management > Templates and, if shown, click the Tenant Clusters tab. 2. Click Create Tenant Cluster Template. 3. Provide your tenant cluster template with a name by replacing the `my-template` placeholder name, or by updating the manifest YAML `metadata.name` field. 4. **\[Optional]** Update the template with any additional [configuration options](#template-configuration-options). 5. Click **Create Template**. Apps are Helm charts, Kubernetes manifests, or bash scripts that define an application to be deployed into a cluster, namespace, or tenant cluster. App templates can be created through the UI. 1. Go to Management > Apps and, if shown, click the Helm Apps tab. 2. Click Create App Template. 3. Provide your tenant cluster template with a name by replacing the `my-template` placeholder name, or by updating the manifest YAML `metadata.name` field. 4. On the Definition tab, select the type of app (`kubectl`, Helm, or bash), and provide the necessary details. These vary depending on your application type and environment. 5. **\[Optional]** Use the Display Options tab to upload an app icon, which appears in the UI. 6. **\[Optional]** Use the Recommend App list to choose whether the app appears as recommended in the UI. 7. Click Create App. A Stack template defines a dependency-ordered group of Apps and Argo CD Applications. 1. Go to Management > Stacks & Apps, and select the Stacks tab. 2. Click Create Stack Template. 3. Add tasks, dependencies, parameters, and published outputs. 4. Click Create Stack Template. See [Create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) for the task schema, outputs, and a complete example. Namespace templates can be created directly in the UI to define default labels, annotations, and resource settings for new namespaces. 1. Go to Management > Templates and, if shown, click the Namespaces tab. 2. Click Create Namespace Template. 3. Provide your tenant cluster template with a name by replacing the `my-template` placeholder name, or by updating the manifest YAML `metadata.name` field. 4. **\[Optional]** Update the template with any additional desired [configuration options](#template-configuration-options). 5. Click Create Template. ## Create a template from a manifest[​](#create-a-template-from-a-manifest "Direct link to Create a template from a manifest") Alternatively, you can create a manifest and apply it to your cluster using `kubectl`, Helm, or ArgoCD. For example, to create a VirtualClusterTemplate manifest: 1. Log in to the vCluster Platform: ``` vcluster platform login ``` 2. Connect your kubeconfig to the platform management API: ``` vcluster platform connect management ``` 3. Create a manifest file named `virtualclustertemplate.yaml` that defines your template. You can use an example manifest as a starting point or export one from the UI. 4. Apply the manifest to the cluster: ``` kubectl apply -f virtualclustertemplate.yaml ``` #### Example tenant cluster manifest[​](#example-tenant-cluster-manifest "Direct link to Example tenant cluster manifest") ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterTemplate metadata: name: development-template labels: team: platform environment: development spec: displayName: "Development environment template" description: "Standardized development vCluster with appropriate resource limits and development tools pre-installed" template: metadata: labels: environment: development cost-center: engineering backup: enabled helmRelease: chart: version: 0.25.0 values: | controlPlane: distro: k8s: version: "1.28.4" resources: limits: cpu: "2000m" memory: "4Gi" requests: cpu: "200m" memory: "512Mi" statefulSet: highAvailability: replicas: 3 persistence: volumeClaim: enabled: true size: "10Gi" storageClass: "fast-ssd" networking: resolveDNS: - hostname: host.cluster.local target: hostname: cluster.local ingress: enabled: true rbac: clusterRole: enabled: true extraRules: - apiGroups: [""] resources: ["pods", "services"] verbs: ["get", "list", "create", "update", "delete"] sync: fromHost: nodes: enabled: true toHost: persistentVolumes: enabled: true ``` ## Configure template options[​](#template-configuration-options "Direct link to Configure template options") Templates include configuration sections that define behavior, content, and access. The following options apply to both tenant cluster templates and namespace templates. ### Template parameters[​](#template-parameters "Direct link to Template parameters") Template parameters make templates dynamic by exposing configurable fields that users can customize during deployment. When creating parameters, you define the variable name, data type, label, and description. Parameters can include validation rules such as allowed options, required fields, and default values. For example, you might create parameters for admin passwords, environment types, resource limits, or feature toggles. Users see these parameters as form fields. They can customize deployments without modifying the underlying template. ### Apps and objects[​](#apps-and-objects "Direct link to Apps and objects") The **Apps** section lets you select pre-built applications that install automatically when tenant clusters are created from the template. Choose from available apps such as ArgoCD, Prometheus Stack, Ingress controllers, or custom applications specific to your organization. Selected apps deploy immediately after tenant cluster creation. Development teams have required tools and services available without manual installation. This approach standardizes the application stack across tenant clusters. It also reduces setup time for development teams. The **Objects** section defines Kubernetes manifests that are applied during tenant cluster creation. These include ConfigMaps, Secrets, NetworkPolicies, or custom ClusterRoles that establish baseline configurations. For an example of injecting a custom ClusterRole to control in-cluster permissions, see [Custom ClusterRoles](https://vcluster.com/docs/platform/administer/users-permissions/permissions/vcluster.md#custom-clusterroles). ### Management access[​](#management-access "Direct link to Management access") Management access defines who can view, edit, or delete the template. It applies to the template itself, not to the tenant clusters created from it. You can assign an owner with full permissions and grant view-only or full-access roles to other users or teams. This ensures that only authorized administrators can modify templates while allowing teams to use them for deployment. The access model follows role-based access control (RBAC) to prevent unauthorized changes that could compromise consistency or security. ## Delete a template[​](#delete-a-template "Direct link to Delete a template") If a template is outdated, misconfigured, or no longer needed, you can remove it to keep your workspace clean and reduce confusion for users. Deleting a template removes it from the platform and prevents it from being used for future cluster creation. This does not affect any tenant clusters or resources that were already created from the template. Delete a template If you delete a template, any tenant cluster, app, or namespace that references it can no longer be edited or upgraded through the Platform UI’s form-based editor. To make changes, you must manually update the corresponding custom resource by editing its manifest directly. 1. Go to Management > Templates or Management > Apps. If a tab bar is shown, click the tab for the template type (Tenant Clusters, Namespaces, ArgoCD Apps, or Helm Apps). 2. Find the template you want to delete. 3. Click **Delete** and confirm the deletion. ## Security hardening[​](#security-hardening "Direct link to Security hardening") When creating templates for production environments, configure sync settings to prevent users from gaining unintended access to control plane cluster resources. ### Restrict sync configurations[​](#restrict-sync-configurations "Direct link to Restrict sync configurations") The vCluster sync configuration determines which resources are shared between the tenant cluster and control plane cluster. Unrestricted sync settings can allow users to: * Access secrets or configmaps from the control plane cluster * Create resources in the host namespace that affect other workloads * Sync custom resources that grant elevated permissions ### Hardened template example[​](#hardened-template-example "Direct link to Hardened template example") The following sync configuration restricts what can be synced to and from the control plane cluster: Hardened sync configuration ``` helmRelease: values: | sync: toHost: # Only sync pods - required for workloads pods: enabled: true # Disable syncing resources that could affect host cluster persistentVolumes: enabled: false priorityClasses: enabled: false fromHost: # Only sync nodes for scheduling information nodes: enabled: true # Do not sync secrets or configmaps from host # This prevents access to host cluster secrets ``` ### Policy enforcement[​](#policy-enforcement "Direct link to Policy enforcement") Use the `policies` section to enforce security standards within the tenant cluster: Security policies ``` helmRelease: values: | policies: # Enforce pod security standards podSecurityStandard: restricted # Set resource limits resourceQuota: enabled: true limitRange: enabled: true ``` For more information on vCluster configuration options, see the [vCluster configuration reference](https://vcluster.com/docs/vcluster/configure/vcluster-yaml). ## Next steps[​](#next-steps "Direct link to Next steps") * Configure [template parameters](https://vcluster.com/docs/platform/administer/templates/parameters.md) to make templates customizable for different environments and use cases. * Set up [template versioning](https://vcluster.com/docs/platform/administer/templates/versioning.md) to manage template updates and maintain deployment compatibility. ## Examples[​](#examples "Direct link to Examples") Tenant cluster template The following template creates a tenant cluster with ResourceQuota and LimitRange objects to control resource usage. It configures persistent storage, includes RBAC rules for operational tasks, and enables ingress support for apps. Template ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterTemplate metadata: name: development-template labels: team: platform environment: development spec: displayName: "Development Environment Template" description: "Standardized development vCluster with appropriate resource limits and operational tools pre-installed" template: metadata: labels: environment: development cost-center: engineering backup: enabled objects: |- apiVersion: v1 kind: Namespace metadata: name: dev-tools labels: environment: development --- apiVersion: v1 kind: ResourceQuota metadata: name: development-quota namespace: prod-tools spec: hard: requests.cpu: "4" requests.memory: 8Gi limits.cpu: "8" limits.memory: 16Gi pods: "20" --- apiVersion: v1 kind: LimitRange metadata: name: development-limits namespace: prod-tools spec: limits: - default: cpu: 500m memory: 512Mi defaultRequest: cpu: 100m memory: 128Mi type: Container helmRelease: chart: version: 0.25.0 values: | controlPlane: distro: k8s: version: "1.28.4" resources: limits: cpu: "2000m" memory: "4Gi" requests: cpu: "200m" memory: "512Mi" statefulSet: persistence: volumeClaim: enabled: true size: "10Gi" storageClass: "fast-ssd" networking: resolveDNS: - hostname: host.cluster.local target: hostname: cluster.local ingress: enabled: true rbac: clusterRole: enabled: true extraRules: - apiGroups: [""] resources: ["pods", "services"] verbs: ["get", "list", "create", "update", "delete"] sync: fromHost: nodes: enabled: true toHost: persistentVolumes: enabled: true accessPoint: ingress: {} spaceTemplate: metadata: {} access: - verbs: - get - list - create - update - delete users: - '*' ``` Preview environment template The following template creates a temporary preview environment with automatic auto sleep and deletion policies. It includes ingress authentication requirements, scheduled auto sleep and wake cycles, and baseline security policies for short-lived preview deployments. The `loft.sh/import-argocd: 'true'` label uses the legacy project-level Argo CD integration. For new setups, configure Argo CD registration through the connector-based integration instead. See [Connect to Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md). Preview environment template ``` kind: VirtualClusterTemplate apiVersion: management.loft.sh/v1 metadata: name: preview-template spec: displayName: vCluster Preview Template template: metadata: labels: loft.sh/import-argocd: 'true' annotations: sleepmode.loft.sh/delete-after: '72000' loft.sh/require-ingress-authentication: 'true' sleepmode.loft.sh/sleep-after: '600' sleepmode.loft.sh/ignore-user-agents: 'argo*' objects: | apiVersion: v1 kind: Namespace metadata: name: preview-hello-world-app --- apiVersion: v1 kind: Secret metadata: name: ghcr-login-secret namespace: preview-hello-world-app labels: loft.sh/project-secret-name: ghcr-login-secret data: .dockerconfigjson: e30K type: kubernetes.io/dockerconfigjson access: rules: - teams: - loft-admins clusterRole: cluster-admin pro: enabled: true helmRelease: chart: version: 0.31.0 values: |- external: platform: autoDelete: afterInactivity: 120000 sleepMode: enabled: true autoSleep: afterInactivity: 3h autoWakeup: schedule: 11 8-16 * *1-5 sync: toHost: ingresses: enabled: true secrets: all: true controlPlane: backingStore: etcd: embedded: enabled: true coredns: embedded: true policies: limitRange: enabled: true podSecurityStandard: baseline resourceQuota: enabled: true accessPoint: ingress: {} spaceTemplate: metadata: {} access: - verbs: - '*' subresources: - '*' users: - admin ``` Argo CD tenant cluster template The following template automatically installs Argo CD with ApplicationSet controller and configures TLS-enabled ingress with team-based RBAC. It includes scheduled auto sleep, enterprise features with embedded etcd and CoreDNS, and provides custom links for quick access to Argo CD UI and GitOps repository. Argo CD virtual cluster template ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterTemplate metadata: name: argo-cd-vcluster spec: displayName: Argo CD vCluster description: >- Creates a vCluster instance with Argo CD for use within a vCluster Platform Project. owner: team: loft-admins template: metadata: {} instanceTemplate: metadata: annotations: loft.sh/custom-links: |- Argo-CD=https://{{ .Values.loft.virtualClusterName }}-{{ .Values.loft.project }}-{{ .Values.loft.clusterAnnotations.domainPrefix }}.{{ .Values.loft.clusterAnnotations.domain }} GitOps-Repo=https://github.com/loft-demos/{{ .Values.loft.clusterAnnotations.domainPrefix }}-app/ apps: - name: argocd version: 2.14.10 parameters: | appValues: |- global: domain: {{ .Values.loft.virtualClusterName }}-{{ .Values.loft.project }}-{{ .Values.loft.clusterAnnotations.domainPrefix }}.{{ .Values.loft.clusterAnnotations.domain }} configs: cm: url: https://{{ .Values.loft.virtualClusterName }}-{{ .Values.loft.project }}-{{ .Values.loft.clusterAnnotations.domainPrefix }}.{{ .Values.loft.clusterAnnotations.domain }} create: true application.resourceTrackingMethod: annotation params: "server.insecure": true timeout.reconciliation: 0 rbac: policy.csv: |- g, Administration, role:admin g, Sales, role:admin g, Engineering, role:admin server: ingress: enabled: true annotations: nginx.ingress.kubernetes.io/force-ssl-redirect: "true" nginx.ingress.kubernetes.io/backend-protocol: "HTTP" ingressClassName: "nginx" # deprecated hostname: {{ .Values.loft.virtualClusterName }}-{{ .Values.loft.project }}-{{ .Values.loft.clusterAnnotations.domainPrefix }}.{{ .Values.loft.clusterAnnotations.domain }} tls: true ingressGrpc: enabled: false applicationSet: enabled: true logLevel: "debug" ingress: enabled: true ingressClassName: "nginx" # deprecated annotations: nginx.ingress.kubernetes.io/force-ssl-redirect: "true" hostname: {{ .Values.loft.virtualClusterName }}-{{ .Values.loft.project }}-{{ .Values.loft.clusterAnnotations.domainPrefix }}-appset.{{ .Values.loft.clusterAnnotations.domain }} tls: true notifications: argocdUrl: https://{{ .Values.loft.virtualClusterName }}-{{ .Values.loft.project }}-{{ .Values.loft.clusterAnnotations.domainPrefix }}.{{ .Values.loft.clusterAnnotations.domain }} cm: create: true logLevel: "debug" secret: labels: loft.sh/project-secret-name: argocd-notifications-secret - name: argo-cd-cluster-application-set objects: |- apiVersion: v1 kind: Namespace metadata: name: argocd --- apiVersion: v1 kind: Secret metadata: name: loft-demo-org-cred namespace: argocd labels: loft.sh/project-secret-name: loft-demo-org-cred argocd.argoproj.io/secret-type: repo-creds type: Opaque --- pro: enabled: true helmRelease: chart: version: 0.31.0 values: | sleepMode: enabled: true autoSleep: schedule: 0 22 * * * timeZone: America/New_York autoWakeup: schedule: 0 6 * * 1-5 external: platform: autoDelete: afterInactivity: 18000000 sync: toHost: ingresses: enabled: true customResources: argocdwebhooks.demo.loft.sh: enabled: true controlPlane: backingStore: etcd: embedded: enabled: true coredns: enabled: true embedded: true accessPoint: ingress: {} spaceTemplate: metadata: {} objects: |- apiVersion: v1 kind: Secret metadata: name: argo-webhook-url type: Opaque stringData: url: "https://{{ .Values.loft.virtualClusterName }}-{{ .Values.loft.project }}-{{ .Values.loft.clusterAnnotations.domainPrefix }}.{{ .Values.loft.clusterAnnotations.domain }}/api/webhook" --- apiVersion: v1 kind: Secret metadata: name: argo-appset-webhook-url type: Opaque stringData: url: "https://{{ .Values.loft.virtualClusterName }}-{{ .Values.loft.project }}-{{ .Values.loft.clusterAnnotations.domainPrefix }}-appset.{{ .Values.loft.clusterAnnotations.domain }}/api/webhook" ``` Parameterized tenant cluster template The following template demonstrates parameters for auto sleep timeout and Kubernetes version selection during deployment. Users can choose from predefined options with default values, and the template uses embedded etcd and CoreDNS for resource efficiency. Parameterized virtual cluster template ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterTemplate metadata: name: sleepy-vcluster-template annotations: argocd.argoproj.io/sync-options: Validate=false spec: displayName: Sleepy vCluster Template description: This template deploys a sleepy virtual cluster template: metadata: {} instanceTemplate: metadata: {} pro: enabled: true helmRelease: chart: name: vcluster version: 0.31.0 values: | sync: toHost: ingresses: enabled: true controlPlane: backingStore: etcd: embedded: enabled: true coredns: embedded: true distro: k8s: image: tag: "{{ .Values.k8sVersion }}" sleepMode: enabled: true autoSleep: afterInactivity: "{{ .Values.sleepAfter }}m" accessPoint: ingress: {} spaceTemplate: metadata: {} parameters: - variable: sleepAfter label: Sleep After Inactivity (minutes) type: number options: - '30' - '45' - '60' - '120' defaultValue: '45' - variable: k8sVersion label: k8sVersion description: Please select Kubernetes version options: - v1.33.1 - v1.32.5 - v1.31.9 - v1.30.13 defaultValue: v1.32.5 access: - verbs: - get users: - '*' ``` Versioned tenant cluster template The following template demonstrates versioning with a default base template and versioned variations. Version 1.0.0 includes enhanced features such as auto sleep configuration, custom links to Helm Dashboard, pre-installed helm-dashboard app, and user-configurable parameters for Kubernetes version and environment selection. Versioned virtual cluster template ``` kind: VirtualClusterTemplate apiVersion: management.loft.sh/v1 metadata: name: default-template spec: displayName: Default Virtual Cluster Template description: This virtual cluster template deploys a vCluster with embedded etcd and integrated CoreDNS. owner: team: loft-admins template: metadata: {} instanceTemplate: metadata: {} pro: {} helmRelease: chart: version: 0.24.0 values: |- sync: toHost: ingresses: enabled: true accessPoint: ingress: {} spaceTemplate: metadata: {} versions: - version: 1.0.0 template: metadata: annotations: sleepmode.loft.sh/ignore-user-agents: argo* instanceTemplate: metadata: annotations: loft.sh/custom-links: >- Helm-Dashboard=https://helm-dashboard-{{ .Values.loft.virtualClusterName }}-{{ .Values.loft.clusterAnnotations.domainPrefix }}.{{ .Values.loft.clusterAnnotations.domain }} labels: env: '{{ .Values.env }}' demos.loft.sh/project: '{{ .Values.loft.project }}' demos.loft.sh/owner: '{{ or (and .Values.loft.user .Values.loft.user.name) (and .Values.loft.team .Values.loft.team.name) }}' apps: - name: helm-dashboard namespace: helm-dashboard pro: enabled: true helmRelease: chart: name: vcluster version: 0.31.0 values: | sleepMode: enabled: true autoSleep: afterInactivity: 30m timeZone: America/New_York autoWakeup: schedule: 11 7-17 * * 1-5 external: platform: autoDelete: afterInactivity: 1800000 sync: toHost: pods: patches: - path: metadata.annotations expression: 'value["demos.loft.sh/project"]="{{ .Values.loft.project }}";value' - path: metadata.labels expression: 'value["demos.loft.sh/owner"]="{{ or (and .Values.loft.user .Values.loft.user.name) (and .Values.loft.team .Values.loft.team.name) }}";value' ingresses: enabled: true secrets: all: true controlPlane: advanced: workloadServiceAccount: enabled: true imagePullSecrets: - name: "hook-image-pull-secret" distro: k8s: image: tag: "{{ .Values.k8sVersion }}" service: labels: env: '{{ .Values.env }}' demos.loft.sh/project: '{{ .Values.loft.project }}' demos.loft.sh/owner: '{{ or (and .Values.loft.user .Values.loft.user.name) (and .Values.loft.team .Values.loft.team.name) }}' statefulSet: annotations: demos.loft.sh/pvClaimName: '{{ .Values.pvClaimName }}' labels: env: '{{ .Values.env }}' demos.loft.sh/project: '{{ .Values.loft.project }}' demos.loft.sh/owner: '{{ or (and .Values.loft.user .Values.loft.user.name) (and .Values.loft.team .Values.loft.team.name) }}' pods: labels: env: '{{ .Values.env }}' demos.loft.sh/project: '{{ .Values.loft.project }}' demos.loft.sh/owner: '{{ or (and .Values.loft.user .Values.loft.user.name) (and .Values.loft.team .Values.loft.team.name) }}' resources: limits: ephemeral-storage: 8Gi memory: 2Gi cpu: 1 backingStore: etcd: embedded: enabled: true coredns: embedded: true accessPoint: ingress: {} spaceTemplate: metadata: {} objects: | apiVersion: v1 kind: Secret metadata: name: hook-image-pull-secret labels: loft.sh/project-secret-name: ghcr-login-secret data: .dockerconfigjson: e30K type: kubernetes.io/dockerconfigjson parameters: - variable: k8sVersion label: k8sVersion description: Please select Kubernetes version options: - v1.33.2 - v1.32.6 - v1.31.10 - v1.30.14 defaultValue: v1.32.6 section: Kubernetes Environment - variable: env label: Deployment Environment description: >- Environment for deployments for this vCluster used as cluster label for Argo CD ApplicationSet Cluster Generator options: - dev - qa - prod defaultValue: dev section: Deployment Environment - version: 0.0.0 template: metadata: {} instanceTemplate: metadata: {} pro: enabled: true helmRelease: chart: version: 0.24.0 values: |- sync: toHost: ingresses: enabled: true accessPoint: ingress: {} spaceTemplate: metadata: {} access: - verbs: - '*' subresources: - '*' users: - admin ``` --- # Private Helm Repositories for Apps vCluster Platform apps allow specifying credentials to access private Helm chart repositories. ### Specify static credentials[​](#specify-static-credentials "Direct link to Specify static credentials") You can specify credentials either directly inside the app or using a project secret. The following example shows specifying the credentials directly within the app itself: ``` apiVersion: management.loft.sh/v1 kind: App metadata: name: my-app spec: config: chart: name: argo-cd repoURL: https://private-repo.com/my-repo username: my-username password: my-password ``` You can also reference a project secret using: ``` apiVersion: management.loft.sh/v1 kind: App metadata: name: my-app spec: config: chart: name: argo-cd repoURL: https://private-repo.com/my-repo usernameRef: projectSecretRef: name: my-secret project: my-project key: my-username-key passwordRef: projectSecretRef: name: my-secret project: my-project key: my-password-key ``` ### Use dynamic credentials (ECR and so on)[​](#use-dynamic-credentials-ecr-and-so-on "Direct link to Use dynamic credentials (ECR and so on)") For dynamic credentials, such as ECR credentials, use [external-secrets](https://external-secrets.io/) and export the credentials to a project secret. For example, for ECR credentials, this can be done using: ``` apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: "ecr-secret" spec: refreshInterval: "1h" target: name: ecr-secret template: metadata: labels: loft.sh/project-secret: "true" dataFrom: - sourceRef: generatorRef: apiVersion: generators.external-secrets.io/v1alpha1 kind: ECRAuthorizationToken name: "ecr-gen" --- apiVersion: generators.external-secrets.io/v1alpha1 kind: ECRAuthorizationToken metadata: name: ecr-gen spec: # specify aws region (mandatory) region: eu-west-1 # choose an authentication strategy # if no auth strategy is defined it falls back to using # credentials from the environment of the controller. auth: # 1: static credentials # point to a secret that contains static credentials # like AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY secretRef: accessKeyIDSecretRef: name: "my-aws-creds" key: "key-id" secretAccessKeySecretRef: name: "my-aws-creds" key: "access-secret" ``` tip Remember to set the label `loft.sh/project-secret` on the Kubernetes secret, in order for Loft to pick it up correctly. Also make sure the secret is created in the correct project namespace in the form of `p-`. Then just reference the created secret within the app itself: ``` apiVersion: management.loft.sh/v1 kind: App metadata: name: my-app spec: config: chart: name: argo-cd repoURL: https://private-repo.com/my-repo usernameRef: projectSecretRef: name: ecr-secret project: my-project key: my-username-key passwordRef: projectSecretRef: name: ecr-secret project: my-project key: my-password-key ``` If an app instance fails to deploy because of a `usernameRef` or `passwordRef`, see [Secret and credential resolution failures](https://vcluster.com/docs/platform/troubleshoot/template-sync-failures.md#secret-and-credential-resolution-failures). --- # Templates overview Templates define reusable configurations for creating tenant clusters, applications, Stacks, and namespaces. They allow platform teams to standardize resource provisioning while giving users a controlled way to customize certain settings. Tenant cluster and namespace templates are implemented using Kubernetes custom resources (VirtualClusterTemplate and SpaceTemplate) and can be managed with tools such as `kubectl`, Helm, or Argo CD. Reusable app configurations use a separate pair of custom resources (App and AppInstance) with their own versioning and drift-detection behavior. StackTemplate and StackInstance resources define dependency-ordered groups of Apps and Argo CD Applications. ArgoCDApplicationTemplate, a separate cluster-scoped resource, defines one reusable Argo CD Application blueprint that a Stack's `argoCDApplication` tasks reference. Each template type serves a specific purpose. Tenant cluster templates define the base configuration for tenant clusters, including the Kubernetes version, control plane, networking, storage settings, and sync behavior. App templates provide reusable application configurations using Helm charts or Kubernetes manifests. Stack templates coordinate several applications through dependencies, health gates, and outputs, and can reference ArgoCDApplicationTemplates for their Argo CD Application tasks. Namespace templates predefine configurations for new namespaces, including role-based access control, resource quotas, and labels. Templates allow team administrators to define which settings users can modify through parameters. For example, you can expose only specific fields, such as the number of replicas or a set of approved storage classes, while keeping the rest of the configuration fixed. This approach reduces the risk of misconfiguration, while enabling flexibility, and helps with security. Tenant cluster and app templates can include pre-installed components, such as monitoring agents or ingress controllers. Additional Kubernetes resources like ConfigMaps, RBAC, or [NetworkPolicies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md) can be applied to either the control plane cluster or the tenant cluster itself. Tenant cluster, namespace, and App templates can support versioned upgrade workflows. StackTemplates don't have Stack revisions. A StackInstance resolves its referenced StackTemplate during reconciliation. ## Template types[​](#template-types "Direct link to Template types") vCluster Platform includes templates for tenant clusters, applications, Stacks, and namespaces.
| Template type | Custom resource | Purpose | Common Use Cases | | ------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------- | | Tenant cluster | VirtualClusterTemplate | Specifies how tenant clusters are provisioned, including control plane settings, synchronization behavior, and metadata. | Provision tenant clusters with consistent architecture, features, and default settings. | | App | App | Defines reusable application configurations, such as Helm charts or Kubernetes manifests. | Deploy standard applications across projects or spaces using consistent settings. | | Stack | StackTemplate | Defines a parameterized task graph of Apps and Argo CD Applications. | Deploy dependencies in order, wait for health, and pass outputs between applications. | | Argo CD Application | ArgoCDApplicationTemplate | Defines a reusable Argo CD Application blueprint. | Reference the same Argo CD Application definition from multiple Stack tasks. | | Namespace | SpaceTemplate | Predefines configurations for new namespaces, including role-based access control, resource quotas, and labels. | Automatically apply baseline policies when a namespace is created. |
You can [create templates](https://vcluster.com/docs/platform/administer/templates/create-templates.md) using the vCluster Platform UI for guided configuration or by applying manifests directly with `kubectl`, Helm, or Argo CD. To define a task graph, see [Create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md). ## Apply parameters and versioning[​](#apply-parameters-and-versioning "Direct link to Apply parameters and versioning") Templates support optional parameterization. Tenant cluster, namespace, and App templates also support versioning. You can apply: * [Parameters](https://vcluster.com/docs/platform/administer/templates/parameters.md) - Enable dynamic customization of template fields, allowing users to specify values for CPU limits, labels, environment types, and other configuration options during vCluster provisioning. * [Versioning](https://vcluster.com/docs/platform/administer/templates/versioning.md) - Manage tenant cluster, namespace, and App template releases. StackTemplates don't use this versioning model. ## Next steps[​](#next-steps "Direct link to Next steps") * [Create a template](https://vcluster.com/docs/platform/administer/templates/create-templates.md) to define a reusable configuration. * [Create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) to coordinate dependent applications. * [Manage Certified Stacks](https://vcluster.com/docs/platform/administer/templates/certified-stacks.md) to use the NVIDIA Run:ai templates bundled with Platform. * Optionally, [configure template parameters](https://vcluster.com/docs/platform/administer/templates/parameters.md) to allow user customization. * Optionally, [set up template versioning](https://vcluster.com/docs/platform/administer/templates/versioning.md) to manage template updates. * If a tenant cluster, namespace, or app instance isn't picking up its template, see [Troubleshoot template and app sync failures](https://vcluster.com/docs/platform/troubleshoot/template-sync-failures.md). --- # Parameters Template parameters enable customization of tenant cluster templates. The same mechanism applies to SpaceTemplate, App, and ArgoCDApplicationTemplate resources. An ArgoCDApplication supplies values to an ArgoCDApplicationTemplate's parameters. It doesn't define a parameter schema of its own. Platform administrators define a set of parameters that expose specific fields to users during tenant cluster creation. These parameters allow customization of values like replica counts, image tags, or ingress domains without granting access to the full configuration. When a user creates a tenant cluster from a parameterized template, the platform presents a form that collects values for each parameter. These values are injected into the rendered configuration using Go templating syntax (`{{ .Values.parameterName }}`). This provides some flexibility within defined boundaries. You can adjust some behaviors, such as increasing memory requests or enabling optional apps, while platform teams can still retain control over foundational settings like network configuration, cluster version, and permissions. Each parameter supports type definitions, validation rules, default values, and allowed value lists. These controls ensure that inputs are valid and align with standards. Parameters work well for supporting self-service provisioning at scale and also maintain governance, consistency, and operational safety across environments. ## Parameter types and validation[​](#parameter-types-and-validation "Direct link to Parameter types and validation") When defining parameters in a vCluster template, you must specify the expected input type and optionally enforce validation rules. ### Supported parameter types[​](#supported-parameter-types "Direct link to Supported parameter types") Templates support basic parameter types for collecting user input during cluster creation:
| Type | Use case | | ----------------------------------- | ------------------------------------------------------- | | [`string`](#string-parameter) | Names, labels, image tags, configuration values | | [`number`](#string-parameter) | Replica counts, port numbers, resource limits | | [`boolean`](#boolean-parameter) | Feature toggles, enable/turn off flags | | [`password`](#password-parameter) | API tokens, secrets, sensitive input values | | [`multiline`](#multiline-parameter) | Configuration files, startup scripts, long descriptions | ### Validation options[​](#validation-options "Direct link to Validation options") You can apply validation rules to ensure that users provide valid, safe, and expected input when filling out template parameters. These rules improve consistency and reduce configuration errors during tenant cluster creation. * **Required fields**: Mark a parameter as required to prevent users from submitting the form without entering a value. This is useful for mandatory inputs like cluster names or resource sizes. * **Allowed values**: Limit a parameter to a predefined list of options, such as regions, environments, or storage classes. This helps simplify user choices and ensures configuration compliance. * **Regular expressions**: Use regular expression patterns to enforce format requirements for string fields—for example, lowercase-only names or domain-style inputs. * **Numeric ranges**: Set minimum and maximum values for number-type parameters to ensure values like CPU, memory, or replica counts fall within safe limits. ## Define parameters using the UI[​](#define-parameters-using-the-ui "Direct link to Define parameters using the UI") The platform UI provides an intuitive interface for creating and managing template parameters. 1. Navigate to your template in the platform UI and click **Add parameter** in the template editor. 2. From the **Section** drop-down list, choose an existing section or create a new one (such as "Environment Settings" or "Resource Configuration"). Sections help organize related parameters and improve the user experience during deployment. 3. Enter a unique **Identifier (variable)** name. This becomes the variable key used in your template (for example, `cpu_limit` becomes `{{ .Values.cpu_limit }}`). Use descriptive names that clearly indicate the parameter's purpose. 4. Select the appropriate **Type** for your parameter from the drop-down list. Consider what kind of input makes most sense for your use case. 5. Provide a clear **Label** that end users can see in the deployment form. Make it descriptive and user-friendly (for example, "CPU Limit" instead of "cpu\_limit"). 6. Add a comprehensive **Description** that explains what the parameter controls and provides guidance on appropriate values. You can include examples when helpful. 7. Toggle **Required** if users must provide a value for this parameter before deploying. Before marking a parameter as required, consider whether providing a default value would better support user workflows and reduce friction during cluster creation. 8. If you want to restrict user choices, use **Allowed Options** to create a drop-down list. Click **+ Add option** for each allowed value. This is particularly useful for environment names, regions, or configuration presets. 9. Click **Done** to save the parameter and add it to your template. ### Parameter configuration guidelines[​](#parameter-configuration-guidelines "Direct link to Parameter configuration guidelines") To make your templates intuitive, scalable, and safe to use, follow these best practices when defining parameters. * **Group related parameters**: Organize inputs into logical sections such as *Resource settings*, *Network configuration*, or *Security options*. * **Use descriptive names**: Choose parameter identifiers that clearly communicate their purpose. * **Use clear descriptions**: Include concise guidance, examples, or constraints in the parameter description field to help users enter correct values. You can also apply advanced parameter options to modify behavior and enforce consistency: * **Default values**: Use defaults to reduce manual input and guide users toward standard configurations. For example, you can set a default replica count or enable a common add-on by default. * **Custom validation**: Add rules to control input format or range. You can limit values using regular expressions, set minimum or maximum numbers, or define a list of allowed options. ## YAML parameter definitions[​](#yaml-parameter-definitions "Direct link to YAML parameter definitions") For templates managed through Git or automated workflows, you can define parameters directly in YAML format. ### Basic parameter structure[​](#basic-parameter-structure "Direct link to Basic parameter structure") ``` parameters: - variable: environment_type label: Environment Type description: Select the target environment for this virtual cluster type: string required: true options: - development - staging - production section: Environment Settings ``` #### Define string parameter with validation[​](#string-parameter "Direct link to Define string parameter with validation") String parameter with validation ``` - variable: cluster_name label: Cluster Name description: Enter a name for your virtual cluster (3-20 characters, lowercase letters and hyphens only) type: string required: true validation: "^[a-z][a-z0-9-]*[a-z0-9]$" section: Basic Configuration ``` #### Number parameter with range[​](#number-parameter "Direct link to Number parameter with range") Number parameter with range ``` - variable: replica_count label: Replica Count description: Number of application replicas (1-10) type: number required: true min: 1 max: 10 defaultValue: 3 section: Scaling Settings ``` #### Boolean parameter for feature toggle[​](#boolean-parameter "Direct link to Boolean parameter for feature toggle") Boolean parameter for feature toggle ``` - variable: enable_monitoring label: Enable Monitoring description: Install Prometheus and Grafana monitoring stack type: boolean defaultValue: true section: Optional Features ``` #### Password parameter for sensitive data[​](#password-parameter "Direct link to Password parameter for sensitive data") Password parameter for sensitive data ``` - variable: admin_password label: Admin Password description: Password for the admin user (minimum 8 characters) type: password required: true validation: "^.{8,}$" section: Security Settings ``` #### Multiline parameter for configuration[​](#multiline-parameter "Direct link to Multiline parameter for configuration") Multiline parameter for configuration ``` - variable: custom_config label: Custom Configuration description: Additional YAML configuration to apply to the virtual cluster type: multiline section: Advanced Settings ``` ### Complex parameter validation[​](#complex-parameter-validation "Direct link to Complex parameter validation") Use advanced validation patterns to restrict and guide user input when filling out template parameters. #### Enum with descriptions[​](#enum-with-descriptions "Direct link to Enum with descriptions") The following example defines a parameter with a fixed list of allowed options, each with a description to help users choose the correct value. This is useful for selecting cluster sizes, environments, or configuration tiers. Enum with descriptions (resource tier selection) ``` - variable: instance_size label: Instance Size description: Choose the resource allocation tier for your virtual cluster type: string required: true options: - value: small description: 1 CPU, 2GB RAM - suitable for development - value: medium description: 2 CPU, 4GB RAM - suitable for testing - value: large description: 4 CPU, 8GB RAM - suitable for production workloads section: Resource Configuration ``` ## Access parameter values in templates[​](#access-parameter-values-in-templates "Direct link to Access parameter values in templates") After you define parameters, you can access their values in your template using Go template syntax. The platform makes all parameter values available under the `.Values` object. ### Basic parameter access[​](#basic-parameter-access "Direct link to Basic parameter access") ``` controlPlane: replicas: {{ .Values.replica_count }} ``` ### Conditional template logic[​](#conditional-template-logic "Direct link to Conditional template logic") Use Go template conditionals to create dynamic configurations based on parameter values: ``` # Set control plane resource limits dynamically based on environment type controlPlane: resources: limits: cpu: {{- if eq .Values.environment_type "production" }}4000m {{- else if eq .Values.environment_type "staging" }}2000m {{- else }}1000m {{- end }} memory: {{- if eq .Values.environment_type "production" }}8Gi {{- else if eq .Values.environment_type "staging" }}4Gi {{- else }}2Gi {{- end }} # Enable persistent volume if requested in values.yaml {{- if .Values.enable_persistence }} storage: persistentVolume: enabled: true size: "{{ .Values.storage_size }}" {{- end }} # Monitoring should be handled outside of vcluster.yaml. Use Helm subcharts or manifests. ``` ### Use string manipulation and defaults[​](#use-string-manipulation-and-defaults "Direct link to Use string manipulation and defaults") Use Go template functions for string manipulation and providing fallback values: ``` metadata: name: "{{ .Values.cluster_name | lower }}" labels: # Use default value if parameter is empty cost-center: "{{ .Values.cost_center | default "engineering" }}" # Combine multiple parameters full-name: "{{ .Values.project_name }}-{{ .Values.environment_type }}" # Transform string values normalized-name: "{{ .Values.cluster_name | lower | replace "_" "-" }}" ``` Available template functions vCluster templates use [Go templates](https://pkg.go.dev/text/template) with Helm. This enables dynamic rendering of values using functions like `default`, `lower`, and `replace`. Most functions are mentioned in the [Sprig library](https://masterminds.github.io/sprig/). The following table summarizes the types of functions that are commonly available.
| Category | Example functions | | ------------- | --------------------------------------------------------- | | String | `trim`, `lower`, `upper`, `title`, `contains`, `replace` | | Numbers | `add`, `sub`, `div`, `mod`, `max`, `min`, `ceil`, `floor` | | Logic / Flow | `default`, `ternary`, `eq`, `ne`, `and`, `or`, `not` | | Lists / Dicts | `merge`, `hasKey`, `pluck`, `keys`, `values`, `len` | | Time | `now`, `date`, `dateInZone` | | Type checks | `kindIs`, `typeIs`, `isSet`, `empty` | Some functions are disabled in vCluster templates due to security restrictions.
| Function | Reason | | ----------- | ------------------------------------------------ | | `env` | Disabled to prevent env access | | `envExpand` | Disabled to avoid string injection from env vars | | `expandenv` | Same as `envExpand` | vCluster Platform extends Helm’s function set with custom helpers for working with structured config formats.
| Function | Description | | --------------- | ----------------------------------------- | | `toYaml` | Converts a map or value to YAML | | `toToml` | Converts a map to TOML format | | `toJson` | Converts a value to JSON | | `fromYaml` | Parses a YAML string into a map | | `fromYamlArray` | Parses a YAML list into an array of maps | | `fromJson` | Parses a JSON string into a map | | `fromJsonArray` | Parses a JSON array into an array of maps | ## Parameter validation and error handling[​](#parameter-validation-and-error-handling "Direct link to Parameter validation and error handling") vCluster templates support input validation and error handling to ensure users provide safe, structured, and meaningful values during cluster creation. ### Input validation strategies[​](#input-validation-strategies "Direct link to Input validation strategies") You can apply format rules, numeric constraints, and other validation checks directly in your parameter definitions. #### Format validation with regular expressions[​](#format-validation-with-regular-expressions "Direct link to Format validation with regular expressions") Use a regular expression pattern to enforce formatting for string inputs like Kubernetes versions, domain names, or custom labels. ``` - variable: kubernetes_version label: Kubernetes Version description: Kubernetes version (e.g., 1.28.4, 1.27.8) type: string required: true validation: "^1\.(2[4-9]|[3-9][0-9])\.[0-9]+$" section: Configuration ``` #### Range validation for numbers[​](#range-validation-for-numbers "Direct link to Range validation for numbers") Set minimum and maximum values to control numeric inputs such as replica counts, node limits, or storage sizes. ``` - variable: worker_nodes label: Worker Nodes description: Number of worker nodes (1-20) type: number required: true min: 1 max: 20 section: Scaling ``` #### Complex validation with multiple constraints[​](#complex-validation-with-multiple-constraints "Direct link to Complex validation with multiple constraints") You can combine allowed values and validation patterns to enforce both selection and formatting rules. ``` - variable: storage_class label: Storage Class description: Kubernetes storage class for persistent volumes type: string required: true options: - fast-ssd - standard - archive validation: "^(fast-ssd|standard|archive)$" section: Storage ``` ### Error handling in templates[​](#error-handling-in-templates "Direct link to Error handling in templates") You can include conditional logic and error handling in your templates to catch missing values or apply fallback defaults. ``` # Fail if required parameter is missing {{- if not .Values.environment_type }} {{- fail "environment_type parameter is required" }} {{- end }} # Configure control plane controlPlane: # Default to 1 replica if none is set replicas: {{ .Values.replica_count | default 1 }} resources: limits: # Set CPU limit based on value or fallback to environment cpu: {{- if .Values.cpu_limit }} {{ .Values.cpu_limit }} {{- else if eq .Values.environment_type "production" }}2000m {{- else }}1000m {{- end }} ``` ## Platform-specific parameter values[​](#platform-specific-parameter-values "Direct link to Platform-specific parameter values") The platform automatically provides built-in parameter values based on the deployment context. These values are available under the `loft` key and can be used without explicit definition: | Context Object | Key | Description | | ----------------------- | ------------------------- | -------------------------------------------- | | Project | `project` | Name of the project that owns the deployment | | Space | `space` | Name of the space (if applicable) | | VirtualCluster | `virtualClusterName` | Name of the tenant cluster (if applicable) | | Cluster | `cluster` | Name of the physical Kubernetes cluster | | VirtualClusterNamespace | `virtualClusterNamespace` | Namespace of the tenant cluster | You can access these platform-specific values in your template: Accessing platform parameters ``` labels: my-label: "{{ .Values.loft.virtualClusterName }}" project: "{{ .Values.loft.project }}" cluster: "{{ .Values.loft.cluster }}" ``` ### Argo CD application parameter values[​](#argo-cd-application-parameter-values "Direct link to Argo CD application parameter values") `ArgoCDApplicationTemplate.spec.parameters` defines the parameter schema (types, validation, defaults) the same way tenant cluster templates do. `ArgoCDApplication.spec.parameters` supplies values against that schema. It doesn't define parameters itself. Both `ArgoCDApplicationTemplate` and inline `ArgoCDApplication` specs can interpolate the built-in `loft` platform-specific values described above, whether or not a parameter schema is involved. When an `ArgoCDApplication` targets a control plane cluster (`destination.cluster`) rather than a tenant cluster, `loft.project` and `loft.virtualClusterName` are empty strings. That destination isn't scoped to a project or tenant cluster. ### Connected control plane cluster parameter values[​](#connected-control-plane-cluster-parameter-values "Direct link to Connected control plane cluster parameter values") In addition to Platform-specific parameter values, annotations on the Platform connected control plane cluster `Cluster` resource are available under the `loft.clusterAnnotations` key for the `clusterRef.cluster` selected to deploy the tenant cluster instance. You can access these `loft.clusterAnnotations` based values in your template. A `Cluster` resource annotation of `example.vcluster.com/timezone: America/New_York` might be used as follows: ``` values: |+ sleepMode: enabled: true autoSleep: afterInactivity: 35m timeZone: '{{ default "America/New_York" (index .Values.loft.clusterAnnotations "example.vcluster.com/timezone") }}' autoWakeup: schedule: 10 7-17 * * 1-5 ``` --- # Versioning Templates | | | Enterprise | | | | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Template Versioning](https://vcluster.com/docs/platform/administer/templates/versioning.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Versioning a template allows you to make changes over time to a template while retaining the current version/iteration. Resources (for example tenant clusters, namespaces) that are deployed with templates that have updated versions show up with a warning indicating that a newer template version is available. ## Template version strings[​](#template-version-strings "Direct link to Template version strings") When creating a tenant cluster from a template, you can specify a template version string. If omitted, the latest available version is selected by default. You can use the `X` wildcard in the major, minor, or patch components of the version string to enable automatic upgrades. This allows tenant clusters to automatically update to newer compatible versions of the template. For example given a template version `1.0.0`: * If you set the version string to `1.X.X`, and a new template version `1.1.0` is added, the tenant cluster will automatically upgrade to `1.1.0`. * If a version `2.0.0` is added instead, the tenant cluster is not upgraded, because the major version does not match. To always upgrade to the latest available version, you can `X.X.X` as the version string. note Avoid using `X.X.X` unless necessary. Use the `X` placeholder only in the minor or patch position—for example, `1.X.X` or `1.2.X`—to maintain greater control over version updates. ## How template changes propagate to existing resources[​](#how-template-changes-propagate-to-existing-resources "Direct link to How template changes propagate to existing resources") Whether a template change reaches an already-created tenant cluster or namespace depends on whether the template is versioned. For a **versioned** template, every reconcile re-evaluates the resource's version string against the versions available on the template, and applies whichever version matches. This is what makes wildcard version strings such as `1.X.X` self-updating. As soon as a new matching version is added, it's applied automatically. If a resource pins an exact version, edits to that specific version's configuration still propagate automatically, but adding a newer version alongside it does not, since the exact pin never matches anything else. For a template with **no versions** (a single, unversioned `spec.template`), an edit to the template only reaches resources that reference it in three cases. The resource is being created for the first time, its `templateRef.syncOnce` field is set to `true`, or its version string is exactly `x.x.x`. Otherwise, the resource keeps running its last-applied configuration, and the platform UI surfaces the pending change as a **Template Changes Available** notice without applying it. To force a one-time sync after editing an unversioned template, set `syncOnce` on the affected resource: Modify the following with your specific values to generate a copyable command: TEMPLATE\_NAMEmy-template ``` spec: templateRef: name: my-template syncOnce: true ``` `syncOnce` resets to `false` automatically once the resource's template resolves, so it needs to be set again for any subsequent edit. A `TemplateResolved` failure, for example `ErrorRenderingTemplate`, leaves `syncOnce` set, since the sync never got that far. Once resolution succeeds, `syncOnce` clears even if applying the change afterward still fails. To have a resource pick up template edits automatically without setting `syncOnce` each time, use `x.x.x` as the version string instead, which behaves like a versioned template's wildcard match even though the referenced template itself has no versions. Unlike the `X`/`x` wildcards on a versioned template, which match regardless of case, this `x.x.x` sentinel is checked as an exact, case-sensitive string. `X.X.X` doesn't trigger it. The version gets cleared instead, and the resource falls back to needing `syncOnce` for each edit. If a template change doesn't reach a resource the way you expect, see [Troubleshoot template and app sync failures](https://vcluster.com/docs/platform/troubleshoot/template-sync-failures.md). ## Create a template version[​](#create-a-template-version "Direct link to Create a template version") You can add new versions to your templates using the UI. The steps are the same for all resources. note Alternatively, you can use a GitOps workflow by applying an updated VirtualClusterInstance resource to the vCluster Platform cluster using tools such as `kubectl` or Argo CD. Semantic versioning While not strictly mandatory, you should follow standard [semantic versioning](https://semver.org) practices for your template versioning. vCluster Platform refers to the version numbers provided when determining whether to indicate if a template has a newer version available. 1. Go to Management > Templates or Management > Apps. If a tab bar is shown, click the tab for the template type you want to version. 2. Find the template you want to add a new version to in the list. Click the actions menu next to its name and click Add New Version. 3. In the pop-up window, enter the new version string for your App. Then, click Add New Version. 4. In the menus displayed in the right, make any modifications desired for your new template version. 5. Click Save Changes. ## Update a template[​](#update-a-template "Direct link to Update a template") You can change the template and/or template version in the UI. 1. Select the project containing the resource using the project selector dropdown. 2. Click the resource type on the project page. 3. Click the actions menu next to the resource you want to modify and select Change Template. 4. In the pop-up window, select the name of the template from the list. The applied template is already selected, so if you are changing the template version, you don't need to do anything for this step. 5. If the template has multiple versions, a version selector appears next to the template selector. Select your desired template version. 6. Click Change Template. --- # Create teams in the platform Teams in the platform provide a way to organize users and manage permissions collectively. Instead of assigning permissions to individual users, you can create teams, add users to these teams, and then assign permissions to the entire team. This approach streamlines access management and creates a more scalable permission structure. ## Create a team[​](#create-a-team "Direct link to Create a team") Teams provide the foundation for a scalable permission model in vCluster platform, allowing you to manage access for groups of users efficiently. 1. Go to Access & Secrets > Users & Roles and select the Teams tab. 2. Click the Add Team button. 3. In the configuration sheet that opens, give your new team a name by replacing the 'my-team' placeholder name, or by updating the manifest YAML 'metadata.name' field. 4. **\[Optional]** Update the team with any additional desired configuration options. 1. The Team Members tab contains settings that govern which users or groups are members of this team. The Team Members selector allows you to include individual users directly into the team, while the SSO groups field allows you to include users whose standard kubernetes groups (clusterroles) match the provided values set here. 2. Advanced Options allow for configuring cluster roles that apply to this team, as well as Image Pull Secrets, that allow users in this team to automatically receive the provided image pull secrets upon logging into vCluster Platform (using the vCluster CLI). 3. Access allows you to define which users and teams have access to modify the team you are currently creating. 5. Click on the Save Changes button. ## Team configuration options[​](#team-configuration-options "Direct link to Team configuration options") When creating a team, you can configure several important settings: ### Team membership[​](#team-membership "Direct link to Team membership") The **Team Members** tab controls which users belong to your team: * **Users as Members**: Add individual users directly to the team. * **SSO Groups as Members**: Include users based on their standard Kubernetes groups (ClusterRoles). Users whose group memberships match the values you set here, automatically become members of this team. This dual approach allows for both manual user assignment and automatic, identity-based team membership through your SSO provider. ### Advanced options[​](#advanced-options "Direct link to Advanced options") The **Advanced Options** section lets you configure: * **Cluster Roles**: Define which Kubernetes cluster roles apply to all members of this team. * **Image Pull Secrets**: Configure registry credentials that team members automatically receive upon logging into vCluster Platform through the vCluster CLI. Image pull secrets are particularly valuable for teams working with private container registries, as they eliminate the need to manually configure registry access for each user. ### Access controls[​](#access-controls "Direct link to Access controls") The **Access** section determines who can modify the team you're creating: * Specify which users and other teams have permission to edit this team's settings, add or remove members, or delete the team. * These controls implement a permission model for managing the team itself. ## Integration[​](#integration "Direct link to Integration") Teams in the platform integrate directly with Kubernetes Role-Based Access Control (RBAC) system. When you create a team: 1. The platform creates corresponding RBAC objects in the underlying Kubernetes cluster. 2. Any cluster roles assigned to the team automatically apply to all team members. 3. Team memberships propagate to all tenant clusters managed by the platform. This integration means that teams defined in vCluster platform affect permissions across all managed environments while maintaining Kubernetes-native security controls. * **Project teams**: Create teams that align with your organizational projects to give all team members appropriate access to relevant resources. * **Role-based teams**: Form teams based on roles (developers, operators, administrators) to standardize permissions across your organization. * **Cross-functional teams**: Create teams that span traditional organizational boundaries to support DevOps and platform engineering approaches. --- # Create users in the platform Proper user configuration is important for effective access control and authentication on the platform. Each user account represents an individual who interacts with the platform. ## Create a user account[​](#create-a-user-account "Direct link to Create a user account") To create a new user in the platform: 1. Go to Access & Secrets > Users & Roles. 2. Click the Add User button. 3. In the configuration sheet that opens, give your new user a name by replacing the 'my-user' placeholder name, or by updating the manifest YAML 'metadata.name' field. 4. **\[Optional]** Update the user with any additional desired configuration options. 1. The Details tab contains user settings. The username is used to login and optionally you can also set the default password in this tab. The email address needs to be unique across users and can also be used to login instead of using the username. The subject is used to map a user to a SSO provider. 2. The Team Memberships tab contains settings that govern which teams the user is part of. The Teams selector allows you to make user part of the selected teams. 3. Advanced Options allow for configuring cluster roles that apply to this team, as well as Image Pull Secrets, that allow users in this team to automatically receive the provided image pull secrets upon logging into vCluster Platform (using the vCluster CLI). 4. Access allows you to define which users and teams have access to modify the team you are currently creating. 5. Click the Save Changes button. ## User configuration options[​](#user-configuration-options "Direct link to User configuration options") When creating a user, you can configure several important settings organized in tabs: ### User details[​](#user-details "Direct link to User details") The **Details** tab contains essential user identification information: * **Username**: The primary identifier used for login authentication. Choose something memorable but specific to the user. * **Password**: Optionally set an initial password. For enhanced security, consider leaving this blank and using the password reset workflow instead. * **Subject**: A unique identifier used to map this user to an external SSO provider. This field enables identity federation with authentication systems. ### Team memberships[​](#team-memberships "Direct link to Team memberships") The **Team Memberships** tab manages which teams this user belongs to: * Use the **Teams** selector to add the user to one or more teams * Team membership automatically grants the user all permissions associated with those teams * Changes to team membership take effect immediately across all resources ### Advanced options[​](#advanced-options "Direct link to Advanced options") The **Advanced Options** section controls low-level permissions and access: * **Cluster Roles**: Define specific Kubernetes cluster roles that apply directly to this user * **Image Pull Secrets**: Configure registry credentials that the user automatically receives when logging into vCluster Platform using the CLI The cluster roles defined here are combined with any roles inherited from team memberships, giving you fine-grained permission control. ### Access controls[​](#access-controls "Direct link to Access controls") The **Access** section determines who can manage this user account: * Specify which users and teams have permission to modify this user's settings * This meta-permission layer is crucial for delegated administration in larger organizations ## User management best practices[​](#user-management-best-practices "Direct link to User management best practices") For effective management of users in vCluster: * **Use descriptive usernames**: Choose usernames that clearly identify the individual user * **Leverage SSO where possible**: For organizations with existing identity providers, configure SSO integration rather than managing separate credentials * **Prefer team-based permissions**: Assign permissions to teams rather than individual users when possible * **Regularly audit user accounts**: Periodically review user accounts to ensure they remain necessary and properly configured ## Integration[​](#integration "Direct link to Integration") User accounts in the platform map to Kubernetes subjects in the underlying RBAC system: * Each user is represented as a subject in Kubernetes role bindings * Permissions granted to the user apply across all clusters managed by the platform * Authentication tokens and credentials are securely managed by the platform --- # Impersonate To test configurations and permissions, vCluster Platform provides a feature to impersonate users within the vCluster Platform UI and see everything through the eyes of a specific users. This is very useful if you want to check if a user has permissions to access an object. ## Enable impersonation[​](#enable-impersonation "Direct link to Enable impersonation") To enable impersonation, make sure the user has appropriate permissions, only vCluster Platform management admins and users that have the management role `Impersonator` assigned, can impersonate other users. 1. Go to Access & Secrets > Users & Roles. 2. In the user row you want to impersonate, select Impersonate 3. To stop impersonation, either press Logout or click the Stop Impersonation button at the top. ## Example cluster access and impersonation[​](#example-cluster-access-and-impersonation "Direct link to Example cluster access and impersonation") The following is a very basic example of using impersonation to validate a users access. This is a somewhat contrived example for demonstration purposes. For your production deployments make sure you are taking advantage of Projects when considering your RBAC strategy. ### 1. Create test user[​](#1-create-test-user "Direct link to 1. Create test user") vCluster Platform lets you connect a variety of SSO providers for authentication but for the sake of simplicity, manually create a user to learn more about vCluster Platform's cluster access features: 1. Go to Access & Secrets > Users & Roles. 2. Click the Add User button. 3. In the configuration sheet that opens, give your new user a name of Anna by replacing the 'my-user' placeholder name, or by updating the manifest YAML 'metadata. name' field. 4. Click the Save Changes button. 5. Close the popup using the button Copy & Close 100% Kubernetes Native Remember: Everything you do in vCluster Platform UI, including creating a user, is effectively a kubectl command under the hood. So, everything you do in this guide creates or changes objects in your cluster and you could also manage these actions using kubectl or any kind of GitOps tool. ### 2. Impersonate user[​](#2-impersonate-user "Direct link to 2. Impersonate user") vCluster Platform allows admins with appropriate RBAC permissions to impersonate users. This demonstrates how vCluster Platform UI appears for the newly created user: 1. Go to Access & Secrets > Users & Roles. 2. Find the user `Anna` in the list of users and click the button to Impersonate the user. 3. In the popup, click the Impersonate button to confirm that you want to start impersonation. 4. After impersonation has started, go to Infrastructure > Control Plane Clusters. 5. Verify that Anna has no access to any clusters (this user should not see any clusters listed in the Clusters display pane). You can also use the vCluster CLI as the impersonated user, to do this, simply run the following command while the impersonation is active. ``` vcluster platform login localhost:9898 --insecure # or use your loft.domain.tld instead of localhost, and ideally with a valid SSL cert and without the --insecure flag ``` You can verify the login and print your user information by running: ``` vcluster platform login ``` ### 3. Configure cluster access[​](#3-configure-cluster-access "Direct link to 3. Configure cluster access") Give the test user Anna access to one of the clusters connected to this vCluster Platform instance: 1. From the project drop-down menu (top left corner), select the project you'd like to create the tenant cluster in. 2. Click on Tenant Clusters. 3. Click the Create Tenant Cluster button. 4. The system redirects you to the configure virtual cluster page. Select your template: * If your project has "require template" enabled, choose from a list of available templates * If not, click Use Template to select your template 5. Follow the steps in the UI to create the virtual cluster. 6. Retrieve a kube-context for a virtual cluster using the CLI: ``` vcluster connect [vcluster-name] --project [project-name] --driver platform ``` Single Sign-On + Cluster Access You can connect a variety of SSO providers to vCluster Platform. To automatically give users access to clusters based on their SSO user groups, you can switch to the Team Members tab to grant cluster access for each member of a team (for example, a group in Active Directory, Okta, or SAML), see the [SSO Group Sync](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md#sso-group-sync) section for more details. ### 4. Verify cluster access[​](#4-verify-cluster-access "Direct link to 4. Verify cluster access") After configuring the cluster access for test user Anna, verify that she can access the cluster: 1. Go to Access & Secrets > Users & Roles. 2. Find the user Anna in the list of users and click the button to Impersonate the user. 3. In the popup, click the Impersonate button to confirm that you want to start impersonation. 4. After impersonation has started, go to Infrastructure > Control Plane Clusters. 5. Verify that Anna now has access to the clusters specified in the previous step. Next Steps With access to a cluster, users can typically: * **[Create Tenant Clusters](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-no-template.md)** vCluster Platform allows you to: * **[Configure sleep mode for tenant clusters](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md)** --- # Reset a user's password As an administrator, you can reset another user's password from the Users management screen. This generates a one-time reset link rather than setting a new password directly, so the user's existing sessions stay active until the link is used. See the [Platform CLI hub](https://vcluster.com/docs/platform/reference/platform-cli.md) for UI-to-CLI equivalents across common Platform tasks, including the CLI password reset command. ## Generate a reset link[​](#generate-a-reset-link "Direct link to Generate a reset link") 1. Go to Access & Secrets > Users & Roles. 2. Click the user whose password you want to reset. 3. Click Reset Password. 4. Click Reset Password again to confirm in the dialog that appears. 5. In the Reset Password Link popup, click Copy & Close to copy the link. 6. Send the copied link to the user. The link is only shown once. If you close the popup without copying it, repeat this flow to generate a new one. You can also start this flow from the row actions menu next to a user in the Users list. ## Effect on the user's sessions and access keys[​](#effect-on-the-users-sessions-and-access-keys "Direct link to Effect on the user's sessions and access keys") Clicking Reset Password only generates the link. It doesn't change the user's password, and it doesn't affect their existing sessions or access keys. The password only changes when the link is opened and a new password is submitted, whether that's by the user themselves or by you on their behalf. At that point: * All of that user's existing [access keys](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) are deleted, including browser sessions on other devices, CLI sessions, and any API tokens they generated. * If the user is logged in elsewhere at that moment, that session ends immediately, and they need to log in again with the new password. ## Change the password immediately[​](#change-the-password-immediately "Direct link to Change the password immediately") A reset link only takes effect once it's used. To force an immediate password change and session invalidation without waiting on the user, reset the password directly from the CLI instead: ``` vcluster platform reset password --user= ``` This works for any user, including the admin account, and deletes that user's existing access keys right away, the same as the reset link does once it's redeemed. ## Related[​](#related "Direct link to Related") * [Access keys](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) * [Change your password](https://vcluster.com/docs/platform/use-platform/change-your-password.md), for users resetting their own password --- # Suspend Users can be locked, to temporarily suspend their access to vCluster Platform. By doing so, the user won't be able to login or access any spaces / tenant clusters anymore. Access keys generated for the user won't work anymore as long as the user is locked. This is a useful feature to temporarily lock an user if you want to preserve the users environments without deleting the user. 1. Go to Access & Secrets > Users & Roles. 2. In the user row you want to lock, click the actions menu and select Lock User. 3. To unlock the user, click the actions menu and select Unlock User. --- # Overview The platform uses [Kubernetes role-based access control (RBAC)](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) to manage access for platform resources and objects, including platform custom resources and tenant clusters. RBAC manages user and team permissions by assigning them roles that specify what actions they can perform. These permissions are applied using *ClusterRoleBindings*, which connect a specific role to a user or team, which defines their access across platform resources and tenant clusters. Depending on the scope of access, these bindings are created in one of three types of clusters: | Cluster Type | Description | | --------------------- | ----------------------------------------------------------------- | | **Local cluster** | The cluster where the platform is installed. | | **Connected cluster** | A Control Plane Cluster that hosts tenant cluster control planes. | | **Tenant cluster** | A certified Kubernetes distribution managed by Platform. | To learn more about the platform’s resource types, see the [API resources](https://www.vcluster.com/docs/platform/api/use-api). You can also review the glossary of terms that relate to RBAC. RBAC terms to know | Component | Description | Relationship | | ------------------- | ---------------------------------------------------- | -------------------------------------------------------------------------------------- | | User | Individual identity that can access the system | Can be assigned roles directly or through team membership. | | Team | Logical grouping of users with shared access rights | Simplifies permission management by allowing role assignment to multiple users. | | Group | SSO identity provider attribute | Maps external identity groups to platform Teams for automatic assignment. | | ClusterRoleTemplate | Defines reusable permission sets | Extends Kubernetes RBAC with platform resources | | Project | Primary boundary for access control | Contains resources and defines which users/teams can access clusters and templates | | Project Role | Special role with loft.sh/project-role: "true" label | Available in UI for assignment within projects | | Permission Level | Degree of access granted | Ranges from view-only to full administrative control | | Security Layer | Components of defense-in-depth approach | Includes identity management, RBAC policies, namespace isolation, and network policies | | Tenant Cluster | Isolated Kubernetes environment | Provides full API compatibility with dedicated control plane | | Namespace | Enhanced namespace with management capabilities | Offers more granular control than standard Kubernetes namespaces | ## RBAC architecture[​](#rbac-architecture "Direct link to RBAC architecture") The platform architecture supports multi-level RBAC enforcement by distributing responsibility across different clusters. Each type of cluster enforces a specific scope of permissions based on the location and role of the resource. The platform manages permissions through `ClusterRoleBindings`. Users and teams are assigned roles that define their access levels. These roles connect to actual permissions through `ClusterRoleBindings`. --- # Cluster Access Cluster access allows you to give certain Users and Teams direct permission to a cluster without assigning them to a project. This is usually only useful for admin users or special privileged teams, while in all other cases you should prefer projects.
Cluster access lets you define access to certain clusters for your users or teams along with the cluster roles that would be synced for them in the connected cluster. You can create your own Cluster Access by following the below steps. ### Create a cluster access[​](#create-a-cluster-access "Direct link to Create a cluster access") 1. Go to Infrastructure > Control Plane Clusters. 2. Click the Cluster Access tab and then click Create Cluster Access. 3. In the configuration sheet that opens, give the access a name by replacing the auto-generated placeholder name, or by updating the manifest YAML 'metadata.name' field. 4. In the Options tab, choose the user or team to grant access from the Cluster Access selector, then choose the cluster role from the Cluster Roles selector. 5. In the Clusters tab, choose the cluster to which the user or team should be granted access from the Select Clusters selector. 6. In the Access tab, select the user or team that can manage this cluster access object, then choose their allowed permissions. 7. Click Save Changes to create the cluster access. --- # Manage Access for a Namespace Namespace access can be managed through the 'Permissions' section inside the namespace configuration sheet. There are a couple of special cases: 1. Global Admins & Project Admins have access and can change **all** namespaces within a project. 2. Tenant Cluster owners always have access and can change their namespaces. 3. Every user or team within the management cluster that has the RBAC permission on the resource "spaceinstances" in api group "management.loft.sh" for the verb "use" can access the namespace. ## How does Access within a namespace work?[​](#how-does-access-within-a-namespace-work "Direct link to How does Access within a namespace work?") Every user or team that has access to a namespace gets automatically the default cluster role assigned within the namespace. By default, this is `loft-cluster-space-admin`. The default cluster role can be either changed in the namespace template or on the namespace object itself. Besides the default rule you can define extra rules on the namespace or template that map a user or team to another cluster role. As soon as one rule matches a user or team, the default cluster role is not assigned. If multiple rules match a user, all the cluster roles defined in the rules are assigned. ## Grant Access to a Namespace[​](#grant-access-to-a-namespace "Direct link to Grant Access to a Namespace") * UI * CLI 1. Select the project from the project selector at the top left, then click Namespaces. 2. Click Edit on the namespace you want to share. 3. In the configuration sheet that opens, click the Permissions tab. 4. Select the user or team you want to grant permissions in the 'User or Team' select. If you don't see the user or team you want to grant access in there, make sure they have project access. 5. Specify the cluster-role you want to assign the user or team within the namespace. 6. Click the Save Changes button. To give someone access to a namespace using vCluster CLI, run: ``` vcluster platform share namespace [optional:name] --user other-user --project my-project ``` --- # Project Membership Members are users or teams that are granted access to a project. Each member is assigned a role that define their permissions. These roles will be granted to each of the [allowed clusters](https://vcluster.com/docs/platform/administer/projects/allowed/clusters.md) of the project. Only users and teams that are members of the project have access to the project's allowed clusters. If you don't have a project, you can create one by following instructions [here](https://vcluster.com/docs/platform/administer/secrets/project/create.md). ## Project Roles[​](#project-roles "Direct link to Project Roles") There are 3 default roles available for every project. | Name | Create New Spaces / Tenant Clusters | Access Spaces / Tenant Clusters | Create Without Template | Create With Template | Access to Project Secrets | | -------------- | ----------------------------------- | --------------------------------------- | ----------------------- | -------------------- | -------------------------------------------------- | | Project admin | Yes | All Spaces and Tenant Clusters | Yes | Yes | Yes | | Project User | Yes | Only owned and with explicit permission | No | Yes | Yes | | Project Viewer | No | Only with explicit permission | No | No | Only in Spaces and Tenant Clusters with permission | Global admins can change these existing roles as well as add new project roles in the Users section of the platform. 1. Go to Access & Secrets > Users & Roles. 2. In the Users page, click the Global Roles at the top. Then click the Create Role. 3. Fill the display name of the role and other information if necessary. warning The created role will be a global role by default. To create the role as a project role, add the `loft.sh/project-role: 'true'` in the labels of the role resource configuration on the right side of the page. Add the project role configuration ``` apiVersion: management.loft.sh/v1 kind: ClusterRoleTemplate metadata: name: '' labels: loft.sh/project-role: 'true' spec: management: true ``` 4. Click the Create Role button at the right bottom of the page. 5. If you created a project role, click Project Settings and find the display name of the created project role in the role selector for each member under the Members and Roles section. ## Users[​](#users "Direct link to Users") Any user can be made members of a project. Follow the instructions below to add and assign roles to users, or remove them from the project. ### Add users[​](#add-users "Direct link to Add users") 1. Select the project you'd like to configure using the project selector dropdown. Click Settings. 2. Click Members. 3. Click the Users tab. 4. Click the Users input and select the user to add. The table below will update to add the selected user. If you wish to change this manually, you may edit the YAML directly. 5. Select the role to assign the user. 6. Once all project options have been specified, click the Save Changes. All Users Instead of adding every user, a special selection named `All Users` can be used to give all platform users access to the project. Use this option with caution. ### Remove users[​](#remove-users "Direct link to Remove users") 1. Select the project you'd like to configure using the project selector dropdown. Click Settings. 2. Click Members. 3. Click the Users tab. 4. Click the trash can icon next to the user that you'd like to remove. If you wish to change this manually, you may edit the YAML directly. 5. Once all project options have been specified, click the Save Changes. ## Teams[​](#teams "Direct link to Teams") Instead of having to assign individual users, teams can be used to assign multiple users to a proejct. ### Add teams[​](#add-teams "Direct link to Add teams") 1. Select the project you'd like to configure using the project selector dropdown. Click Settings. 2. Click Members. 3. Click the Teams tab. 4. Click the Teams input and select the team to add. The table below will update to include the selected team. If you wish to change this manually, you may edit the YAML directly. 5. Select the role to assign the team. 6. Once all project options have been specified, click the Save Changes. ### Remove teams[​](#remove-teams "Direct link to Remove teams") 1. Select the project you'd like to configure using the project selector dropdown. Click Settings. 2. Click Members. 3. Click the Teams tab 4. Click the trash can icon next to the team that you'd like to remove. If you wish to change this manually, you may edit the YAML directly. 5. Once all project options have been specified, click the Save Changes. --- # Stack Permissions [Stack](https://vcluster.com/docs/platform/understand/what-are-stacks.md) permissions use Kubernetes role-based access control (RBAC) in the `management.loft.sh` API group. `StackTemplate` resources are cluster-scoped. `StackInstance` resources and their outputs subresource are project-scoped. Unlike tenant clusters or namespaces, Stacks don't have a per-object **Permissions** tab in the UI. Grant access with a ClusterRoleTemplate instead, as shown below. ## Permission map[​](#permission-map "Direct link to Permission map") Grant only the verbs required by each role: | Task | Resource | Verbs | | ------------------------ | ------------------------ | ------------------------------------------------------------- | | Browse the Stack catalog | `stacktemplates` | `get`, `list` | | Create Stack templates | `stacktemplates` | `create` | | Manage Stack templates | `stacktemplates` | `get`, `list`, `watch`, `create`, `update`, `patch`, `delete` | | Install a Stack | `stackinstances` | `create` | | View Stack status | `stackinstances` | `get`, `list`, `watch` | | Change or remove a Stack | `stackinstances` | `update`, `patch`, `delete` | | Read published outputs | `stackinstances/outputs` | `get` | Reading a StackInstance does not grant access to its published values. Grant `get` on `stackinstances/outputs` separately because published values can include data captured from Secrets. The outputs API marks values captured from Secrets as sensitive. A user who creates a StackInstance must also be authorized for its destination tenant cluster or control plane cluster. Task type licensing and destination access checks are independent of the resource rules in this page. ## Default roles in Platform 4.12[​](#default-roles-in-platform-412 "Direct link to Default roles in Platform 4.12") The built-in **Template Admin**, **Template Creator**, and **Template Viewer** global roles include `stacktemplates` alongside the other template resources. Template Admin grants all verbs, and Template Viewer grants `get`. Template Creator's rule grants only `create`, but a StackTemplate defaults its owner to whoever creates it, and owner access separately grants that creator `get`, `update`, `patch`, and `delete` on templates they created. The built-in **Project Admin** role grants all verbs on both `stackinstances` and `stackinstances/outputs`. The built-in **Project User** and **Project Viewer** roles do not grant StackInstance permissions in Platform 4.12. Add the required rules to a custom project role when those members need to install or operate Stacks. ## Custom project role example[​](#custom-project-role-example "Direct link to Custom project role example") The following rule lets a project member create and monitor StackInstances and read their published outputs. The rule itself doesn't grant update or delete access. StackInstance creation defaults the owner to the caller, though, and owner access separately grants that member `get`, `update`, `patch`, `delete`, and the `outputs` subresource on instances they created. It doesn't extend to instances someone else created. ``` apiVersion: management.loft.sh/v1 kind: ClusterRoleTemplate metadata: name: stack-user labels: loft.sh/project-role: "true" spec: displayName: Stack User management: true clusterRoleTemplate: rules: - apiGroups: - management.loft.sh resources: - stackinstances verbs: - create - get - list - watch - apiGroups: - management.loft.sh resources: - stackinstances/outputs verbs: - get ``` Assign the custom role to project members as described in [Project membership](https://vcluster.com/docs/platform/administer/users-permissions/permissions/project.md#project-roles). For API request paths and resource fields, see the [`StackTemplate`](https://vcluster.com/docs/platform/api/resources/stacktemplate.md), [`StackInstance`](https://vcluster.com/docs/platform/api/resources/stackinstance.md), and [outputs](https://vcluster.com/docs/platform/api/resources/stackinstanceoutputs.md) references. --- # Management access to the tenant cluster Management access controls who can view, edit, or configure the tenant cluster object in the platform. It is separate from in-cluster access, which governs permissions inside the Kubernetes environment of the tenant cluster. ## Default management access[​](#default-management-access "Direct link to Default management access") By default, the following users and roles have management access to a tenant cluster: * **Global administrators**—Access all tenant clusters on the platform. * **Project administrators**—Access all tenant clusters within their assigned projects. * **Tenant cluster owners**—Automatically receive access to the specific tenant cluster they create or own. * **Users with physical cluster permissions**—Any user or team with RBAC permissions on the underlying (physical) cluster and the `use` verb on the `virtualclusterinstances` resource in the `management.loft.sh` API group can access tenant clusters running on that cluster. ## Grant management access using the UI[​](#grant-management-access-using-the-ui "Direct link to Grant management access using the UI") To extend access to additional users or teams, use the **Permissions** section for each tenant cluster in the platform UI: 1. Open the **Project** dropdown in the top-left corner and select the project that contains the tenant cluster. 2. Click **Tenant Clusters** in the sidebar to view the list. 3. Click **Edit** on the target tenant cluster. 4. Open the **Permissions** tab. 5. Use the **Add permission to** field to select the user or team. 6. If the user or team does not appear, confirm that they have access to the project. 7. Choose a **ClusterRole** to assign (e.g., `cluster-admin`, `edit`, or `view`). This role determines the user's Kubernetes permissions inside the tenant cluster. 8. Click **Save changes**. The platform grants the selected user or team management access to the tenant cluster object. ## vCluster roles[​](#vcluster-roles "Direct link to vCluster roles") vCluster roles define what users and teams can do inside the tenant cluster. Kubernetes RBAC governs this access. ### Default cluster role assignment[​](#default-cluster-role-assignment "Direct link to Default cluster role assignment") By default, the platform assigns the `cluster-admin` Kubernetes ClusterRole to users with tenant cluster access. This role grants full access in all namespaces. ### Change the default cluster role[​](#change-the-default-cluster-role "Direct link to Change the default cluster role") Administrators can override the default role in the tenant cluster template or in the tenant cluster configuration. 1. Open the tenant cluster or template configuration. 2. Locate the **Default Cluster Role** field. 3. Enter a more limited role such as `edit` or `view`. 4. Save the updated configuration. ### Define custom role mapping rules[​](#define-custom-role-mapping-rules "Direct link to Define custom role mapping rules") Use mapping rules to assign specific users or teams to specific cluster roles. * A user or team that matches at least one rule does **not** receive the default role. * If multiple rules match, the system assigns all specified roles. * If no rule matches, the system assigns the default role. 1. Open the tenant cluster or template YAML configuration. 2. Locate or create the `access.rules` section. 3. List `subjects` for each user or team. 4. Define the ClusterRole for each rule. 5. Save and apply the configuration. #### Example[​](#example "Direct link to Example") ``` apiVersion: management.loft.sh/v1 kind: VirtualCluster metadata: name: example-vcluster spec: defaultClusterRole: cluster-admin access: rules: - subjects: - kind: User name: Person 1 clusterRole: edit - subjects: - kind: Team name: DevTeam clusterRole: view ``` In this example: * Person 1 receives the `edit` role. * DevTeam receives the `view` role. * Person 1, if part of DevTeam, receives both roles. * All other users default to `cluster-admin`. Custom mapping rules allow more precise and secure access control inside the tenant cluster. ### Custom ClusterRoles[​](#custom-clusterroles "Direct link to Custom ClusterRoles") The platform UI's **Permissions** section lists only the four built-in roles: `cluster-admin`, `admin`, `edit`, and `view`. To use any other role, inject it into the tenant cluster using the `objects` field and reference it by name in `access.rules`. Two approaches are available depending on what you need: * **Define a new ClusterRole** — specify exactly the rules you want. Use this when you need to remove specific permissions from a built-in role. * **Aggregate into a built-in role** — add rules to an existing role using Kubernetes label selectors. Use this when you need to extend a built-in role with additional access, without redefining its full rule set. note If the tenant cluster is deployed from a template, add `objects` to the **template's** `spec.template`. The template controls those fields and overrides any changes made directly on the tenant cluster. #### Define a new ClusterRole[​](#define-a-new-clusterrole "Direct link to Define a new ClusterRole") Kubernetes RBAC is additive. A ClusterRole only grants what its rules explicitly allow. To restrict specific verbs from a built-in role, define a new ClusterRole that includes only the permissions you want. One common scenario is a team that uses `edit` but must not be able to create PersistentVolumeClaims. For example, a shared PersistentVolume references a sensitive secret, and unrestricted PVC creation would expose that access. ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterTemplate metadata: name: restricted-storage spec: template: objects: | apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: edit-no-pvc-write rules: - apiGroups: [""] resources: ["pods", "services", "configmaps", "secrets", "endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "delete"] - apiGroups: ["apps"] resources: ["deployments", "statefulsets", "daemonsets", "replicasets"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] access: defaultClusterRole: cluster-admin rules: - teams: - storage-restricted-team clusterRole: edit-no-pvc-write ``` `create`, `update`, and `patch` are omitted from the `persistentvolumeclaims` entry. Those verbs return `Forbidden` for users assigned this role. To build from the full `edit` rule set, run `kubectl get clusterrole edit -o yaml`. Copy its `rules` section into `objects` and remove the verbs you want to block. #### Extend a built-in role using aggregation[​](#extend-a-built-in-role-using-aggregation "Direct link to Extend a built-in role using aggregation") Kubernetes ClusterRole aggregation lets you inject additional rules into a built-in role using label selectors. The built-in `admin`, `edit`, and `view` roles each automatically aggregate any ClusterRole carrying the corresponding label: | Label | Aggregates into | | ------------------------------------------------------ | --------------- | | `rbac.authorization.k8s.io/aggregate-to-admin: "true"` | `admin` | | `rbac.authorization.k8s.io/aggregate-to-edit: "true"` | `edit` | | `rbac.authorization.k8s.io/aggregate-to-view: "true"` | `view` | The following example adds namespace creation to the `admin` role. Any team assigned `admin` automatically gains the added permission. ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterInstance metadata: name: vcluster-rbac-example spec: template: objects: | apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: admin-create-namespace labels: rbac.authorization.k8s.io/aggregate-to-admin: "true" rules: - apiGroups: [""] resources: ["namespaces"] verbs: ["create"] access: defaultClusterRole: view rules: - teams: - api-framework clusterRole: admin ``` --- # Data volumes | | | Enterprise | | | | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [VM Management with KubeVirt](https://vcluster.com/docs/platform/administer/virtual-machines/overview.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). KubeVirt `DataVolume` resources define the disk content that Containerized Data Importer (CDI) imports, clones, or creates for virtual machines. In vCluster Platform, administrators use DataVolumes to prepare reusable VM disks, inspect import progress, and understand which VMs depend on a disk. Open the DataVolumes view from **Virtual Machines > Data Volumes**, or go directly to `/virtual-machines/data-volumes`. ## When to use data volumes[​](#when-to-use-data-volumes "Direct link to When to use data volumes") Use DataVolumes when you need to: * Import a VM disk from an HTTP, HTTPS, or registry source. * Clone disk content from an existing PVC. * Create a blank disk that a VM can format and use. * Track the import, clone, or population status of a VM disk across connected KubeVirt clusters. * Identify direct VM references and investigate other dependencies before changing or deleting a DataVolume. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before creating or using DataVolumes, make sure: * The target Control Plane Cluster is [connected to vCluster Platform](https://vcluster.com/docs/platform/administer/clusters/connect-cluster.md). * KubeVirt and CDI are installed on the target cluster, or the [KubeVirt node provider is configured to deploy them](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md#infrastructure-deployment). * The user has permission to list, create, update, and delete `datavolumes.cdi.kubevirt.io` resources in the target namespace. * Cross-namespace clones need additional RBAC. See [Cross-namespace clones](#cross-namespace-clones). * The StorageClass used by the DataVolume can provision PVCs in the target cluster. * Image URLs, registries, PVC sources, secrets, and certificates are reachable from CDI importer pods. ## Create a standalone data volume[​](#create-a-standalone-data-volume "Direct link to Create a standalone data volume") tip Root disk DataVolumes for KubeVirt-backed Machines are usually generated automatically from [image properties](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md#image-configuration) or an [OS image](https://vcluster.com/docs/platform/use-platform/machines/os-images.md). Use the steps below instead for secondary or manually managed disks. The platform UI creates an administrator-managed DataVolume that a VM can reference directly. 1. Open Virtual Machines > Data Volumes. 2. Click Create Data Volume. 3. Choose the target KubeVirt cluster and namespace. 4. Enter a name and storage size. Select one source type: * HTTP: import from an HTTP or HTTPS image URL. * Registry: import from a container registry URL such as `docker://quay.io/containerdisks/ubuntu:22.04`. * PVC: clone from an existing PVC. * Blank: create an empty disk. 5. Click Save and watch its lifecycle in the table. See the [CDI documentation](https://kubevirt.io/user-guide/storage/containerized_data_importer/) for the full set of source types and import semantics. The DataVolumes table shows the cluster, namespace, source type, requested storage, platform lifecycle status, and **Used by machine**. The **Used by machine** column lists VMs that reference the volume directly. Before deleting a DataVolume An empty **Used by machine** value doesn't mean a DataVolume is unused. A DataVolume can back persistent storage without currently being attached to a VM. Some DataVolumes also back tenant PVCs through the vCluster KubeVirt CSI driver instead of a VM template disk, a case this column can never detect. See [Operational ownership](#operational-ownership) for the full breakdown before you delete anything this column doesn't explain. ### Cross-namespace clones[​](#cross-namespace-clones "Direct link to Cross-namespace clones") Cloning a PVC across namespaces needs RBAC beyond what's listed in [Prerequisites](#prerequisites). This applies whether the clone happens through the platform UI, `kubectl`, or the KubeVirt node provider's `kubevirt.vcluster.com/image-datasource` property. The identity performing the clone must be able to create `datavolumes/source` or pods in the source namespace. For DataVolumes the KubeVirt node provider synthesizes, that identity is the connected-cluster credentials the provider uses to reach the target cluster, not the identity of whoever created the Machine, NodeType, or NodeProvider. ## Understand the data volume lifecycle[​](#understand-the-data-volume-lifecycle "Direct link to Understand the data volume lifecycle") CDI reports the raw lifecycle in the DataVolume `.status.phase` field. vCluster Platform maps that to a simplified **Status** column that groups related CDI phases. Use the CDI phase when you troubleshoot with `kubectl`. Use the platform status when you scan the UI. | Platform status | Common CDI phases or signals | Meaning | | --------------- | ----------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `Importing` | `ImportScheduled`, `ImportInProgress` | CDI scheduled or started an image import. | | `Pending` | `Pending`, `PVCBound`, clone phases, upload phases, `WaitForFirstConsumer`, `PendingPopulation` | CDI or the storage provider is preparing the PVC or waiting for scheduling. | | `Ready` | `Succeeded` | CDI finished preparing the volume. | | `Failed` | `Failed` or a `Running` condition with reason `Error` | CDI or the storage provider reported an error. | | `Paused` | `Paused` | CDI paused the operation. | | `Terminating` | `metadata.deletionTimestamp` is set | Kubernetes is deleting the DataVolume. | | `Unknown` | Missing, empty, or unrecognized phase | The platform can't classify the lifecycle state. | For example, the UI can show `Pending` for a DataVolume whose CDI phase is `WaitForFirstConsumer`. That usually means the PVC waits for a VM pod to schedule before the storage provider binds it. ## Create a reusable data source[​](#create-a-reusable-data-source "Direct link to Create a reusable data source") Three resources work together to prepare and consume VM storage: * `DataSource`: a reusable reference to prepared disk content. * `DataVolume`: the request and lifecycle for creating or populating that content into a PVC. * PersistentVolumeClaim: the storage a VM ultimately mounts. To let multiple VMs reuse the same disk content without importing it more than once, wrap a DataVolume's PVC in a CDI `DataSource`. Import or clone the disk once into a DataVolume through the platform UI, then create a `DataSource` that points to the resulting PVC: ``` apiVersion: cdi.kubevirt.io/v1beta1 kind: DataSource metadata: name: ubuntu-22-04 namespace: golden-images spec: source: pvc: name: ubuntu-root-disk namespace: golden-images ``` Reference the DataSource from a NodeProvider, NodeType, NodeClaim, or Machine with `kubevirt.vcluster.com/image-datasource`, the same property described in [Import a root disk through a Machine](#import-a-root-disk-through-a-machine). vCluster Platform doesn't create or manage DataSource resources directly. Create and update them with `kubectl`, and treat them as long-lived golden images. Deleting the underlying DataVolume or PVC prevents new target DataVolumes from being populated and can interrupt clones that are still in progress. VMs whose target PVCs are already populated continue to use those independent volumes. ## Use data volumes with node providers[​](#use-data-volumes-with-node-providers "Direct link to Use data volumes with node providers") The [KubeVirt node provider](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md) can also synthesize a root-disk `DataVolumeTemplate` automatically when a Machine, NodeType, NodeProvider, or [OS image](https://vcluster.com/docs/platform/use-platform/machines/os-images.md) sets `kubevirt.vcluster.com/image-url` or `kubevirt.vcluster.com/image-datasource`. That generated DataVolume appears in the DataVolumes table alongside any administrator-created volumes. ### Import a root disk through a Machine[​](#import-a-root-disk-through-a-machine "Direct link to Import a root disk through a Machine") For VM-backed Machines and auto nodes, you usually do not need to create the root DataVolume manually. Instead, set image properties on the KubeVirt provider, node type, NodeClaim, Machine, or referenced OSImage. ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: kubevirt-provider spec: properties: kubevirt.vcluster.com/image-url: docker://quay.io/containerdisks/ubuntu:22.04 kubevirt.vcluster.com/root-disk-size: 20Gi kubeVirt: clusterRef: cluster: kubevirt-host namespace: vcluster-platform virtualMachineTemplate: spec: template: spec: domain: resources: requests: cpu: "2" memory: 4Gi nodeTypes: - name: ubuntu maxCapacity: 10 ``` The provider creates a root-disk `DataVolumeTemplate`, a volume, and a disk for the VM. HTTP and HTTPS image URLs become CDI `http` sources. `docker://` image URLs become CDI `registry` sources. To use a prepared CDI `DataSource`, set `kubevirt.vcluster.com/image-datasource` instead: ``` spec: properties: kubevirt.vcluster.com/image-datasource: golden-images/ubuntu-22-04 kubevirt.vcluster.com/root-disk-size: 20Gi ``` Use `` to reference a DataSource in the node provider namespace from `spec.kubeVirt.clusterRef.namespace`, or `/` to reference one in another namespace. Configure either `kubevirt.vcluster.com/image-url` or `kubevirt.vcluster.com/image-datasource`. If both are set, the provider uses the DataSource and ignores the URL and checksum. See [Cross-namespace clones](#cross-namespace-clones) for the RBAC a `/` reference requires. ### Attach data volumes to VM templates[​](#attach-data-volumes-to-vm-templates "Direct link to Attach data volumes to VM templates") You can attach additional DataVolumes by adding a KubeVirt volume and disk to the VM template. Use `dataVolumeTemplates` for volumes that should be created with the VM, or reference an existing DataVolume by name. Use `dataVolumeTemplates` for inline volumes: ``` spec: kubeVirt: virtualMachineTemplate: spec: dataVolumeTemplates: - metadata: name: tools-disk spec: source: blank: {} pvc: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi template: spec: domain: devices: disks: - name: tools-disk disk: bus: virtio volumes: - name: tools-disk dataVolume: name: tools-disk ``` When the KubeVirt node provider creates a VM from a template, it prefixes user-provided `dataVolumeTemplates` so DataVolume names stay unique for each node. To attach a standalone DataVolume that already exists in the VM namespace, reference it from the VM template's `volumes` list: ``` spec: kubeVirt: virtualMachineTemplate: spec: template: spec: domain: devices: disks: - name: my-data-disk disk: bus: virtio volumes: - name: my-data-disk dataVolume: name: my-existing-datavolume ``` The referenced DataVolume must exist in the namespace where KubeVirt creates the VM. This is the path to use for DataVolumes created through the **Data Volumes** UI. Every VM created from this template references the same DataVolume and underlying PVC. warning DataVolumes created through the UI use `ReadWriteOnce` by default, which restricts read-write mounts to one node at a time. Only share a DataVolume when its access mode, storage provider, and workload support the intended concurrent access. When each VM needs an independent disk cloned from the same image, use a [reusable DataSource](#create-a-reusable-data-source) instead. ## Operational ownership[​](#operational-ownership "Direct link to Operational ownership") DataVolumes are namespaced resources on connected KubeVirt clusters, not vCluster Platform resources. vCluster Platform gives administrators a UI for the connected KubeVirt clusters, but CDI owns the import and PVC population process. KubeVirt owns DataVolumes created from a VM's `dataVolumeTemplates`, and those resources usually follow the VM lifecycle. Use this ownership model: | Owner | Responsibility | | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Platform administrator | Create reusable DataVolumes, choose storage classes, configure image sources, and control RBAC. | | vCluster Platform | Lists DataVolumes, creates administrator-requested DataVolumes, shows platform lifecycle status and directly referencing VMs, and generates root-disk templates from KubeVirt image properties. | | vCluster KubeVirt CSI driver | Creates host-cluster DataVolumes to back tenant PVCs and removes them according to the tenant PV reclaim lifecycle. | | CDI | Imports, clones, uploads, or populates PVC content and reports DataVolume phase and conditions. | | KubeVirt | Creates DataVolumes from VM templates and attaches the resulting PVCs to VMs. | | Storage provider | Provisions PVCs and enforces access modes, binding mode, expansion, and quota. | Some DataVolumes in this list back tenant-cluster PVCs rather than a VM template disk. The vCluster KubeVirt CSI driver creates one on the host cluster for each such PVC and hot-plugs the resulting PVC into the VM backing the selected tenant node. Because the VM references the PVC directly rather than the DataVolume, **Used by machine** never shows these VMs, even while the PVC is attached. Treat these the same as any other DataVolume you didn't create yourself. Before deleting a DataVolume, check the **Used by machine** column and inspect its metadata: ``` kubectl get datavolume -n vcluster-platform my-data-volume -o yaml ``` The column only detects direct VM references, so an empty value doesn't mean that the volume is unused. A DataVolume with the `csi.kubevirt.io/vcluster-claim` annotation backs the named PVC inside a vCluster. The `vcluster.loft.sh/managed-by` label also indicates that a vCluster manages the resource. Do not delete these DataVolumes directly from the platform UI or host cluster. Manage their lifecycle through the corresponding tenant PVC and PV reclaim policy instead. Deleting any DataVolume or its PVC while a VM or tenant PVC depends on it can prevent workloads from starting, detach required disk content, or cause data loss. ## Troubleshoot data volumes[​](#troubleshoot-data-volumes "Direct link to Troubleshoot data volumes") ### Import is stuck or slow[​](#import-is-stuck-or-slow "Direct link to Import is stuck or slow") Check the raw CDI phase and DataVolume conditions in the namespace where the DataVolume was created: ``` kubectl get datavolume -n vcluster-platform kubectl describe datavolume -n vcluster-platform ubuntu-root-disk ``` If the phase stays `ImportScheduled`, `ImportInProgress`, `WaitForFirstConsumer`, or `PendingPopulation`, inspect CDI importer pods and PVC events. ``` kubectl get pods -n vcluster-platform -l cdi.kubevirt.io kubectl describe pvc -n vcluster-platform ubuntu-root-disk ``` Common causes are an unreachable image URL, missing registry credentials, an invalid certificate ConfigMap, storage quota, an unavailable StorageClass, or a StorageClass with `WaitForFirstConsumer` binding that needs the VM pod to schedule before the PVC binds. ### Data source can't be resolved[​](#data-source-cant-be-resolved "Direct link to Data source can't be resolved") If a VM or Machine uses `kubevirt.vcluster.com/image-datasource`, verify the DataSource exists in the expected namespace: ``` kubectl get datasource -A kubectl describe datasource -n golden-images ubuntu-22-04 ``` Use `name` for a DataSource in the node provider namespace from `spec.kubeVirt.clusterRef.namespace`, and use `namespace/name` to select a different namespace explicitly. Also confirm the referenced DataSource points to a valid PVC or VolumeSnapshot. For a cross-namespace clone, verify that the connected-cluster credentials the KubeVirt node provider uses can create `datavolumes/source` or pods in the DataSource namespace. ### Virtual machine console opens but the guest doesn't boot[​](#virtual-machine-console-opens-but-the-guest-doesnt-boot "Direct link to Virtual machine console opens but the guest doesn't boot") Inspect the VM's disks and DataVolumes: ``` kubectl get vm -n vcluster-platform my-vm -o yaml kubectl get datavolume -n vcluster-platform ``` The root disk must be present in both `spec.template.spec.domain.devices.disks` and `spec.template.spec.volumes`. For provider-generated root disks, `kubevirt.vcluster.com/root-disk-size` is required whenever `kubevirt.vcluster.com/image-url` or `kubevirt.vcluster.com/image-datasource` is set. If the image boots but does not join a tenant cluster, make sure the guest image supports `cloudInitNoCloud`. ### Data volume fails[​](#data-volume-fails "Direct link to Data volume fails") Describe the DataVolume and review CDI importer logs: ``` kubectl describe datavolume -n vcluster-platform ubuntu-root-disk kubectl logs -n vcluster-platform -l cdi.kubevirt.io ``` Failures usually point to source access, authentication, checksum mismatch, unsupported image format, insufficient storage, or storage provider errors. After correcting the source or storage issue, recreate the DataVolume if CDI cannot retry the failed import. --- # Network attachment definitions | | | Enterprise | | | | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [VM Management with KubeVirt](https://vcluster.com/docs/platform/administer/virtual-machines/overview.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Multus `NetworkAttachmentDefinition` resources describe additional Kubernetes networks that KubeVirt VMs can attach to. Use them when a VM needs a bridge, VLAN, SR-IOV network, or another CNI-backed network in addition to, or instead of, the default pod network. Open the Network Attachment Definitions view from **Virtual Machines > Network Attachment Definitions**, or go directly to `/virtual-machines/network-attachment-definitions`. ## When to use network attachment definitions[​](#when-to-use-network-attachment-definitions "Direct link to When to use network attachment definitions") Use a network attachment definition when the default pod network isn't suitable for VM traffic. Common examples are reaching an external L2 network or underlay, or controlling IPAM and other networking details explicitly. Also use network attachment definitions to: * Attach KubeVirt VMs to an existing Multus network. * Make a Multus network the VM's default network. * Provide L2 access through a bridge such as `br0` or a Netris-created bridge. * Standardize network attachments across KubeVirt node types. * Inspect or create NAD resources across connected KubeVirt clusters from the platform UI. The KubeVirt node provider attaches VMs to a NAD when the effective properties include `kubevirt.vcluster.com/network-attachment-definition`. The property can be set on a NodeProvider, NodeType, NodeClaim, Machine, or [NodeEnvironment](https://vcluster.com/docs/platform/administer/node-providers/overview.md#node-environments). ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before creating or using a NAD, make sure: * The target Control Plane Cluster is [connected to vCluster Platform](https://vcluster.com/docs/platform/administer/clusters/connect-cluster.md). * [KubeVirt](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md#infrastructure-deployment), Multus, and the CNI plugins referenced by the NAD are installed on the target cluster. * Any network binding plugin selected through `kubevirt.vcluster.com/interface-template` is installed and registered in the KubeVirt custom resource. * The user has permission to list, create, update, and delete `network-attachment-definitions.k8s.cni.cncf.io` resources in the target namespace. * The referenced host bridge, device, VLAN, or network exists on the nodes where VM launcher pods can schedule. ## Understand the networking layers[​](#understand-the-networking-layers "Direct link to Understand the networking layers") KubeVirt VM networking spans three layers, and each is configured separately: * The NAD's CNI configuration defines the host-side network, for example a Linux bridge or SR-IOV device. * The KubeVirt interface binding (`kubevirt.vcluster.com/interface-binding` or `interface-template`) connects the VM to that network. * Cloud-init network data (`kubevirt.vcluster.com/network-data`, `network-interface`, `network-nameservers`) configures the interface inside the guest OS. Keep these layers separate when troubleshooting. A working NAD and binding only get traffic to the VM's virtual NIC. The guest still needs correct IP configuration to use it. ## Create a network attachment definition[​](#create-a-network-attachment-definition "Direct link to Create a network attachment definition") Use the platform UI to create an administrator-managed NAD in the namespace the VMs will use. After saving, reference the NAD name from a node type or NodeClaim as described in [Reference a NAD from a node type](#reference-a-nad-from-a-node-type). 1. Open Virtual Machines > Network Attachment Definitions. 2. Click Create Network Attachment Definition. 3. Choose the target KubeVirt cluster and namespace. 4. Configure the network. * Enter the NAD name. * Add the Container Network Interface (CNI) configuration as JSON. 5. Click Save. The UI validates that `spec.config` is valid JSON before saving. The following bridge example matches the placeholder used by the create form: ``` apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: bridge-network namespace: vcluster-platform spec: config: | { "cniVersion": "0.3.1", "name": "bridge-network", "type": "bridge", "bridge": "br0", "ipam": { "type": "host-local", "subnet": "10.10.0.0/24" } } ``` The exact CNI configuration depends on your network plugin and node setup. For example, a bridge NAD requires the named Linux bridge to exist on the relevant cluster nodes. ## Reference a NAD from a node type[​](#reference-a-nad-from-a-node-type "Direct link to Reference a NAD from a node type") Set [`kubevirt.vcluster.com/network-attachment-definition`](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md#kubevirt-vcluster-com-network-attachment-definition) to the NAD name. Use `` for a NAD in the VM namespace or `/` for a NAD in another namespace. ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: kubevirt-bridge-provider spec: properties: kubevirt.vcluster.com/image-url: docker://quay.io/containerdisks/ubuntu:22.04 kubevirt.vcluster.com/root-disk-size: 20Gi kubeVirt: clusterRef: cluster: kubevirt-host namespace: vcluster-platform virtualMachineTemplate: spec: template: spec: domain: resources: requests: cpu: "2" memory: 4Gi nodeTypes: - name: bridge-vm properties: kubevirt.vcluster.com/network-attachment-definition: bridge-network kubevirt.vcluster.com/interface-binding: bridge maxCapacity: 5 ``` When this property is set, the provider injects a KubeVirt network named `multus` and a matching interface named `multus`. If a network or interface with that name already exists in the VM template, the provider overwrites it. Only one NAD can be attached to a VM through this property. To attach multiple additional networks, define the extra KubeVirt networks and interfaces directly in `virtualMachineTemplate`. Use these optional properties to tune the injected interface: | Property | Description | | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `kubevirt.vcluster.com/interface-binding` | Sets an in-tree KubeVirt binding method. Supported values are `bridge`, `masquerade`, and `sriov`. Defaults to `bridge`. | | `kubevirt.vcluster.com/interface-template` | YAML fragment merged into the injected KubeVirt interface. Use `binding: {name: }` for a network binding plugin that is already [installed and registered with KubeVirt](https://kubevirt.io/user-guide/network/network_binding_plugins/). The interface name is always forced to `multus`. | | `kubevirt.vcluster.com/multus-default` | When set to `"true"`, makes the Multus network the default network and removes the pod network from the VM template. | | `kubevirt.vcluster.com/network-data` | Base64-encoded cloud-init network data. Use this when automatic network configuration is not enough. See [network-data templates](https://vcluster.com/docs/platform/use-platform/machines/machine-config-templates.md#network-data-templates). | | `kubevirt.vcluster.com/network-interface` | Guest interface name to configure, for example `enp1s0` or `eth1`. | | `kubevirt.vcluster.com/network-nameservers` | Comma-separated DNS nameservers for generated network data. | ## Use with Netris integration[​](#use-with-netris-integration "Direct link to Use with Netris integration") note This section only applies if the Netris integration is enabled on your cluster. Netris doesn't create the NAD itself. It provisions the underlying bridge network that the NAD references. When the Netris integration is enabled and `netris.vcluster.com/server-cluster` is set, vCluster Platform creates or updates a Multus NAD automatically. The generated name follows the Netris server cluster and node provider configuration, and the provider then attaches the VM through `kubevirt.vcluster.com/network-attachment-definition`. If you set `kubevirt.vcluster.com/network-attachment-definition` yourself, the Netris flow uses that existing NAD instead of creating another one. See the [KubeVirt node provider Netris example](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md#netris-integration) for the full provider configuration. ## Operational ownership[​](#operational-ownership "Direct link to Operational ownership") NetworkAttachmentDefinitions are namespaced resources on connected Kubernetes clusters, not vCluster Platform resources. vCluster Platform can create, list, edit, and delete them from the VM networking UI, but Multus and the referenced CNI plugin own the actual network attachment behavior. Use this ownership model: | Owner | Responsibility | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | Platform administrator | Creates NADs, validates CNI JSON, verifies host networking, and assigns RBAC. | | vCluster Platform | Lists and creates NADs, filters them to KubeVirt-enabled clusters, and injects NAD references into KubeVirt VMs through node provider properties. | | Multus | Reads the NAD and attaches the requested network to the VM launcher pod. | | CNI plugin | Implements the network described by `spec.config`. | | KubeVirt | Connects VM interfaces to the pod networks and enforces interface/network name matching. | Treat NAD changes as infrastructure changes. Changing `spec.config` can affect new VM starts immediately and may affect running workloads after restart or migration. ## Troubleshoot network attachment definitions[​](#troubleshoot-network-attachment-definitions "Direct link to Troubleshoot network attachment definitions") ### Network attachment definition doesn't appear in the UI[​](#network-attachment-definition-doesnt-appear-in-the-ui "Direct link to Network attachment definition doesn't appear in the UI") The Network Attachment Definitions view lists resources from connected clusters where KubeVirt is installed. If a NAD is missing, verify the cluster is connected, KubeVirt is detected, and the resource exists: ``` kubectl get network-attachment-definitions -A kubectl get kubevirt -A ``` Also check RBAC for the user and the platform agent on `network-attachment-definitions.k8s.cni.cncf.io`. ### Virtual machine fails to start after adding a network attachment definition[​](#virtual-machine-fails-to-start-after-adding-a-network-attachment-definition "Direct link to Virtual machine fails to start after adding a network attachment definition") Describe the VM and VMI: ``` kubectl describe vm -n vcluster-platform my-vm kubectl describe vmi -n vcluster-platform my-vm ``` Common causes are a missing NAD, invalid `spec.config` JSON, a CNI plugin that is not installed on the node, a missing host bridge or device, or an unsupported interface binding. If `kubevirt.vcluster.com/multus-default` is `"true"`, the provider removes the pod network because KubeVirt rejects a VM with both a default Multus network and the pod network. ### Virtual machine console works but networking doesn't[​](#virtual-machine-console-works-but-networking-doesnt "Direct link to Virtual machine console works but networking doesn't") Confirm that the Multus interface was injected into the VM template: ``` kubectl get vm -n vcluster-platform my-vm -o yaml ``` The VM should have a network named `multus` and an interface named `multus`. Then check that cloud-init config targets the guest interface name that the OS actually uses. The VM template, cloud-init network data, and guest OS interface names must agree on which interface receives the configured IP address. For generated network data, set `kubevirt.vcluster.com/network-interface` to the guest interface name and set `kubevirt.vcluster.com/network-nameservers`. ### Network attachment definition configuration is invalid[​](#network-attachment-definition-configuration-is-invalid "Direct link to Network attachment definition configuration is invalid") Validate the JSON in `spec.config` and confirm the CNI plugin accepts the fields: ``` kubectl get network-attachment-definition -n vcluster-platform bridge-network -o yaml ``` The platform UI catches malformed JSON, but it cannot verify that a referenced bridge, VLAN, SR-IOV device, IPAM range, or plugin-specific option exists on every node. Review Multus and CNI plugin logs on the target cluster if the JSON is valid but attachment still fails. --- # Overview | | | Enterprise | | | | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [VM Management with KubeVirt](https://vcluster.com/docs/platform/administer/virtual-machines/overview.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Experimental Feature Virtual machine management is experimental. Breaking changes may occur between releases. The feature is enabled on a per-customer basis. Don't use it for production workloads without guidance from vCluster Labs. vCluster Platform integrates with [KubeVirt](https://kubevirt.io/) to manage virtual machines (VMs) on connected control plane clusters. Administrators can deploy KubeVirt, create reusable VM templates, and manage KubeVirt `VirtualMachine` resources from the platform UI. ## Basic setup[​](#basic-setup "Direct link to Basic setup") Create a [KubeVirt node provider](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md) to connect vCluster Platform to a cluster where VMs should run. The provider can deploy KubeVirt automatically, or it can use an existing KubeVirt installation. Modify the following with your specific values to generate a copyable command: NODE\_PROVIDERkubevirt-vms NETWORK\_ENVIRONMENTdefault CLUSTER\_NAMEmy-kubevirt-cluster NAMESPACEvcluster-platform kubevirt-vms.yaml ``` # Connect vCluster Platform to the KubeVirt cluster. apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: kubevirt-vms spec: displayName: "KubeVirt VMs" # Defaults that apply to Machines created through this provider. properties: # Use vcluster.com/os-image instead when you have a reusable OSImage resource. kubevirt.vcluster.com/image-url: docker://quay.io/containerdisks/ubuntu:22.04 kubevirt.vcluster.com/root-disk-size: 20Gi # Optional: reference a NodeEnvironment for network defaults. # vcluster.com/network-environment: default kubeVirt: # Create VMs in this connected control plane cluster. clusterRef: cluster: my-kubevirt-cluster namespace: vcluster-platform # Deploy KubeVirt automatically if it isn't already installed. deploy: kubevirt: enabled: true # Base KubeVirt VirtualMachine template for this provider. virtualMachineTemplate: spec: runStrategy: Always template: spec: domain: resources: requests: cpu: "1" memory: 2Gi nodeTypes: - name: small-ubuntu displayName: "Small Ubuntu VM" maxCapacity: 10 ``` This configuration creates a KubeVirt provider with one VM size. vCluster Platform uses the provider template and properties when it creates VMs. Apply the provider to the management cluster: Modify the following with your specific values to generate a copyable command: FILEkubevirt-vms.yaml ``` kubectl apply -f kubevirt-vms.yaml ``` ## When to use virtual machines[​](#when-to-use-virtual-machines "Direct link to When to use virtual machines") Virtual machines are a good fit when teams need VM-level isolation or guest operating system control while keeping provisioning inside Kubernetes. Common use cases include: * **Self-service compute**: Give teams a catalog of VM sizes without granting direct access to the KubeVirt cluster. * **Isolated workloads**: Run workloads that need stronger isolation than containers, custom kernels, or full guest operating systems. * **Tenant cluster private nodes**: Provision VMs that join tenant clusters as private worker nodes through [auto nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/auto-nodes.md). * **Reusable templates**: Standardize VM CPU and memory through node types and templates. Configure images and network defaults with properties. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before managing virtual machines, you need: * A connected control plane cluster where VMs should run. * KubeVirt installed on that cluster, or a KubeVirt node provider configured with `spec.kubeVirt.deploy.kubevirt.enabled: true`. * Containerized disk images, Containerized Data Importer (CDI) data volumes, or data sources that KubeVirt can use for VM storage. * Optional [OS images](https://vcluster.com/docs/platform/use-platform/machines/os-images.md) and [SSH keys](https://vcluster.com/docs/platform/use-platform/machines/ssh-keys.md) for reusable provisioning configuration. ## Manage virtual machines[​](#manage-virtual-machines "Direct link to Manage virtual machines") After the provider is configured, administrators can use the platform's virtual machine views to inspect and operate KubeVirt resources across connected clusters. The VM management views cover: | Resource | Purpose | | -------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | | `VirtualMachine` | Desired VM configuration and run strategy. | | `VirtualMachineInstance` | Running VM instance status, phase, IP addresses, and console access. | | [`DataVolume`](https://vcluster.com/docs/platform/administer/virtual-machines/data-volumes.md) | Persistent disk content imported or cloned through CDI. | | [`NetworkAttachmentDefinition`](https://vcluster.com/docs/platform/administer/virtual-machines/network-attachment-definitions.md) | Multus network definition that can attach VMs to additional networks. | | `VirtualMachineSnapshot` | Snapshot request for a VM. | | `VirtualMachineRestore` | Restore request from a VM snapshot. | Available actions depend on KubeVirt state and user permissions. For example, administrators can start, stop, restart, pause, and unpause VMs. They can also open a VM console, inspect guest logs, create snapshots, and restore from snapshots. ## Workflow[​](#workflow "Direct link to Workflow") Use this workflow to configure a provider, create a VM-backed machine, and verify the KubeVirt VM on the target cluster. 1. ### Configure the provider[​](#configure-the-provider "Direct link to Configure the provider") Create a KubeVirt `NodeProvider` with a base `virtualMachineTemplate`. The provider controls where VMs run and which base template they use. 2. ### Define available VM sizes[​](#define-available-vm-sizes "Direct link to Define available VM sizes") Define one or more node types that represent available VM sizes. Each node type can override or merge parts of the provider template. 3. ### Create a Machine[​](#create-a-machine "Direct link to Create a Machine") Create a Machine from the platform UI. A tenant cluster can also create a Machine through auto nodes. 4. ### Verify the VM[​](#verify-the-vm "Direct link to Verify the VM") Verify the platform-created VM on the KubeVirt cluster: ``` kubectl get virtualmachines -n vcluster-platform kubectl get virtualmachineinstances -n vcluster-platform ``` Use the platform UI to view VM status, open the console, inspect logs, and manage snapshots. ## Configure images, storage, and networking[​](#configure-images-storage-and-networking "Direct link to Configure images, storage, and networking") KubeVirt VM configuration is controlled by the provider template, node types, Machine properties, and reusable resources. Common properties include: | Property | Description | | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `vcluster.com/os-image` | References a reusable [KubeVirt OSImage](https://vcluster.com/docs/platform/use-platform/machines/os-images.md#kubevirt). | | `vcluster.com/ssh-keys` | References one or more `SSHKey` resources. | | `vcluster.com/network-environment` | References a `NodeEnvironment` whose properties merge into the Machine. | | `vcluster.com/user-data` | Provides inline cloud-init user data. | | `vcluster.com/user-data-template-config` | References a [MachineConfigTemplate](https://vcluster.com/docs/platform/use-platform/machines/machine-config-templates.md) that renders user data. | | `kubevirt.vcluster.com/vm-template` | Supplies a full VM template override. | | `kubevirt.vcluster.com/merge-virtual-machine-template` | Merges VM template changes into the provider template. | | `kubevirt.vcluster.com/image-url` | Uses an HTTP, HTTPS, or `docker://` image source. | | `kubevirt.vcluster.com/image-datasource` | References a CDI `DataSource`. | | `kubevirt.vcluster.com/root-disk-size` | Sets the root disk size when image configuration is used. | | `kubevirt.vcluster.com/network-data` | Provides base64-encoded cloud-init network data. | | `kubevirt.vcluster.com/network-attachment-definition` | Attaches the VM to a NetworkAttachmentDefinition. | For standalone storage and networking administration, see [Data Volumes](https://vcluster.com/docs/platform/administer/virtual-machines/data-volumes.md) and [Network Attachment Definitions](https://vcluster.com/docs/platform/administer/virtual-machines/network-attachment-definitions.md). For the full provider configuration, template behavior, image properties, and Netris networking example, see the [KubeVirt node provider documentation](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md). ## Access control[​](#access-control "Direct link to Access control") Virtual machine management uses platform permissions and Kubernetes role-based access control (RBAC) on the connected cluster. Users must have permission to view or operate the relevant KubeVirt resources before the platform can show VM details or perform actions on their behalf. If a user shouldn't manage infrastructure, don't grant access to the VM management views or the underlying KubeVirt resources. --- # App Instance AppInstance deploys an App into a tenant cluster, space, or connected cluster, and reports the resulting Helm release status. It replaces the Task and HelmRelease resources removed in Platform 4.12. See [What are Apps](https://vcluster.com/docs/platform/understand/what-are-apps.md) for the concept, [Upgrade to Platform 4.12](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade.md#upgrade-to-4-12) for migrating an existing app deployment, and [Retrieve App Instance Logs](https://vcluster.com/docs/platform/api/resources/appinstancelog.md) for the log subresource. ## AppInstance example[​](#appinstance-example "Direct link to AppInstance example") An example AppInstance: ``` apiVersion: management.loft.sh/v1 kind: AppInstance metadata: name: my-argo-cd namespace: p-my-project spec: destination: virtualCluster: name: development target: vCluster displayName: ArgoCD owner: user: admin templateRef: name: my-app status: {} ``` ## AppInstance reference[​](#appinstance-reference "Direct link to AppInstance reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes the app instance ### `templateRef` object [​](#spec-templateRef "Direct link to spec-templateRef")TemplateRef holds the reference to the app to deploy #### `name` string [​](#spec-templateRef-name "Direct link to spec-templateRef-name")Name holds the name of the app to reference. #### `version` string [​](#spec-templateRef-version "Direct link to spec-templateRef-version")Version of the app to deploy. If empty, the latest version is deployed. ### `template` object [​](#spec-template "Direct link to spec-template")Template is the inline app configuration to deploy. Mutually exclusive with TemplateRef. #### `defaultNamespace` string [​](#spec-template-defaultNamespace "Direct link to spec-template-defaultNamespace")DefaultNamespace is the default namespace this app should installed in. #### `readme` string [​](#spec-template-readme "Direct link to spec-template-readme")Readme is a longer markdown string that describes the app. #### `icon` string [​](#spec-template-icon "Direct link to spec-template-icon")Icon holds an URL to the app icon #### `config` object [​](#spec-template-config "Direct link to spec-template-config")Config is the helm config to use to deploy the helm release ##### `chart` object [​](#spec-template-config-chart "Direct link to spec-template-config-chart")Chart holds information about a chart that should get deployed ##### `name` string [​](#spec-template-config-chart-name "Direct link to spec-template-config-chart-name")Name is the chart name in the repository ##### `version` string [​](#spec-template-config-chart-version "Direct link to spec-template-config-chart-version")Version is the chart version in the repository ##### `repoURL` string [​](#spec-template-config-chart-repoURL "Direct link to spec-template-config-chart-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#spec-template-config-chart-username "Direct link to spec-template-config-chart-username")The username that is required for this repository ##### `usernameRef` object [​](#spec-template-config-chart-usernameRef "Direct link to spec-template-config-chart-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#spec-template-config-chart-usernameRef-projectSecretRef "Direct link to spec-template-config-chart-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-config-chart-usernameRef-projectSecretRef-project "Direct link to spec-template-config-chart-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-config-chart-usernameRef-projectSecretRef-name "Direct link to spec-template-config-chart-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-config-chart-usernameRef-projectSecretRef-key "Direct link to spec-template-config-chart-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#spec-template-config-chart-password "Direct link to spec-template-config-chart-password")The password that is required for this repository ##### `passwordRef` object [​](#spec-template-config-chart-passwordRef "Direct link to spec-template-config-chart-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#spec-template-config-chart-passwordRef-projectSecretRef "Direct link to spec-template-config-chart-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-config-chart-passwordRef-projectSecretRef-project "Direct link to spec-template-config-chart-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-config-chart-passwordRef-projectSecretRef-name "Direct link to spec-template-config-chart-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-config-chart-passwordRef-projectSecretRef-key "Direct link to spec-template-config-chart-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#spec-template-config-chart-insecureSkipTlsVerify "Direct link to spec-template-config-chart-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `manifests` string [​](#spec-template-config-manifests "Direct link to spec-template-config-manifests")Manifests holds kube manifests that will be deployed as a chart ##### `bash` object [​](#spec-template-config-bash "Direct link to spec-template-config-bash")Bash holds the bash script to execute in a container in the target ##### `script` string [​](#spec-template-config-bash-script "Direct link to spec-template-config-bash-script")Script is the script to execute. ##### `image` string [​](#spec-template-config-bash-image "Direct link to spec-template-config-bash-image")Image is the image to use for this app ##### `clusterRole` string [​](#spec-template-config-bash-clusterRole "Direct link to spec-template-config-bash-clusterRole")ClusterRole is the cluster role to use for this job ##### `podSecurityContext` object [​](#spec-template-config-bash-podSecurityContext "Direct link to spec-template-config-bash-podSecurityContext")PodSecurityContext for the bash pod. ##### `seLinuxOptions` object [​](#spec-template-config-bash-podSecurityContext-seLinuxOptions "Direct link to spec-template-config-bash-podSecurityContext-seLinuxOptions")The SELinux context to be applied to all containers. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `user` string [​](#spec-template-config-bash-podSecurityContext-seLinuxOptions-user "Direct link to spec-template-config-bash-podSecurityContext-seLinuxOptions-user")User is a SELinux user label that applies to the container. ##### `role` string [​](#spec-template-config-bash-podSecurityContext-seLinuxOptions-role "Direct link to spec-template-config-bash-podSecurityContext-seLinuxOptions-role")Role is a SELinux role label that applies to the container. ##### `type` string [​](#spec-template-config-bash-podSecurityContext-seLinuxOptions-type "Direct link to spec-template-config-bash-podSecurityContext-seLinuxOptions-type")Type is a SELinux type label that applies to the container. ##### `level` string [​](#spec-template-config-bash-podSecurityContext-seLinuxOptions-level "Direct link to spec-template-config-bash-podSecurityContext-seLinuxOptions-level")Level is SELinux level label that applies to the container. ##### `windowsOptions` object [​](#spec-template-config-bash-podSecurityContext-windowsOptions "Direct link to spec-template-config-bash-podSecurityContext-windowsOptions")The Windows specific settings applied to all containers. If unspecified, the options within a container's SecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. ##### `gmsaCredentialSpecName` string [​](#spec-template-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpecName "Direct link to spec-template-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpecName")GMSACredentialSpecName is the name of the GMSA credential spec to use. ##### `gmsaCredentialSpec` string [​](#spec-template-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpec "Direct link to spec-template-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpec")GMSACredentialSpec is where the GMSA admission webhook () inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. ##### `runAsUserName` string [​](#spec-template-config-bash-podSecurityContext-windowsOptions-runAsUserName "Direct link to spec-template-config-bash-podSecurityContext-windowsOptions-runAsUserName")The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `hostProcess` boolean [​](#spec-template-config-bash-podSecurityContext-windowsOptions-hostProcess "Direct link to spec-template-config-bash-podSecurityContext-windowsOptions-hostProcess")HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. ##### `runAsUser` integer [​](#spec-template-config-bash-podSecurityContext-runAsUser "Direct link to spec-template-config-bash-podSecurityContext-runAsUser")The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `runAsGroup` integer [​](#spec-template-config-bash-podSecurityContext-runAsGroup "Direct link to spec-template-config-bash-podSecurityContext-runAsGroup")The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `runAsNonRoot` boolean [​](#spec-template-config-bash-podSecurityContext-runAsNonRoot "Direct link to spec-template-config-bash-podSecurityContext-runAsNonRoot")Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `supplementalGroups` integer\[] [​](#spec-template-config-bash-podSecurityContext-supplementalGroups "Direct link to spec-template-config-bash-podSecurityContext-supplementalGroups")A list of groups applied to the first process run in each container, in addition to the container's primary GID and fsGroup (if specified). If the SupplementalGroupsPolicy feature is enabled, the supplementalGroupsPolicy field determines whether these are in addition to or instead of any group memberships defined in the container image. If unspecified, no additional groups are added, though group memberships defined in the container image may still be used, depending on the supplementalGroupsPolicy field. Note that this field cannot be set when spec.os.name is windows. ##### `supplementalGroupsPolicy` string [​](#spec-template-config-bash-podSecurityContext-supplementalGroupsPolicy "Direct link to spec-template-config-bash-podSecurityContext-supplementalGroupsPolicy")Defines how supplemental groups of the first container processes are calculated. Valid values are "Merge" and "Strict". If not specified, "Merge" is used. (Alpha) Using the field requires the SupplementalGroupsPolicy feature gate to be enabled and the container runtime must implement support for this feature. Note that this field cannot be set when spec.os.name is windows. TODO: update the default value to "Merge" when spec.os.name is not windows in v1.34 ##### `fsGroup` integer [​](#spec-template-config-bash-podSecurityContext-fsGroup "Direct link to spec-template-config-bash-podSecurityContext-fsGroup")A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod:1) The owning GID will be the FSGroup 2) The setgid bit is set (new files created in the volume will be owned by FSGroup) 3) The permission bits are OR'd with rw-rw----If unset, the Kubelet will not modify the ownership and permissions of any volume. Note that this field cannot be set when spec.os.name is windows. ##### `sysctls` object\[] [​](#spec-template-config-bash-podSecurityContext-sysctls "Direct link to spec-template-config-bash-podSecurityContext-sysctls")Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported sysctls (by the container runtime) might fail to launch. Note that this field cannot be set when spec.os.name is windows. ##### `name` required string [​](#spec-template-config-bash-podSecurityContext-sysctls-name "Direct link to spec-template-config-bash-podSecurityContext-sysctls-name")Name of a property to set ##### `value` required string [​](#spec-template-config-bash-podSecurityContext-sysctls-value "Direct link to spec-template-config-bash-podSecurityContext-sysctls-value")Value of a property to set ##### `fsGroupChangePolicy` string [​](#spec-template-config-bash-podSecurityContext-fsGroupChangePolicy "Direct link to spec-template-config-bash-podSecurityContext-fsGroupChangePolicy")fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. Note that this field cannot be set when spec.os.name is windows. ##### `seccompProfile` object [​](#spec-template-config-bash-podSecurityContext-seccompProfile "Direct link to spec-template-config-bash-podSecurityContext-seccompProfile")The seccomp options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-template-config-bash-podSecurityContext-seccompProfile-type "Direct link to spec-template-config-bash-podSecurityContext-seccompProfile-type")type indicates which kind of seccomp profile will be applied. Valid options are:Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. ##### `localhostProfile` string [​](#spec-template-config-bash-podSecurityContext-seccompProfile-localhostProfile "Direct link to spec-template-config-bash-podSecurityContext-seccompProfile-localhostProfile")localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. ##### `appArmorProfile` object [​](#spec-template-config-bash-podSecurityContext-appArmorProfile "Direct link to spec-template-config-bash-podSecurityContext-appArmorProfile")appArmorProfile is the AppArmor options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-template-config-bash-podSecurityContext-appArmorProfile-type "Direct link to spec-template-config-bash-podSecurityContext-appArmorProfile-type")type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. ##### `localhostProfile` string [​](#spec-template-config-bash-podSecurityContext-appArmorProfile-localhostProfile "Direct link to spec-template-config-bash-podSecurityContext-appArmorProfile-localhostProfile")localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". ##### `seLinuxChangePolicy` string [​](#spec-template-config-bash-podSecurityContext-seLinuxChangePolicy "Direct link to spec-template-config-bash-podSecurityContext-seLinuxChangePolicy")seLinuxChangePolicy defines how the container's SELinux label is applied to all volumes used by the Pod. It has no effect on nodes that do not support SELinux or to volumes does not support SELinux. Valid values are "MountOption" and "Recursive"."Recursive" means relabeling of all files on all Pod volumes by the container runtime. This may be slow for large volumes, but allows mixing privileged and unprivileged Pods sharing the same volume on the same node."MountOption" mounts all eligible Pod volumes with `-o context` mount option. This requires all Pods that share the same volume to use the same SELinux label. It is not possible to share the same volume among privileged and unprivileged Pods. Eligible volumes are in-tree FibreChannel and iSCSI volumes, and all CSI volumes whose CSI driver announces SELinux support by setting spec.seLinuxMount: true in their CSIDriver instance. Other volumes are always re-labelled recursively. "MountOption" value is allowed only when SELinuxMount feature gate is enabled.If not specified and SELinuxMount feature gate is enabled, "MountOption" is used. If not specified and SELinuxMount feature gate is disabled, "MountOption" is used for ReadWriteOncePod volumes and "Recursive" for all other volumes.This field affects only Pods that have SELinux label set, either in PodSecurityContext or in SecurityContext of all containers.All Pods that use the same volume should use the same seLinuxChangePolicy, otherwise some pods can get stuck in ContainerCreating state. Note that this field cannot be set when spec.os.name is windows. ##### `securityContext` object [​](#spec-template-config-bash-securityContext "Direct link to spec-template-config-bash-securityContext")SecurityContext for the bash container. ##### `capabilities` object [​](#spec-template-config-bash-securityContext-capabilities "Direct link to spec-template-config-bash-securityContext-capabilities")The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container runtime. Note that this field cannot be set when spec.os.name is windows. ##### `add` string\[] [​](#spec-template-config-bash-securityContext-capabilities-add "Direct link to spec-template-config-bash-securityContext-capabilities-add")Added capabilities ##### `drop` string\[] [​](#spec-template-config-bash-securityContext-capabilities-drop "Direct link to spec-template-config-bash-securityContext-capabilities-drop")Removed capabilities ##### `privileged` boolean [​](#spec-template-config-bash-securityContext-privileged "Direct link to spec-template-config-bash-securityContext-privileged")Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false. Note that this field cannot be set when spec.os.name is windows. ##### `seLinuxOptions` object [​](#spec-template-config-bash-securityContext-seLinuxOptions "Direct link to spec-template-config-bash-securityContext-seLinuxOptions")The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `user` string [​](#spec-template-config-bash-securityContext-seLinuxOptions-user "Direct link to spec-template-config-bash-securityContext-seLinuxOptions-user")User is a SELinux user label that applies to the container. ##### `role` string [​](#spec-template-config-bash-securityContext-seLinuxOptions-role "Direct link to spec-template-config-bash-securityContext-seLinuxOptions-role")Role is a SELinux role label that applies to the container. ##### `type` string [​](#spec-template-config-bash-securityContext-seLinuxOptions-type "Direct link to spec-template-config-bash-securityContext-seLinuxOptions-type")Type is a SELinux type label that applies to the container. ##### `level` string [​](#spec-template-config-bash-securityContext-seLinuxOptions-level "Direct link to spec-template-config-bash-securityContext-seLinuxOptions-level")Level is SELinux level label that applies to the container. ##### `windowsOptions` object [​](#spec-template-config-bash-securityContext-windowsOptions "Direct link to spec-template-config-bash-securityContext-windowsOptions")The Windows specific settings applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. ##### `gmsaCredentialSpecName` string [​](#spec-template-config-bash-securityContext-windowsOptions-gmsaCredentialSpecName "Direct link to spec-template-config-bash-securityContext-windowsOptions-gmsaCredentialSpecName")GMSACredentialSpecName is the name of the GMSA credential spec to use. ##### `gmsaCredentialSpec` string [​](#spec-template-config-bash-securityContext-windowsOptions-gmsaCredentialSpec "Direct link to spec-template-config-bash-securityContext-windowsOptions-gmsaCredentialSpec")GMSACredentialSpec is where the GMSA admission webhook () inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. ##### `runAsUserName` string [​](#spec-template-config-bash-securityContext-windowsOptions-runAsUserName "Direct link to spec-template-config-bash-securityContext-windowsOptions-runAsUserName")The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `hostProcess` boolean [​](#spec-template-config-bash-securityContext-windowsOptions-hostProcess "Direct link to spec-template-config-bash-securityContext-windowsOptions-hostProcess")HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. ##### `runAsUser` integer [​](#spec-template-config-bash-securityContext-runAsUser "Direct link to spec-template-config-bash-securityContext-runAsUser")The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `runAsGroup` integer [​](#spec-template-config-bash-securityContext-runAsGroup "Direct link to spec-template-config-bash-securityContext-runAsGroup")The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `runAsNonRoot` boolean [​](#spec-template-config-bash-securityContext-runAsNonRoot "Direct link to spec-template-config-bash-securityContext-runAsNonRoot")Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `readOnlyRootFilesystem` boolean [​](#spec-template-config-bash-securityContext-readOnlyRootFilesystem "Direct link to spec-template-config-bash-securityContext-readOnlyRootFilesystem")Whether this container has a read-only root filesystem. Default is false. Note that this field cannot be set when spec.os.name is windows. ##### `allowPrivilegeEscalation` boolean [​](#spec-template-config-bash-securityContext-allowPrivilegeEscalation "Direct link to spec-template-config-bash-securityContext-allowPrivilegeEscalation")AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no\_new\_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is:1) run as Privileged 2) has CAP\_SYS\_ADMIN Note that this field cannot be set when spec.os.name is windows. ##### `procMount` string [​](#spec-template-config-bash-securityContext-procMount "Direct link to spec-template-config-bash-securityContext-procMount")procMount denotes the type of proc mount to use for the containers. The default value is Default which uses the container runtime defaults for readonly paths and masked paths. Note that this field cannot be set when spec.os.name is windows. ##### `seccompProfile` object [​](#spec-template-config-bash-securityContext-seccompProfile "Direct link to spec-template-config-bash-securityContext-seccompProfile")The seccomp options to use by this container. If seccomp options are provided at both the pod & container level, the container options override the pod options. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-template-config-bash-securityContext-seccompProfile-type "Direct link to spec-template-config-bash-securityContext-seccompProfile-type")type indicates which kind of seccomp profile will be applied. Valid options are:Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. ##### `localhostProfile` string [​](#spec-template-config-bash-securityContext-seccompProfile-localhostProfile "Direct link to spec-template-config-bash-securityContext-seccompProfile-localhostProfile")localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. ##### `appArmorProfile` object [​](#spec-template-config-bash-securityContext-appArmorProfile "Direct link to spec-template-config-bash-securityContext-appArmorProfile")appArmorProfile is the AppArmor options to use by this container. If set, this profile overrides the pod's appArmorProfile. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-template-config-bash-securityContext-appArmorProfile-type "Direct link to spec-template-config-bash-securityContext-appArmorProfile-type")type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. ##### `localhostProfile` string [​](#spec-template-config-bash-securityContext-appArmorProfile-localhostProfile "Direct link to spec-template-config-bash-securityContext-appArmorProfile-localhostProfile")localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". ##### `values` string [​](#spec-template-config-values "Direct link to spec-template-config-values")Values is the set of extra Values added to the chart. These values merge with the default values inside of the chart. You can use golang templating in here with values from parameters. ##### `parameters` string [​](#spec-template-config-parameters "Direct link to spec-template-config-parameters")Parameters are additional helm chart values that will get merged with config and are then used to deploy the helm chart. ##### `annotations` object [​](#spec-template-config-annotations "Direct link to spec-template-config-annotations")Annotations are extra annotations for this helm release #### `wait` boolean [​](#spec-template-wait "Direct link to spec-template-wait")Wait determines if Loft should wait during deploy for the app to become ready #### `timeout` string [​](#spec-template-timeout "Direct link to spec-template-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) #### `parameters` object\[] [​](#spec-template-parameters "Direct link to spec-template-parameters")Parameters define additional app parameters that will set helm values ##### `variable` string [​](#spec-template-parameters-variable "Direct link to spec-template-parameters-variable")Variable is the path of the variable. Can be foo or foo.bar for nested objects. ##### `label` string [​](#spec-template-parameters-label "Direct link to spec-template-parameters-label")Label is the label to show for this parameter ##### `description` string [​](#spec-template-parameters-description "Direct link to spec-template-parameters-description")Description is the description to show for this parameter ##### `type` string [​](#spec-template-parameters-type "Direct link to spec-template-parameters-type")Type of the parameter. Can be one of: string, multiline, boolean, number and password ##### `options` string\[] [​](#spec-template-parameters-options "Direct link to spec-template-parameters-options")Options is a slice of strings, where each string represents a mutually exclusive choice. ##### `min` integer [​](#spec-template-parameters-min "Direct link to spec-template-parameters-min")Min is the minimum number if type is number ##### `max` integer [​](#spec-template-parameters-max "Direct link to spec-template-parameters-max")Max is the maximum number if type is number ##### `required` boolean [​](#spec-template-parameters-required "Direct link to spec-template-parameters-required")Required specifies if this parameter is required ##### `defaultValue` string [​](#spec-template-parameters-defaultValue "Direct link to spec-template-parameters-defaultValue")DefaultValue is the default value if none is specified ##### `placeholder` string [​](#spec-template-parameters-placeholder "Direct link to spec-template-parameters-placeholder")Placeholder shown in the UI ##### `invalidation` string [​](#spec-template-parameters-invalidation "Direct link to spec-template-parameters-invalidation")Invalidation regex that if matched will reject the input ##### `validation` string [​](#spec-template-parameters-validation "Direct link to spec-template-parameters-validation")Validation regex that if matched will allow the input ##### `section` string [​](#spec-template-parameters-section "Direct link to spec-template-parameters-section")Section where this app should be displayed. Apps with the same section name will be grouped together #### `streamContainer` object [​](#spec-template-streamContainer "Direct link to spec-template-streamContainer")DEPRECATED: Use config.bash instead StreamContainer can be used to stream a containers logs instead of the helm output. ##### `selector` object [​](#spec-template-streamContainer-selector "Direct link to spec-template-streamContainer-selector")Label selector for pods. The newest matching pod will be used to stream logs from ##### `matchLabels` object [​](#spec-template-streamContainer-selector-matchLabels "Direct link to spec-template-streamContainer-selector-matchLabels")matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. ##### `matchExpressions` object\[] [​](#spec-template-streamContainer-selector-matchExpressions "Direct link to spec-template-streamContainer-selector-matchExpressions")matchExpressions is a list of label selector requirements. The requirements are ANDed. ##### `key` required string [​](#spec-template-streamContainer-selector-matchExpressions-key "Direct link to spec-template-streamContainer-selector-matchExpressions-key")key is the label key that the selector applies to. ##### `operator` required string [​](#spec-template-streamContainer-selector-matchExpressions-operator "Direct link to spec-template-streamContainer-selector-matchExpressions-operator")operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. ##### `values` string\[] [​](#spec-template-streamContainer-selector-matchExpressions-values "Direct link to spec-template-streamContainer-selector-matchExpressions-values")values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. ##### `container` string [​](#spec-template-streamContainer-container "Direct link to spec-template-streamContainer-container")Container is the container name to use ### `parameters` object [​](#spec-parameters "Direct link to spec-parameters")Parameters are values to pass to the app. The values should be encoded as YAML string where each parameter is represented as a top-level field key. ### `releaseName` string [​](#spec-releaseName "Direct link to spec-releaseName")ReleaseName is the name of the helm release that is created for this app instance. If empty, defaults to the name of the AppInstance. ### `destination` object [​](#spec-destination "Direct link to spec-destination")Destination defines where the app is deployed to #### `virtualCluster` object [​](#spec-destination-virtualCluster "Direct link to spec-destination-virtualCluster")VirtualCluster deploys the app into a virtual cluster instance. Mutually exclusive with Space and Cluster. ##### `name` string [​](#spec-destination-virtualCluster-name "Direct link to spec-destination-virtualCluster-name")Name of the virtual cluster instance within the project ##### `namespace` string [​](#spec-destination-virtualCluster-namespace "Direct link to spec-destination-virtualCluster-namespace")Namespace the helm release is deployed into. Only used when target is vCluster; for the host target the release is always deployed into the virtual cluster's host namespace. If empty, uses the app's default namespace only for initial release resolution. Existing instances retain their recorded release coordinates. ##### `target` string [​](#spec-destination-virtualCluster-target "Direct link to spec-destination-virtualCluster-target")Target selects whether the app is deployed inside the virtual cluster or into its host namespace #### `space` object [​](#spec-destination-space "Direct link to spec-destination-space")Space deploys the app into the namespace of a space instance. Mutually exclusive with VirtualCluster and Cluster. ##### `name` string [​](#spec-destination-space-name "Direct link to spec-destination-space-name")Name of the space instance within the project. The helm release is deployed into the namespace of the space. #### `cluster` object [​](#spec-destination-cluster "Direct link to spec-destination-cluster")Cluster deploys the app into a connected cluster. Mutually exclusive with VirtualCluster and Space. ##### `name` string [​](#spec-destination-cluster-name "Direct link to spec-destination-cluster-name")Name of the connected cluster ##### `namespace` string [​](#spec-destination-cluster-namespace "Direct link to spec-destination-cluster-namespace")Namespace in the cluster the helm release is deployed into. If empty, uses the app's default namespace only for initial release resolution. Existing instances retain their recorded release coordinates. ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ## `status` object [​](#status "Direct link to status") ### `phase` string [​](#status-phase "Direct link to status-phase")Phase describes the current phase the app instance is in ### `reason` string [​](#status-reason "Direct link to status-reason")Reason describes the reason in machine-readable form why the instance is in the current phase ### `message` string [​](#status-message "Direct link to status-message")Message describes the reason in human-readable form why the instance is in the current phase ### `observedGeneration` integer [​](#status-observedGeneration "Direct link to status-observedGeneration")ObservedGeneration is the last generation that was reconciled ### `conditions` object [​](#status-conditions "Direct link to status-conditions")Conditions holds several conditions the app instance might be in ### `app` object [​](#status-app "Direct link to status-app")App holds the resolved app configuration that was last deployed #### `defaultNamespace` string [​](#status-app-defaultNamespace "Direct link to status-app-defaultNamespace")DefaultNamespace is the default namespace this app should installed in. #### `readme` string [​](#status-app-readme "Direct link to status-app-readme")Readme is a longer markdown string that describes the app. #### `icon` string [​](#status-app-icon "Direct link to status-app-icon")Icon holds an URL to the app icon #### `config` object [​](#status-app-config "Direct link to status-app-config")Config is the helm config to use to deploy the helm release ##### `chart` object [​](#status-app-config-chart "Direct link to status-app-config-chart")Chart holds information about a chart that should get deployed ##### `name` string [​](#status-app-config-chart-name "Direct link to status-app-config-chart-name")Name is the chart name in the repository ##### `version` string [​](#status-app-config-chart-version "Direct link to status-app-config-chart-version")Version is the chart version in the repository ##### `repoURL` string [​](#status-app-config-chart-repoURL "Direct link to status-app-config-chart-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#status-app-config-chart-username "Direct link to status-app-config-chart-username")The username that is required for this repository ##### `usernameRef` object [​](#status-app-config-chart-usernameRef "Direct link to status-app-config-chart-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#status-app-config-chart-usernameRef-projectSecretRef "Direct link to status-app-config-chart-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#status-app-config-chart-usernameRef-projectSecretRef-project "Direct link to status-app-config-chart-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#status-app-config-chart-usernameRef-projectSecretRef-name "Direct link to status-app-config-chart-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#status-app-config-chart-usernameRef-projectSecretRef-key "Direct link to status-app-config-chart-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#status-app-config-chart-password "Direct link to status-app-config-chart-password")The password that is required for this repository ##### `passwordRef` object [​](#status-app-config-chart-passwordRef "Direct link to status-app-config-chart-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#status-app-config-chart-passwordRef-projectSecretRef "Direct link to status-app-config-chart-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#status-app-config-chart-passwordRef-projectSecretRef-project "Direct link to status-app-config-chart-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#status-app-config-chart-passwordRef-projectSecretRef-name "Direct link to status-app-config-chart-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#status-app-config-chart-passwordRef-projectSecretRef-key "Direct link to status-app-config-chart-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#status-app-config-chart-insecureSkipTlsVerify "Direct link to status-app-config-chart-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `manifests` string [​](#status-app-config-manifests "Direct link to status-app-config-manifests")Manifests holds kube manifests that will be deployed as a chart ##### `bash` object [​](#status-app-config-bash "Direct link to status-app-config-bash")Bash holds the bash script to execute in a container in the target ##### `script` string [​](#status-app-config-bash-script "Direct link to status-app-config-bash-script")Script is the script to execute. ##### `image` string [​](#status-app-config-bash-image "Direct link to status-app-config-bash-image")Image is the image to use for this app ##### `clusterRole` string [​](#status-app-config-bash-clusterRole "Direct link to status-app-config-bash-clusterRole")ClusterRole is the cluster role to use for this job ##### `podSecurityContext` object [​](#status-app-config-bash-podSecurityContext "Direct link to status-app-config-bash-podSecurityContext")PodSecurityContext for the bash pod. ##### `seLinuxOptions` object [​](#status-app-config-bash-podSecurityContext-seLinuxOptions "Direct link to status-app-config-bash-podSecurityContext-seLinuxOptions")The SELinux context to be applied to all containers. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `user` string [​](#status-app-config-bash-podSecurityContext-seLinuxOptions-user "Direct link to status-app-config-bash-podSecurityContext-seLinuxOptions-user")User is a SELinux user label that applies to the container. ##### `role` string [​](#status-app-config-bash-podSecurityContext-seLinuxOptions-role "Direct link to status-app-config-bash-podSecurityContext-seLinuxOptions-role")Role is a SELinux role label that applies to the container. ##### `type` string [​](#status-app-config-bash-podSecurityContext-seLinuxOptions-type "Direct link to status-app-config-bash-podSecurityContext-seLinuxOptions-type")Type is a SELinux type label that applies to the container. ##### `level` string [​](#status-app-config-bash-podSecurityContext-seLinuxOptions-level "Direct link to status-app-config-bash-podSecurityContext-seLinuxOptions-level")Level is SELinux level label that applies to the container. ##### `windowsOptions` object [​](#status-app-config-bash-podSecurityContext-windowsOptions "Direct link to status-app-config-bash-podSecurityContext-windowsOptions")The Windows specific settings applied to all containers. If unspecified, the options within a container's SecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. ##### `gmsaCredentialSpecName` string [​](#status-app-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpecName "Direct link to status-app-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpecName")GMSACredentialSpecName is the name of the GMSA credential spec to use. ##### `gmsaCredentialSpec` string [​](#status-app-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpec "Direct link to status-app-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpec")GMSACredentialSpec is where the GMSA admission webhook () inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. ##### `runAsUserName` string [​](#status-app-config-bash-podSecurityContext-windowsOptions-runAsUserName "Direct link to status-app-config-bash-podSecurityContext-windowsOptions-runAsUserName")The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `hostProcess` boolean [​](#status-app-config-bash-podSecurityContext-windowsOptions-hostProcess "Direct link to status-app-config-bash-podSecurityContext-windowsOptions-hostProcess")HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. ##### `runAsUser` integer [​](#status-app-config-bash-podSecurityContext-runAsUser "Direct link to status-app-config-bash-podSecurityContext-runAsUser")The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `runAsGroup` integer [​](#status-app-config-bash-podSecurityContext-runAsGroup "Direct link to status-app-config-bash-podSecurityContext-runAsGroup")The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `runAsNonRoot` boolean [​](#status-app-config-bash-podSecurityContext-runAsNonRoot "Direct link to status-app-config-bash-podSecurityContext-runAsNonRoot")Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `supplementalGroups` integer\[] [​](#status-app-config-bash-podSecurityContext-supplementalGroups "Direct link to status-app-config-bash-podSecurityContext-supplementalGroups")A list of groups applied to the first process run in each container, in addition to the container's primary GID and fsGroup (if specified). If the SupplementalGroupsPolicy feature is enabled, the supplementalGroupsPolicy field determines whether these are in addition to or instead of any group memberships defined in the container image. If unspecified, no additional groups are added, though group memberships defined in the container image may still be used, depending on the supplementalGroupsPolicy field. Note that this field cannot be set when spec.os.name is windows. ##### `supplementalGroupsPolicy` string [​](#status-app-config-bash-podSecurityContext-supplementalGroupsPolicy "Direct link to status-app-config-bash-podSecurityContext-supplementalGroupsPolicy")Defines how supplemental groups of the first container processes are calculated. Valid values are "Merge" and "Strict". If not specified, "Merge" is used. (Alpha) Using the field requires the SupplementalGroupsPolicy feature gate to be enabled and the container runtime must implement support for this feature. Note that this field cannot be set when spec.os.name is windows. TODO: update the default value to "Merge" when spec.os.name is not windows in v1.34 ##### `fsGroup` integer [​](#status-app-config-bash-podSecurityContext-fsGroup "Direct link to status-app-config-bash-podSecurityContext-fsGroup")A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod:1) The owning GID will be the FSGroup 2) The setgid bit is set (new files created in the volume will be owned by FSGroup) 3) The permission bits are OR'd with rw-rw----If unset, the Kubelet will not modify the ownership and permissions of any volume. Note that this field cannot be set when spec.os.name is windows. ##### `sysctls` object\[] [​](#status-app-config-bash-podSecurityContext-sysctls "Direct link to status-app-config-bash-podSecurityContext-sysctls")Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported sysctls (by the container runtime) might fail to launch. Note that this field cannot be set when spec.os.name is windows. ##### `name` required string [​](#status-app-config-bash-podSecurityContext-sysctls-name "Direct link to status-app-config-bash-podSecurityContext-sysctls-name")Name of a property to set ##### `value` required string [​](#status-app-config-bash-podSecurityContext-sysctls-value "Direct link to status-app-config-bash-podSecurityContext-sysctls-value")Value of a property to set ##### `fsGroupChangePolicy` string [​](#status-app-config-bash-podSecurityContext-fsGroupChangePolicy "Direct link to status-app-config-bash-podSecurityContext-fsGroupChangePolicy")fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. Note that this field cannot be set when spec.os.name is windows. ##### `seccompProfile` object [​](#status-app-config-bash-podSecurityContext-seccompProfile "Direct link to status-app-config-bash-podSecurityContext-seccompProfile")The seccomp options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#status-app-config-bash-podSecurityContext-seccompProfile-type "Direct link to status-app-config-bash-podSecurityContext-seccompProfile-type")type indicates which kind of seccomp profile will be applied. Valid options are:Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. ##### `localhostProfile` string [​](#status-app-config-bash-podSecurityContext-seccompProfile-localhostProfile "Direct link to status-app-config-bash-podSecurityContext-seccompProfile-localhostProfile")localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. ##### `appArmorProfile` object [​](#status-app-config-bash-podSecurityContext-appArmorProfile "Direct link to status-app-config-bash-podSecurityContext-appArmorProfile")appArmorProfile is the AppArmor options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#status-app-config-bash-podSecurityContext-appArmorProfile-type "Direct link to status-app-config-bash-podSecurityContext-appArmorProfile-type")type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. ##### `localhostProfile` string [​](#status-app-config-bash-podSecurityContext-appArmorProfile-localhostProfile "Direct link to status-app-config-bash-podSecurityContext-appArmorProfile-localhostProfile")localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". ##### `seLinuxChangePolicy` string [​](#status-app-config-bash-podSecurityContext-seLinuxChangePolicy "Direct link to status-app-config-bash-podSecurityContext-seLinuxChangePolicy")seLinuxChangePolicy defines how the container's SELinux label is applied to all volumes used by the Pod. It has no effect on nodes that do not support SELinux or to volumes does not support SELinux. Valid values are "MountOption" and "Recursive"."Recursive" means relabeling of all files on all Pod volumes by the container runtime. This may be slow for large volumes, but allows mixing privileged and unprivileged Pods sharing the same volume on the same node."MountOption" mounts all eligible Pod volumes with `-o context` mount option. This requires all Pods that share the same volume to use the same SELinux label. It is not possible to share the same volume among privileged and unprivileged Pods. Eligible volumes are in-tree FibreChannel and iSCSI volumes, and all CSI volumes whose CSI driver announces SELinux support by setting spec.seLinuxMount: true in their CSIDriver instance. Other volumes are always re-labelled recursively. "MountOption" value is allowed only when SELinuxMount feature gate is enabled.If not specified and SELinuxMount feature gate is enabled, "MountOption" is used. If not specified and SELinuxMount feature gate is disabled, "MountOption" is used for ReadWriteOncePod volumes and "Recursive" for all other volumes.This field affects only Pods that have SELinux label set, either in PodSecurityContext or in SecurityContext of all containers.All Pods that use the same volume should use the same seLinuxChangePolicy, otherwise some pods can get stuck in ContainerCreating state. Note that this field cannot be set when spec.os.name is windows. ##### `securityContext` object [​](#status-app-config-bash-securityContext "Direct link to status-app-config-bash-securityContext")SecurityContext for the bash container. ##### `capabilities` object [​](#status-app-config-bash-securityContext-capabilities "Direct link to status-app-config-bash-securityContext-capabilities")The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container runtime. Note that this field cannot be set when spec.os.name is windows. ##### `add` string\[] [​](#status-app-config-bash-securityContext-capabilities-add "Direct link to status-app-config-bash-securityContext-capabilities-add")Added capabilities ##### `drop` string\[] [​](#status-app-config-bash-securityContext-capabilities-drop "Direct link to status-app-config-bash-securityContext-capabilities-drop")Removed capabilities ##### `privileged` boolean [​](#status-app-config-bash-securityContext-privileged "Direct link to status-app-config-bash-securityContext-privileged")Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false. Note that this field cannot be set when spec.os.name is windows. ##### `seLinuxOptions` object [​](#status-app-config-bash-securityContext-seLinuxOptions "Direct link to status-app-config-bash-securityContext-seLinuxOptions")The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `user` string [​](#status-app-config-bash-securityContext-seLinuxOptions-user "Direct link to status-app-config-bash-securityContext-seLinuxOptions-user")User is a SELinux user label that applies to the container. ##### `role` string [​](#status-app-config-bash-securityContext-seLinuxOptions-role "Direct link to status-app-config-bash-securityContext-seLinuxOptions-role")Role is a SELinux role label that applies to the container. ##### `type` string [​](#status-app-config-bash-securityContext-seLinuxOptions-type "Direct link to status-app-config-bash-securityContext-seLinuxOptions-type")Type is a SELinux type label that applies to the container. ##### `level` string [​](#status-app-config-bash-securityContext-seLinuxOptions-level "Direct link to status-app-config-bash-securityContext-seLinuxOptions-level")Level is SELinux level label that applies to the container. ##### `windowsOptions` object [​](#status-app-config-bash-securityContext-windowsOptions "Direct link to status-app-config-bash-securityContext-windowsOptions")The Windows specific settings applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. ##### `gmsaCredentialSpecName` string [​](#status-app-config-bash-securityContext-windowsOptions-gmsaCredentialSpecName "Direct link to status-app-config-bash-securityContext-windowsOptions-gmsaCredentialSpecName")GMSACredentialSpecName is the name of the GMSA credential spec to use. ##### `gmsaCredentialSpec` string [​](#status-app-config-bash-securityContext-windowsOptions-gmsaCredentialSpec "Direct link to status-app-config-bash-securityContext-windowsOptions-gmsaCredentialSpec")GMSACredentialSpec is where the GMSA admission webhook () inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. ##### `runAsUserName` string [​](#status-app-config-bash-securityContext-windowsOptions-runAsUserName "Direct link to status-app-config-bash-securityContext-windowsOptions-runAsUserName")The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `hostProcess` boolean [​](#status-app-config-bash-securityContext-windowsOptions-hostProcess "Direct link to status-app-config-bash-securityContext-windowsOptions-hostProcess")HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. ##### `runAsUser` integer [​](#status-app-config-bash-securityContext-runAsUser "Direct link to status-app-config-bash-securityContext-runAsUser")The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `runAsGroup` integer [​](#status-app-config-bash-securityContext-runAsGroup "Direct link to status-app-config-bash-securityContext-runAsGroup")The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `runAsNonRoot` boolean [​](#status-app-config-bash-securityContext-runAsNonRoot "Direct link to status-app-config-bash-securityContext-runAsNonRoot")Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `readOnlyRootFilesystem` boolean [​](#status-app-config-bash-securityContext-readOnlyRootFilesystem "Direct link to status-app-config-bash-securityContext-readOnlyRootFilesystem")Whether this container has a read-only root filesystem. Default is false. Note that this field cannot be set when spec.os.name is windows. ##### `allowPrivilegeEscalation` boolean [​](#status-app-config-bash-securityContext-allowPrivilegeEscalation "Direct link to status-app-config-bash-securityContext-allowPrivilegeEscalation")AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no\_new\_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is:1) run as Privileged 2) has CAP\_SYS\_ADMIN Note that this field cannot be set when spec.os.name is windows. ##### `procMount` string [​](#status-app-config-bash-securityContext-procMount "Direct link to status-app-config-bash-securityContext-procMount")procMount denotes the type of proc mount to use for the containers. The default value is Default which uses the container runtime defaults for readonly paths and masked paths. Note that this field cannot be set when spec.os.name is windows. ##### `seccompProfile` object [​](#status-app-config-bash-securityContext-seccompProfile "Direct link to status-app-config-bash-securityContext-seccompProfile")The seccomp options to use by this container. If seccomp options are provided at both the pod & container level, the container options override the pod options. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#status-app-config-bash-securityContext-seccompProfile-type "Direct link to status-app-config-bash-securityContext-seccompProfile-type")type indicates which kind of seccomp profile will be applied. Valid options are:Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. ##### `localhostProfile` string [​](#status-app-config-bash-securityContext-seccompProfile-localhostProfile "Direct link to status-app-config-bash-securityContext-seccompProfile-localhostProfile")localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. ##### `appArmorProfile` object [​](#status-app-config-bash-securityContext-appArmorProfile "Direct link to status-app-config-bash-securityContext-appArmorProfile")appArmorProfile is the AppArmor options to use by this container. If set, this profile overrides the pod's appArmorProfile. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#status-app-config-bash-securityContext-appArmorProfile-type "Direct link to status-app-config-bash-securityContext-appArmorProfile-type")type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. ##### `localhostProfile` string [​](#status-app-config-bash-securityContext-appArmorProfile-localhostProfile "Direct link to status-app-config-bash-securityContext-appArmorProfile-localhostProfile")localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". ##### `values` string [​](#status-app-config-values "Direct link to status-app-config-values")Values is the set of extra Values added to the chart. These values merge with the default values inside of the chart. You can use golang templating in here with values from parameters. ##### `parameters` string [​](#status-app-config-parameters "Direct link to status-app-config-parameters")Parameters are additional helm chart values that will get merged with config and are then used to deploy the helm chart. ##### `annotations` object [​](#status-app-config-annotations "Direct link to status-app-config-annotations")Annotations are extra annotations for this helm release #### `wait` boolean [​](#status-app-wait "Direct link to status-app-wait")Wait determines if Loft should wait during deploy for the app to become ready #### `timeout` string [​](#status-app-timeout "Direct link to status-app-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) #### `parameters` object\[] [​](#status-app-parameters "Direct link to status-app-parameters")Parameters define additional app parameters that will set helm values ##### `variable` string [​](#status-app-parameters-variable "Direct link to status-app-parameters-variable")Variable is the path of the variable. Can be foo or foo.bar for nested objects. ##### `label` string [​](#status-app-parameters-label "Direct link to status-app-parameters-label")Label is the label to show for this parameter ##### `description` string [​](#status-app-parameters-description "Direct link to status-app-parameters-description")Description is the description to show for this parameter ##### `type` string [​](#status-app-parameters-type "Direct link to status-app-parameters-type")Type of the parameter. Can be one of: string, multiline, boolean, number and password ##### `options` string\[] [​](#status-app-parameters-options "Direct link to status-app-parameters-options")Options is a slice of strings, where each string represents a mutually exclusive choice. ##### `min` integer [​](#status-app-parameters-min "Direct link to status-app-parameters-min")Min is the minimum number if type is number ##### `max` integer [​](#status-app-parameters-max "Direct link to status-app-parameters-max")Max is the maximum number if type is number ##### `required` boolean [​](#status-app-parameters-required "Direct link to status-app-parameters-required")Required specifies if this parameter is required ##### `defaultValue` string [​](#status-app-parameters-defaultValue "Direct link to status-app-parameters-defaultValue")DefaultValue is the default value if none is specified ##### `placeholder` string [​](#status-app-parameters-placeholder "Direct link to status-app-parameters-placeholder")Placeholder shown in the UI ##### `invalidation` string [​](#status-app-parameters-invalidation "Direct link to status-app-parameters-invalidation")Invalidation regex that if matched will reject the input ##### `validation` string [​](#status-app-parameters-validation "Direct link to status-app-parameters-validation")Validation regex that if matched will allow the input ##### `section` string [​](#status-app-parameters-section "Direct link to status-app-parameters-section")Section where this app should be displayed. Apps with the same section name will be grouped together #### `streamContainer` object [​](#status-app-streamContainer "Direct link to status-app-streamContainer")DEPRECATED: Use config.bash instead StreamContainer can be used to stream a containers logs instead of the helm output. ##### `selector` object [​](#status-app-streamContainer-selector "Direct link to status-app-streamContainer-selector")Label selector for pods. The newest matching pod will be used to stream logs from ##### `matchLabels` object [​](#status-app-streamContainer-selector-matchLabels "Direct link to status-app-streamContainer-selector-matchLabels")matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. ##### `matchExpressions` object\[] [​](#status-app-streamContainer-selector-matchExpressions "Direct link to status-app-streamContainer-selector-matchExpressions")matchExpressions is a list of label selector requirements. The requirements are ANDed. ##### `key` required string [​](#status-app-streamContainer-selector-matchExpressions-key "Direct link to status-app-streamContainer-selector-matchExpressions-key")key is the label key that the selector applies to. ##### `operator` required string [​](#status-app-streamContainer-selector-matchExpressions-operator "Direct link to status-app-streamContainer-selector-matchExpressions-operator")operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. ##### `values` string\[] [​](#status-app-streamContainer-selector-matchExpressions-values "Direct link to status-app-streamContainer-selector-matchExpressions-values")values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. ##### `container` string [​](#status-app-streamContainer-container "Direct link to status-app-streamContainer-container")Container is the container name to use ### `version` string [​](#status-version "Direct link to status-version")Version is the resolved app version that was last deployed ### `releaseName` string [​](#status-releaseName "Direct link to status-releaseName")ReleaseName is the effective name of the helm release created by the last deployment. It is recorded so the release can be uninstalled even if the referenced app or its default namespace can no longer be resolved. ### `releaseNamespace` string [​](#status-releaseNamespace "Direct link to status-releaseNamespace")ReleaseNamespace is the effective namespace the helm release was deployed into by the last deployment. ### `revision` integer [​](#status-revision "Direct link to status-revision")Revision is the revision of the helm release created by the last deployment ### `deployAttempts` integer [​](#status-deployAttempts "Direct link to status-deployAttempts")DeployAttempts counts the consecutive failed deploy attempts for the current spec generation and resolved app configuration. It backs the automatic retry of failed deploys and is reset whenever the deploy input changes or a deploy succeeds. ### `lastDeployTime` object [​](#status-lastDeployTime "Direct link to status-lastDeployTime")LastDeployTime is when the last deploy attempt finished, successful or not. Together with DeployAttempts it schedules the automatic retries of failed deploys. ## Retrieve: AppInstances[​](#retrieve-appinstances "Direct link to Retrieve: AppInstances") You can either use curl or kubectl to retrieve AppInstances. * kubectl * curl ### Retrieve a list of AppInstances[​](#retrieve-a-list-of-appinstances "Direct link to Retrieve a list of AppInstances") Run the following command to list all AppInstances in project `my-project`: ``` kubectl get appinstances.management.loft.sh -n p-my-project -o yaml ``` ### Retrieve a single AppInstance by name[​](#retrieve-a-single-appinstance-by-name "Direct link to Retrieve a single AppInstance by name") Run the following kubectl command to get AppInstance `my-argo-cd` in project `my-project`: ``` kubectl get appinstances.management.loft.sh my-argo-cd -n p-my-project -o yaml ``` ### Retrieve a list of AppInstances[​](#retrieve-a-list-of-appinstances-1 "Direct link to Retrieve a list of AppInstances") Run the following curl command to list all AppInstances in project `my-project`: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/appinstances" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single AppInstance by name[​](#get-a-single-appinstance-by-name "Direct link to Get a single AppInstance by name") Run the following curl command to get AppInstance `my-argo-cd` in project `my-project`: ``` # Exchange my-argo-cd in the url below with the name of the AppInstance curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/appinstances/my-argo-cd" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: AppInstance[​](#create-appinstance "Direct link to Create: AppInstance") You can either use curl or kubectl to create a new AppInstance. Make sure to set the project in the `metadata.namespace` field you want to create the AppInstance in. If your project has the id `my-project`, the corresponding namespace would be `p-my-project`. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: AppInstance metadata: name: my-argo-cd namespace: p-my-project spec: destination: virtualCluster: name: development target: vCluster displayName: ArgoCD owner: user: admin templateRef: name: my-app status: {} ``` Then create the AppInstance `my-argo-cd` in project `my-project` with: ``` kubectl create -f object.yaml -n p-my-project ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: AppInstance metadata: name: my-argo-cd namespace: p-my-project spec: destination: virtualCluster: name: development target: vCluster displayName: ArgoCD owner: user: admin templateRef: name: my-app status: {} ``` Run the following curl command to create a new AppInstance `my-argo-cd` in project `my-project`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/appinstances" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: AppInstance[​](#update-appinstance "Direct link to Update: AppInstance") You can either use curl or kubectl to update AppInstances. * kubectl * curl ### Update AppInstance[​](#update-appinstance "Direct link to Update AppInstance") Run the following command to update AppInstance `my-argo-cd` in project `my-project`: ``` kubectl edit appinstances.management.loft.sh my-argo-cd -n p-my-project ``` Then edit the object and upon save, kubectl will update the resource. ### Patch AppInstance[​](#patch-appinstance "Direct link to Patch AppInstance") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the AppInstance `my-argo-cd` in project `my-project` via a patch: ``` kubectl patch appinstances.management.loft.sh my-argo-cd -n p-my-project \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update AppInstance[​](#update-appinstance-1 "Direct link to Update AppInstance") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: AppInstance metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-argo-cd namespace: p-my-project resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: destination: virtualCluster: name: development target: vCluster displayName: ArgoCD owner: user: admin templateRef: name: my-app status: {} ``` Run the following curl command to update a single AppInstance `my-argo-cd` in project `my-project`: ``` # Replace the my-argo-cd in the url below with the name of the AppInstance you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/appinstances/my-argo-cd" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch AppInstance[​](#patch-appinstance-1 "Direct link to Patch AppInstance") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the AppInstance `my-argo-cd` in project `my-project` via a patch: ``` # Replace the my-argo-cd in the url below with the name of the AppInstance you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/appinstances/my-argo-cd" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: AppInstance[​](#delete-appinstance "Direct link to Delete: AppInstance") You can either use curl or kubectl to delete AppInstances. * kubectl * curl Run the following command to delete AppInstance `my-argo-cd` in project `my-project`: ``` kubectl delete appinstances.management.loft.sh my-argo-cd -n p-my-project ``` Run the following curl command to delete AppInstance `my-argo-cd` in project `my-project`: ``` # Replace the my-argo-cd in the url below with the name of the AppInstance you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/appinstances/my-argo-cd" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Retrieve App Instance Logs Stream deployment logs for an AppInstance. Set the `follow` query parameter to `true` to continue streaming as new output is stored. Platform 4.12 doesn't apply the other fields exposed by AppInstanceLogOptions. ## App instance log example[​](#app-instance-log-example "Direct link to App instance log example") An example App instance log: ``` apiVersion: management.loft.sh/v1 kind: AppInstanceLog metadata: name: my-argo-cd namespace: p-my-project ``` ## App instance log reference[​](#app-instance-log-reference "Direct link to App instance log reference") ## App instance log (Get)[​](#app-instance-log-get "Direct link to App instance log (Get)") Stream the deployment logs for an AppInstance. Append `?follow=true` to continue streaming as new output is stored. * kubectl * curl Run the following command: ``` # Exchange my-argo-cd in the url below with the name of the AppInstance kubectl get --raw "/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/appinstances/my-argo-cd/log" ``` Run the following curl command: ``` # Exchange my-argo-cd in the url below with the name of the AppInstance curl -s -X GET --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/appinstances/my-argo-cd/log" \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # App Apps in the platform are a way for admins to package applications and scripts in consumable packages. These Apps can then be deployed into clusters, spaces, or virtual clusters. ## App example[​](#app-example "Direct link to App example") An example App: ``` apiVersion: management.loft.sh/v1 kind: App metadata: name: my-app spec: access: - users: - '*' verbs: - get config: chart: name: argo-cd repoURL: https://argoproj.github.io/argo-helm description: Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes displayName: ArgoCD icon: https://argo-cd.readthedocs.io/en/stable/assets/logo.png recommendedApp: - cluster status: {} ``` ## App reference[​](#app-reference "Direct link to App reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes an app ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `clusters` string\[] [​](#spec-clusters "Direct link to spec-clusters")Clusters are the clusters this app can be installed in. ### `recommendedApp` string\[] [​](#spec-recommendedApp "Direct link to spec-recommendedApp")RecommendedApp specifies where this app should show up as recommended app ### `defaultNamespace` string [​](#spec-defaultNamespace "Direct link to spec-defaultNamespace")DefaultNamespace is the default namespace this app should installed in. ### `readme` string [​](#spec-readme "Direct link to spec-readme")Readme is a longer markdown string that describes the app. ### `icon` string [​](#spec-icon "Direct link to spec-icon")Icon holds an URL to the app icon ### `config` object [​](#spec-config "Direct link to spec-config")Config is the helm config to use to deploy the helm release #### `chart` object [​](#spec-config-chart "Direct link to spec-config-chart")Chart holds information about a chart that should get deployed ##### `name` string [​](#spec-config-chart-name "Direct link to spec-config-chart-name")Name is the chart name in the repository ##### `version` string [​](#spec-config-chart-version "Direct link to spec-config-chart-version")Version is the chart version in the repository ##### `repoURL` string [​](#spec-config-chart-repoURL "Direct link to spec-config-chart-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#spec-config-chart-username "Direct link to spec-config-chart-username")The username that is required for this repository ##### `usernameRef` object [​](#spec-config-chart-usernameRef "Direct link to spec-config-chart-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#spec-config-chart-usernameRef-projectSecretRef "Direct link to spec-config-chart-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-config-chart-usernameRef-projectSecretRef-project "Direct link to spec-config-chart-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-config-chart-usernameRef-projectSecretRef-name "Direct link to spec-config-chart-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-config-chart-usernameRef-projectSecretRef-key "Direct link to spec-config-chart-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#spec-config-chart-password "Direct link to spec-config-chart-password")The password that is required for this repository ##### `passwordRef` object [​](#spec-config-chart-passwordRef "Direct link to spec-config-chart-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#spec-config-chart-passwordRef-projectSecretRef "Direct link to spec-config-chart-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-config-chart-passwordRef-projectSecretRef-project "Direct link to spec-config-chart-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-config-chart-passwordRef-projectSecretRef-name "Direct link to spec-config-chart-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-config-chart-passwordRef-projectSecretRef-key "Direct link to spec-config-chart-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#spec-config-chart-insecureSkipTlsVerify "Direct link to spec-config-chart-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped #### `manifests` string [​](#spec-config-manifests "Direct link to spec-config-manifests")Manifests holds kube manifests that will be deployed as a chart #### `bash` object [​](#spec-config-bash "Direct link to spec-config-bash")Bash holds the bash script to execute in a container in the target ##### `script` string [​](#spec-config-bash-script "Direct link to spec-config-bash-script")Script is the script to execute. ##### `image` string [​](#spec-config-bash-image "Direct link to spec-config-bash-image")Image is the image to use for this app ##### `clusterRole` string [​](#spec-config-bash-clusterRole "Direct link to spec-config-bash-clusterRole")ClusterRole is the cluster role to use for this job ##### `podSecurityContext` object [​](#spec-config-bash-podSecurityContext "Direct link to spec-config-bash-podSecurityContext")PodSecurityContext for the bash pod. ##### `seLinuxOptions` object [​](#spec-config-bash-podSecurityContext-seLinuxOptions "Direct link to spec-config-bash-podSecurityContext-seLinuxOptions")The SELinux context to be applied to all containers. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `user` string [​](#spec-config-bash-podSecurityContext-seLinuxOptions-user "Direct link to spec-config-bash-podSecurityContext-seLinuxOptions-user")User is a SELinux user label that applies to the container. ##### `role` string [​](#spec-config-bash-podSecurityContext-seLinuxOptions-role "Direct link to spec-config-bash-podSecurityContext-seLinuxOptions-role")Role is a SELinux role label that applies to the container. ##### `type` string [​](#spec-config-bash-podSecurityContext-seLinuxOptions-type "Direct link to spec-config-bash-podSecurityContext-seLinuxOptions-type")Type is a SELinux type label that applies to the container. ##### `level` string [​](#spec-config-bash-podSecurityContext-seLinuxOptions-level "Direct link to spec-config-bash-podSecurityContext-seLinuxOptions-level")Level is SELinux level label that applies to the container. ##### `windowsOptions` object [​](#spec-config-bash-podSecurityContext-windowsOptions "Direct link to spec-config-bash-podSecurityContext-windowsOptions")The Windows specific settings applied to all containers. If unspecified, the options within a container's SecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. ##### `gmsaCredentialSpecName` string [​](#spec-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpecName "Direct link to spec-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpecName")GMSACredentialSpecName is the name of the GMSA credential spec to use. ##### `gmsaCredentialSpec` string [​](#spec-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpec "Direct link to spec-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpec")GMSACredentialSpec is where the GMSA admission webhook () inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. ##### `runAsUserName` string [​](#spec-config-bash-podSecurityContext-windowsOptions-runAsUserName "Direct link to spec-config-bash-podSecurityContext-windowsOptions-runAsUserName")The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `hostProcess` boolean [​](#spec-config-bash-podSecurityContext-windowsOptions-hostProcess "Direct link to spec-config-bash-podSecurityContext-windowsOptions-hostProcess")HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. ##### `runAsUser` integer [​](#spec-config-bash-podSecurityContext-runAsUser "Direct link to spec-config-bash-podSecurityContext-runAsUser")The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `runAsGroup` integer [​](#spec-config-bash-podSecurityContext-runAsGroup "Direct link to spec-config-bash-podSecurityContext-runAsGroup")The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `runAsNonRoot` boolean [​](#spec-config-bash-podSecurityContext-runAsNonRoot "Direct link to spec-config-bash-podSecurityContext-runAsNonRoot")Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `supplementalGroups` integer\[] [​](#spec-config-bash-podSecurityContext-supplementalGroups "Direct link to spec-config-bash-podSecurityContext-supplementalGroups")A list of groups applied to the first process run in each container, in addition to the container's primary GID and fsGroup (if specified). If the SupplementalGroupsPolicy feature is enabled, the supplementalGroupsPolicy field determines whether these are in addition to or instead of any group memberships defined in the container image. If unspecified, no additional groups are added, though group memberships defined in the container image may still be used, depending on the supplementalGroupsPolicy field. Note that this field cannot be set when spec.os.name is windows. ##### `supplementalGroupsPolicy` string [​](#spec-config-bash-podSecurityContext-supplementalGroupsPolicy "Direct link to spec-config-bash-podSecurityContext-supplementalGroupsPolicy")Defines how supplemental groups of the first container processes are calculated. Valid values are "Merge" and "Strict". If not specified, "Merge" is used. (Alpha) Using the field requires the SupplementalGroupsPolicy feature gate to be enabled and the container runtime must implement support for this feature. Note that this field cannot be set when spec.os.name is windows. TODO: update the default value to "Merge" when spec.os.name is not windows in v1.34 ##### `fsGroup` integer [​](#spec-config-bash-podSecurityContext-fsGroup "Direct link to spec-config-bash-podSecurityContext-fsGroup")A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod:1) The owning GID will be the FSGroup 2) The setgid bit is set (new files created in the volume will be owned by FSGroup) 3) The permission bits are OR'd with rw-rw----If unset, the Kubelet will not modify the ownership and permissions of any volume. Note that this field cannot be set when spec.os.name is windows. ##### `sysctls` object\[] [​](#spec-config-bash-podSecurityContext-sysctls "Direct link to spec-config-bash-podSecurityContext-sysctls")Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported sysctls (by the container runtime) might fail to launch. Note that this field cannot be set when spec.os.name is windows. ##### `name` required string [​](#spec-config-bash-podSecurityContext-sysctls-name "Direct link to spec-config-bash-podSecurityContext-sysctls-name")Name of a property to set ##### `value` required string [​](#spec-config-bash-podSecurityContext-sysctls-value "Direct link to spec-config-bash-podSecurityContext-sysctls-value")Value of a property to set ##### `fsGroupChangePolicy` string [​](#spec-config-bash-podSecurityContext-fsGroupChangePolicy "Direct link to spec-config-bash-podSecurityContext-fsGroupChangePolicy")fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. Note that this field cannot be set when spec.os.name is windows. ##### `seccompProfile` object [​](#spec-config-bash-podSecurityContext-seccompProfile "Direct link to spec-config-bash-podSecurityContext-seccompProfile")The seccomp options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-config-bash-podSecurityContext-seccompProfile-type "Direct link to spec-config-bash-podSecurityContext-seccompProfile-type")type indicates which kind of seccomp profile will be applied. Valid options are:Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. ##### `localhostProfile` string [​](#spec-config-bash-podSecurityContext-seccompProfile-localhostProfile "Direct link to spec-config-bash-podSecurityContext-seccompProfile-localhostProfile")localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. ##### `appArmorProfile` object [​](#spec-config-bash-podSecurityContext-appArmorProfile "Direct link to spec-config-bash-podSecurityContext-appArmorProfile")appArmorProfile is the AppArmor options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-config-bash-podSecurityContext-appArmorProfile-type "Direct link to spec-config-bash-podSecurityContext-appArmorProfile-type")type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. ##### `localhostProfile` string [​](#spec-config-bash-podSecurityContext-appArmorProfile-localhostProfile "Direct link to spec-config-bash-podSecurityContext-appArmorProfile-localhostProfile")localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". ##### `seLinuxChangePolicy` string [​](#spec-config-bash-podSecurityContext-seLinuxChangePolicy "Direct link to spec-config-bash-podSecurityContext-seLinuxChangePolicy")seLinuxChangePolicy defines how the container's SELinux label is applied to all volumes used by the Pod. It has no effect on nodes that do not support SELinux or to volumes does not support SELinux. Valid values are "MountOption" and "Recursive"."Recursive" means relabeling of all files on all Pod volumes by the container runtime. This may be slow for large volumes, but allows mixing privileged and unprivileged Pods sharing the same volume on the same node."MountOption" mounts all eligible Pod volumes with `-o context` mount option. This requires all Pods that share the same volume to use the same SELinux label. It is not possible to share the same volume among privileged and unprivileged Pods. Eligible volumes are in-tree FibreChannel and iSCSI volumes, and all CSI volumes whose CSI driver announces SELinux support by setting spec.seLinuxMount: true in their CSIDriver instance. Other volumes are always re-labelled recursively. "MountOption" value is allowed only when SELinuxMount feature gate is enabled.If not specified and SELinuxMount feature gate is enabled, "MountOption" is used. If not specified and SELinuxMount feature gate is disabled, "MountOption" is used for ReadWriteOncePod volumes and "Recursive" for all other volumes.This field affects only Pods that have SELinux label set, either in PodSecurityContext or in SecurityContext of all containers.All Pods that use the same volume should use the same seLinuxChangePolicy, otherwise some pods can get stuck in ContainerCreating state. Note that this field cannot be set when spec.os.name is windows. ##### `securityContext` object [​](#spec-config-bash-securityContext "Direct link to spec-config-bash-securityContext")SecurityContext for the bash container. ##### `capabilities` object [​](#spec-config-bash-securityContext-capabilities "Direct link to spec-config-bash-securityContext-capabilities")The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container runtime. Note that this field cannot be set when spec.os.name is windows. ##### `add` string\[] [​](#spec-config-bash-securityContext-capabilities-add "Direct link to spec-config-bash-securityContext-capabilities-add")Added capabilities ##### `drop` string\[] [​](#spec-config-bash-securityContext-capabilities-drop "Direct link to spec-config-bash-securityContext-capabilities-drop")Removed capabilities ##### `privileged` boolean [​](#spec-config-bash-securityContext-privileged "Direct link to spec-config-bash-securityContext-privileged")Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false. Note that this field cannot be set when spec.os.name is windows. ##### `seLinuxOptions` object [​](#spec-config-bash-securityContext-seLinuxOptions "Direct link to spec-config-bash-securityContext-seLinuxOptions")The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `user` string [​](#spec-config-bash-securityContext-seLinuxOptions-user "Direct link to spec-config-bash-securityContext-seLinuxOptions-user")User is a SELinux user label that applies to the container. ##### `role` string [​](#spec-config-bash-securityContext-seLinuxOptions-role "Direct link to spec-config-bash-securityContext-seLinuxOptions-role")Role is a SELinux role label that applies to the container. ##### `type` string [​](#spec-config-bash-securityContext-seLinuxOptions-type "Direct link to spec-config-bash-securityContext-seLinuxOptions-type")Type is a SELinux type label that applies to the container. ##### `level` string [​](#spec-config-bash-securityContext-seLinuxOptions-level "Direct link to spec-config-bash-securityContext-seLinuxOptions-level")Level is SELinux level label that applies to the container. ##### `windowsOptions` object [​](#spec-config-bash-securityContext-windowsOptions "Direct link to spec-config-bash-securityContext-windowsOptions")The Windows specific settings applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. ##### `gmsaCredentialSpecName` string [​](#spec-config-bash-securityContext-windowsOptions-gmsaCredentialSpecName "Direct link to spec-config-bash-securityContext-windowsOptions-gmsaCredentialSpecName")GMSACredentialSpecName is the name of the GMSA credential spec to use. ##### `gmsaCredentialSpec` string [​](#spec-config-bash-securityContext-windowsOptions-gmsaCredentialSpec "Direct link to spec-config-bash-securityContext-windowsOptions-gmsaCredentialSpec")GMSACredentialSpec is where the GMSA admission webhook () inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. ##### `runAsUserName` string [​](#spec-config-bash-securityContext-windowsOptions-runAsUserName "Direct link to spec-config-bash-securityContext-windowsOptions-runAsUserName")The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `hostProcess` boolean [​](#spec-config-bash-securityContext-windowsOptions-hostProcess "Direct link to spec-config-bash-securityContext-windowsOptions-hostProcess")HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. ##### `runAsUser` integer [​](#spec-config-bash-securityContext-runAsUser "Direct link to spec-config-bash-securityContext-runAsUser")The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `runAsGroup` integer [​](#spec-config-bash-securityContext-runAsGroup "Direct link to spec-config-bash-securityContext-runAsGroup")The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `runAsNonRoot` boolean [​](#spec-config-bash-securityContext-runAsNonRoot "Direct link to spec-config-bash-securityContext-runAsNonRoot")Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `readOnlyRootFilesystem` boolean [​](#spec-config-bash-securityContext-readOnlyRootFilesystem "Direct link to spec-config-bash-securityContext-readOnlyRootFilesystem")Whether this container has a read-only root filesystem. Default is false. Note that this field cannot be set when spec.os.name is windows. ##### `allowPrivilegeEscalation` boolean [​](#spec-config-bash-securityContext-allowPrivilegeEscalation "Direct link to spec-config-bash-securityContext-allowPrivilegeEscalation")AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no\_new\_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is:1) run as Privileged 2) has CAP\_SYS\_ADMIN Note that this field cannot be set when spec.os.name is windows. ##### `procMount` string [​](#spec-config-bash-securityContext-procMount "Direct link to spec-config-bash-securityContext-procMount")procMount denotes the type of proc mount to use for the containers. The default value is Default which uses the container runtime defaults for readonly paths and masked paths. Note that this field cannot be set when spec.os.name is windows. ##### `seccompProfile` object [​](#spec-config-bash-securityContext-seccompProfile "Direct link to spec-config-bash-securityContext-seccompProfile")The seccomp options to use by this container. If seccomp options are provided at both the pod & container level, the container options override the pod options. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-config-bash-securityContext-seccompProfile-type "Direct link to spec-config-bash-securityContext-seccompProfile-type")type indicates which kind of seccomp profile will be applied. Valid options are:Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. ##### `localhostProfile` string [​](#spec-config-bash-securityContext-seccompProfile-localhostProfile "Direct link to spec-config-bash-securityContext-seccompProfile-localhostProfile")localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. ##### `appArmorProfile` object [​](#spec-config-bash-securityContext-appArmorProfile "Direct link to spec-config-bash-securityContext-appArmorProfile")appArmorProfile is the AppArmor options to use by this container. If set, this profile overrides the pod's appArmorProfile. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-config-bash-securityContext-appArmorProfile-type "Direct link to spec-config-bash-securityContext-appArmorProfile-type")type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. ##### `localhostProfile` string [​](#spec-config-bash-securityContext-appArmorProfile-localhostProfile "Direct link to spec-config-bash-securityContext-appArmorProfile-localhostProfile")localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". #### `values` string [​](#spec-config-values "Direct link to spec-config-values")Values is the set of extra Values added to the chart. These values merge with the default values inside of the chart. You can use golang templating in here with values from parameters. #### `parameters` string [​](#spec-config-parameters "Direct link to spec-config-parameters")Parameters are additional helm chart values that will get merged with config and are then used to deploy the helm chart. #### `annotations` object [​](#spec-config-annotations "Direct link to spec-config-annotations")Annotations are extra annotations for this helm release ### `wait` boolean [​](#spec-wait "Direct link to spec-wait")Wait determines if Loft should wait during deploy for the app to become ready ### `timeout` string [​](#spec-timeout "Direct link to spec-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) ### `parameters` object\[] [​](#spec-parameters "Direct link to spec-parameters")Parameters define additional app parameters that will set helm values #### `variable` string [​](#spec-parameters-variable "Direct link to spec-parameters-variable")Variable is the path of the variable. Can be foo or foo.bar for nested objects. #### `label` string [​](#spec-parameters-label "Direct link to spec-parameters-label")Label is the label to show for this parameter #### `description` string [​](#spec-parameters-description "Direct link to spec-parameters-description")Description is the description to show for this parameter #### `type` string [​](#spec-parameters-type "Direct link to spec-parameters-type")Type of the parameter. Can be one of: string, multiline, boolean, number and password #### `options` string\[] [​](#spec-parameters-options "Direct link to spec-parameters-options")Options is a slice of strings, where each string represents a mutually exclusive choice. #### `min` integer [​](#spec-parameters-min "Direct link to spec-parameters-min")Min is the minimum number if type is number #### `max` integer [​](#spec-parameters-max "Direct link to spec-parameters-max")Max is the maximum number if type is number #### `required` boolean [​](#spec-parameters-required "Direct link to spec-parameters-required")Required specifies if this parameter is required #### `defaultValue` string [​](#spec-parameters-defaultValue "Direct link to spec-parameters-defaultValue")DefaultValue is the default value if none is specified #### `placeholder` string [​](#spec-parameters-placeholder "Direct link to spec-parameters-placeholder")Placeholder shown in the UI #### `invalidation` string [​](#spec-parameters-invalidation "Direct link to spec-parameters-invalidation")Invalidation regex that if matched will reject the input #### `validation` string [​](#spec-parameters-validation "Direct link to spec-parameters-validation")Validation regex that if matched will allow the input #### `section` string [​](#spec-parameters-section "Direct link to spec-parameters-section")Section where this app should be displayed. Apps with the same section name will be grouped together ### `streamContainer` object [​](#spec-streamContainer "Direct link to spec-streamContainer")DEPRECATED: Use config.bash instead StreamContainer can be used to stream a containers logs instead of the helm output. #### `selector` object [​](#spec-streamContainer-selector "Direct link to spec-streamContainer-selector")Label selector for pods. The newest matching pod will be used to stream logs from ##### `matchLabels` object [​](#spec-streamContainer-selector-matchLabels "Direct link to spec-streamContainer-selector-matchLabels")matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. ##### `matchExpressions` object\[] [​](#spec-streamContainer-selector-matchExpressions "Direct link to spec-streamContainer-selector-matchExpressions")matchExpressions is a list of label selector requirements. The requirements are ANDed. ##### `key` required string [​](#spec-streamContainer-selector-matchExpressions-key "Direct link to spec-streamContainer-selector-matchExpressions-key")key is the label key that the selector applies to. ##### `operator` required string [​](#spec-streamContainer-selector-matchExpressions-operator "Direct link to spec-streamContainer-selector-matchExpressions-operator")operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. ##### `values` string\[] [​](#spec-streamContainer-selector-matchExpressions-values "Direct link to spec-streamContainer-selector-matchExpressions-values")values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. #### `container` string [​](#spec-streamContainer-container "Direct link to spec-streamContainer-container")Container is the container name to use ### `versions` object\[] [​](#spec-versions "Direct link to spec-versions")Versions are different app versions that can be referenced #### `defaultNamespace` string [​](#spec-versions-defaultNamespace "Direct link to spec-versions-defaultNamespace")DefaultNamespace is the default namespace this app should installed in. #### `readme` string [​](#spec-versions-readme "Direct link to spec-versions-readme")Readme is a longer markdown string that describes the app. #### `icon` string [​](#spec-versions-icon "Direct link to spec-versions-icon")Icon holds an URL to the app icon #### `config` object [​](#spec-versions-config "Direct link to spec-versions-config")Config is the helm config to use to deploy the helm release ##### `chart` object [​](#spec-versions-config-chart "Direct link to spec-versions-config-chart")Chart holds information about a chart that should get deployed ##### `name` string [​](#spec-versions-config-chart-name "Direct link to spec-versions-config-chart-name")Name is the chart name in the repository ##### `version` string [​](#spec-versions-config-chart-version "Direct link to spec-versions-config-chart-version")Version is the chart version in the repository ##### `repoURL` string [​](#spec-versions-config-chart-repoURL "Direct link to spec-versions-config-chart-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#spec-versions-config-chart-username "Direct link to spec-versions-config-chart-username")The username that is required for this repository ##### `usernameRef` object [​](#spec-versions-config-chart-usernameRef "Direct link to spec-versions-config-chart-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#spec-versions-config-chart-usernameRef-projectSecretRef "Direct link to spec-versions-config-chart-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-versions-config-chart-usernameRef-projectSecretRef-project "Direct link to spec-versions-config-chart-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-versions-config-chart-usernameRef-projectSecretRef-name "Direct link to spec-versions-config-chart-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-versions-config-chart-usernameRef-projectSecretRef-key "Direct link to spec-versions-config-chart-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#spec-versions-config-chart-password "Direct link to spec-versions-config-chart-password")The password that is required for this repository ##### `passwordRef` object [​](#spec-versions-config-chart-passwordRef "Direct link to spec-versions-config-chart-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#spec-versions-config-chart-passwordRef-projectSecretRef "Direct link to spec-versions-config-chart-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-versions-config-chart-passwordRef-projectSecretRef-project "Direct link to spec-versions-config-chart-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-versions-config-chart-passwordRef-projectSecretRef-name "Direct link to spec-versions-config-chart-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-versions-config-chart-passwordRef-projectSecretRef-key "Direct link to spec-versions-config-chart-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#spec-versions-config-chart-insecureSkipTlsVerify "Direct link to spec-versions-config-chart-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `manifests` string [​](#spec-versions-config-manifests "Direct link to spec-versions-config-manifests")Manifests holds kube manifests that will be deployed as a chart ##### `bash` object [​](#spec-versions-config-bash "Direct link to spec-versions-config-bash")Bash holds the bash script to execute in a container in the target ##### `script` string [​](#spec-versions-config-bash-script "Direct link to spec-versions-config-bash-script")Script is the script to execute. ##### `image` string [​](#spec-versions-config-bash-image "Direct link to spec-versions-config-bash-image")Image is the image to use for this app ##### `clusterRole` string [​](#spec-versions-config-bash-clusterRole "Direct link to spec-versions-config-bash-clusterRole")ClusterRole is the cluster role to use for this job ##### `podSecurityContext` object [​](#spec-versions-config-bash-podSecurityContext "Direct link to spec-versions-config-bash-podSecurityContext")PodSecurityContext for the bash pod. ##### `seLinuxOptions` object [​](#spec-versions-config-bash-podSecurityContext-seLinuxOptions "Direct link to spec-versions-config-bash-podSecurityContext-seLinuxOptions")The SELinux context to be applied to all containers. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `user` string [​](#spec-versions-config-bash-podSecurityContext-seLinuxOptions-user "Direct link to spec-versions-config-bash-podSecurityContext-seLinuxOptions-user")User is a SELinux user label that applies to the container. ##### `role` string [​](#spec-versions-config-bash-podSecurityContext-seLinuxOptions-role "Direct link to spec-versions-config-bash-podSecurityContext-seLinuxOptions-role")Role is a SELinux role label that applies to the container. ##### `type` string [​](#spec-versions-config-bash-podSecurityContext-seLinuxOptions-type "Direct link to spec-versions-config-bash-podSecurityContext-seLinuxOptions-type")Type is a SELinux type label that applies to the container. ##### `level` string [​](#spec-versions-config-bash-podSecurityContext-seLinuxOptions-level "Direct link to spec-versions-config-bash-podSecurityContext-seLinuxOptions-level")Level is SELinux level label that applies to the container. ##### `windowsOptions` object [​](#spec-versions-config-bash-podSecurityContext-windowsOptions "Direct link to spec-versions-config-bash-podSecurityContext-windowsOptions")The Windows specific settings applied to all containers. If unspecified, the options within a container's SecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. ##### `gmsaCredentialSpecName` string [​](#spec-versions-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpecName "Direct link to spec-versions-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpecName")GMSACredentialSpecName is the name of the GMSA credential spec to use. ##### `gmsaCredentialSpec` string [​](#spec-versions-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpec "Direct link to spec-versions-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpec")GMSACredentialSpec is where the GMSA admission webhook () inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. ##### `runAsUserName` string [​](#spec-versions-config-bash-podSecurityContext-windowsOptions-runAsUserName "Direct link to spec-versions-config-bash-podSecurityContext-windowsOptions-runAsUserName")The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `hostProcess` boolean [​](#spec-versions-config-bash-podSecurityContext-windowsOptions-hostProcess "Direct link to spec-versions-config-bash-podSecurityContext-windowsOptions-hostProcess")HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. ##### `runAsUser` integer [​](#spec-versions-config-bash-podSecurityContext-runAsUser "Direct link to spec-versions-config-bash-podSecurityContext-runAsUser")The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `runAsGroup` integer [​](#spec-versions-config-bash-podSecurityContext-runAsGroup "Direct link to spec-versions-config-bash-podSecurityContext-runAsGroup")The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `runAsNonRoot` boolean [​](#spec-versions-config-bash-podSecurityContext-runAsNonRoot "Direct link to spec-versions-config-bash-podSecurityContext-runAsNonRoot")Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `supplementalGroups` integer\[] [​](#spec-versions-config-bash-podSecurityContext-supplementalGroups "Direct link to spec-versions-config-bash-podSecurityContext-supplementalGroups")A list of groups applied to the first process run in each container, in addition to the container's primary GID and fsGroup (if specified). If the SupplementalGroupsPolicy feature is enabled, the supplementalGroupsPolicy field determines whether these are in addition to or instead of any group memberships defined in the container image. If unspecified, no additional groups are added, though group memberships defined in the container image may still be used, depending on the supplementalGroupsPolicy field. Note that this field cannot be set when spec.os.name is windows. ##### `supplementalGroupsPolicy` string [​](#spec-versions-config-bash-podSecurityContext-supplementalGroupsPolicy "Direct link to spec-versions-config-bash-podSecurityContext-supplementalGroupsPolicy")Defines how supplemental groups of the first container processes are calculated. Valid values are "Merge" and "Strict". If not specified, "Merge" is used. (Alpha) Using the field requires the SupplementalGroupsPolicy feature gate to be enabled and the container runtime must implement support for this feature. Note that this field cannot be set when spec.os.name is windows. TODO: update the default value to "Merge" when spec.os.name is not windows in v1.34 ##### `fsGroup` integer [​](#spec-versions-config-bash-podSecurityContext-fsGroup "Direct link to spec-versions-config-bash-podSecurityContext-fsGroup")A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod:1) The owning GID will be the FSGroup 2) The setgid bit is set (new files created in the volume will be owned by FSGroup) 3) The permission bits are OR'd with rw-rw----If unset, the Kubelet will not modify the ownership and permissions of any volume. Note that this field cannot be set when spec.os.name is windows. ##### `sysctls` object\[] [​](#spec-versions-config-bash-podSecurityContext-sysctls "Direct link to spec-versions-config-bash-podSecurityContext-sysctls")Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported sysctls (by the container runtime) might fail to launch. Note that this field cannot be set when spec.os.name is windows. ##### `name` required string [​](#spec-versions-config-bash-podSecurityContext-sysctls-name "Direct link to spec-versions-config-bash-podSecurityContext-sysctls-name")Name of a property to set ##### `value` required string [​](#spec-versions-config-bash-podSecurityContext-sysctls-value "Direct link to spec-versions-config-bash-podSecurityContext-sysctls-value")Value of a property to set ##### `fsGroupChangePolicy` string [​](#spec-versions-config-bash-podSecurityContext-fsGroupChangePolicy "Direct link to spec-versions-config-bash-podSecurityContext-fsGroupChangePolicy")fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. Note that this field cannot be set when spec.os.name is windows. ##### `seccompProfile` object [​](#spec-versions-config-bash-podSecurityContext-seccompProfile "Direct link to spec-versions-config-bash-podSecurityContext-seccompProfile")The seccomp options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-versions-config-bash-podSecurityContext-seccompProfile-type "Direct link to spec-versions-config-bash-podSecurityContext-seccompProfile-type")type indicates which kind of seccomp profile will be applied. Valid options are:Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. ##### `localhostProfile` string [​](#spec-versions-config-bash-podSecurityContext-seccompProfile-localhostProfile "Direct link to spec-versions-config-bash-podSecurityContext-seccompProfile-localhostProfile")localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. ##### `appArmorProfile` object [​](#spec-versions-config-bash-podSecurityContext-appArmorProfile "Direct link to spec-versions-config-bash-podSecurityContext-appArmorProfile")appArmorProfile is the AppArmor options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-versions-config-bash-podSecurityContext-appArmorProfile-type "Direct link to spec-versions-config-bash-podSecurityContext-appArmorProfile-type")type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. ##### `localhostProfile` string [​](#spec-versions-config-bash-podSecurityContext-appArmorProfile-localhostProfile "Direct link to spec-versions-config-bash-podSecurityContext-appArmorProfile-localhostProfile")localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". ##### `seLinuxChangePolicy` string [​](#spec-versions-config-bash-podSecurityContext-seLinuxChangePolicy "Direct link to spec-versions-config-bash-podSecurityContext-seLinuxChangePolicy")seLinuxChangePolicy defines how the container's SELinux label is applied to all volumes used by the Pod. It has no effect on nodes that do not support SELinux or to volumes does not support SELinux. Valid values are "MountOption" and "Recursive"."Recursive" means relabeling of all files on all Pod volumes by the container runtime. This may be slow for large volumes, but allows mixing privileged and unprivileged Pods sharing the same volume on the same node."MountOption" mounts all eligible Pod volumes with `-o context` mount option. This requires all Pods that share the same volume to use the same SELinux label. It is not possible to share the same volume among privileged and unprivileged Pods. Eligible volumes are in-tree FibreChannel and iSCSI volumes, and all CSI volumes whose CSI driver announces SELinux support by setting spec.seLinuxMount: true in their CSIDriver instance. Other volumes are always re-labelled recursively. "MountOption" value is allowed only when SELinuxMount feature gate is enabled.If not specified and SELinuxMount feature gate is enabled, "MountOption" is used. If not specified and SELinuxMount feature gate is disabled, "MountOption" is used for ReadWriteOncePod volumes and "Recursive" for all other volumes.This field affects only Pods that have SELinux label set, either in PodSecurityContext or in SecurityContext of all containers.All Pods that use the same volume should use the same seLinuxChangePolicy, otherwise some pods can get stuck in ContainerCreating state. Note that this field cannot be set when spec.os.name is windows. ##### `securityContext` object [​](#spec-versions-config-bash-securityContext "Direct link to spec-versions-config-bash-securityContext")SecurityContext for the bash container. ##### `capabilities` object [​](#spec-versions-config-bash-securityContext-capabilities "Direct link to spec-versions-config-bash-securityContext-capabilities")The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container runtime. Note that this field cannot be set when spec.os.name is windows. ##### `add` string\[] [​](#spec-versions-config-bash-securityContext-capabilities-add "Direct link to spec-versions-config-bash-securityContext-capabilities-add")Added capabilities ##### `drop` string\[] [​](#spec-versions-config-bash-securityContext-capabilities-drop "Direct link to spec-versions-config-bash-securityContext-capabilities-drop")Removed capabilities ##### `privileged` boolean [​](#spec-versions-config-bash-securityContext-privileged "Direct link to spec-versions-config-bash-securityContext-privileged")Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false. Note that this field cannot be set when spec.os.name is windows. ##### `seLinuxOptions` object [​](#spec-versions-config-bash-securityContext-seLinuxOptions "Direct link to spec-versions-config-bash-securityContext-seLinuxOptions")The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `user` string [​](#spec-versions-config-bash-securityContext-seLinuxOptions-user "Direct link to spec-versions-config-bash-securityContext-seLinuxOptions-user")User is a SELinux user label that applies to the container. ##### `role` string [​](#spec-versions-config-bash-securityContext-seLinuxOptions-role "Direct link to spec-versions-config-bash-securityContext-seLinuxOptions-role")Role is a SELinux role label that applies to the container. ##### `type` string [​](#spec-versions-config-bash-securityContext-seLinuxOptions-type "Direct link to spec-versions-config-bash-securityContext-seLinuxOptions-type")Type is a SELinux type label that applies to the container. ##### `level` string [​](#spec-versions-config-bash-securityContext-seLinuxOptions-level "Direct link to spec-versions-config-bash-securityContext-seLinuxOptions-level")Level is SELinux level label that applies to the container. ##### `windowsOptions` object [​](#spec-versions-config-bash-securityContext-windowsOptions "Direct link to spec-versions-config-bash-securityContext-windowsOptions")The Windows specific settings applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. ##### `gmsaCredentialSpecName` string [​](#spec-versions-config-bash-securityContext-windowsOptions-gmsaCredentialSpecName "Direct link to spec-versions-config-bash-securityContext-windowsOptions-gmsaCredentialSpecName")GMSACredentialSpecName is the name of the GMSA credential spec to use. ##### `gmsaCredentialSpec` string [​](#spec-versions-config-bash-securityContext-windowsOptions-gmsaCredentialSpec "Direct link to spec-versions-config-bash-securityContext-windowsOptions-gmsaCredentialSpec")GMSACredentialSpec is where the GMSA admission webhook () inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. ##### `runAsUserName` string [​](#spec-versions-config-bash-securityContext-windowsOptions-runAsUserName "Direct link to spec-versions-config-bash-securityContext-windowsOptions-runAsUserName")The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `hostProcess` boolean [​](#spec-versions-config-bash-securityContext-windowsOptions-hostProcess "Direct link to spec-versions-config-bash-securityContext-windowsOptions-hostProcess")HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. ##### `runAsUser` integer [​](#spec-versions-config-bash-securityContext-runAsUser "Direct link to spec-versions-config-bash-securityContext-runAsUser")The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `runAsGroup` integer [​](#spec-versions-config-bash-securityContext-runAsGroup "Direct link to spec-versions-config-bash-securityContext-runAsGroup")The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `runAsNonRoot` boolean [​](#spec-versions-config-bash-securityContext-runAsNonRoot "Direct link to spec-versions-config-bash-securityContext-runAsNonRoot")Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `readOnlyRootFilesystem` boolean [​](#spec-versions-config-bash-securityContext-readOnlyRootFilesystem "Direct link to spec-versions-config-bash-securityContext-readOnlyRootFilesystem")Whether this container has a read-only root filesystem. Default is false. Note that this field cannot be set when spec.os.name is windows. ##### `allowPrivilegeEscalation` boolean [​](#spec-versions-config-bash-securityContext-allowPrivilegeEscalation "Direct link to spec-versions-config-bash-securityContext-allowPrivilegeEscalation")AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no\_new\_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is:1) run as Privileged 2) has CAP\_SYS\_ADMIN Note that this field cannot be set when spec.os.name is windows. ##### `procMount` string [​](#spec-versions-config-bash-securityContext-procMount "Direct link to spec-versions-config-bash-securityContext-procMount")procMount denotes the type of proc mount to use for the containers. The default value is Default which uses the container runtime defaults for readonly paths and masked paths. Note that this field cannot be set when spec.os.name is windows. ##### `seccompProfile` object [​](#spec-versions-config-bash-securityContext-seccompProfile "Direct link to spec-versions-config-bash-securityContext-seccompProfile")The seccomp options to use by this container. If seccomp options are provided at both the pod & container level, the container options override the pod options. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-versions-config-bash-securityContext-seccompProfile-type "Direct link to spec-versions-config-bash-securityContext-seccompProfile-type")type indicates which kind of seccomp profile will be applied. Valid options are:Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. ##### `localhostProfile` string [​](#spec-versions-config-bash-securityContext-seccompProfile-localhostProfile "Direct link to spec-versions-config-bash-securityContext-seccompProfile-localhostProfile")localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. ##### `appArmorProfile` object [​](#spec-versions-config-bash-securityContext-appArmorProfile "Direct link to spec-versions-config-bash-securityContext-appArmorProfile")appArmorProfile is the AppArmor options to use by this container. If set, this profile overrides the pod's appArmorProfile. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-versions-config-bash-securityContext-appArmorProfile-type "Direct link to spec-versions-config-bash-securityContext-appArmorProfile-type")type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. ##### `localhostProfile` string [​](#spec-versions-config-bash-securityContext-appArmorProfile-localhostProfile "Direct link to spec-versions-config-bash-securityContext-appArmorProfile-localhostProfile")localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". ##### `values` string [​](#spec-versions-config-values "Direct link to spec-versions-config-values")Values is the set of extra Values added to the chart. These values merge with the default values inside of the chart. You can use golang templating in here with values from parameters. ##### `parameters` string [​](#spec-versions-config-parameters "Direct link to spec-versions-config-parameters")Parameters are additional helm chart values that will get merged with config and are then used to deploy the helm chart. ##### `annotations` object [​](#spec-versions-config-annotations "Direct link to spec-versions-config-annotations")Annotations are extra annotations for this helm release #### `wait` boolean [​](#spec-versions-wait "Direct link to spec-versions-wait")Wait determines if Loft should wait during deploy for the app to become ready #### `timeout` string [​](#spec-versions-timeout "Direct link to spec-versions-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) #### `parameters` object\[] [​](#spec-versions-parameters "Direct link to spec-versions-parameters")Parameters define additional app parameters that will set helm values ##### `variable` string [​](#spec-versions-parameters-variable "Direct link to spec-versions-parameters-variable")Variable is the path of the variable. Can be foo or foo.bar for nested objects. ##### `label` string [​](#spec-versions-parameters-label "Direct link to spec-versions-parameters-label")Label is the label to show for this parameter ##### `description` string [​](#spec-versions-parameters-description "Direct link to spec-versions-parameters-description")Description is the description to show for this parameter ##### `type` string [​](#spec-versions-parameters-type "Direct link to spec-versions-parameters-type")Type of the parameter. Can be one of: string, multiline, boolean, number and password ##### `options` string\[] [​](#spec-versions-parameters-options "Direct link to spec-versions-parameters-options")Options is a slice of strings, where each string represents a mutually exclusive choice. ##### `min` integer [​](#spec-versions-parameters-min "Direct link to spec-versions-parameters-min")Min is the minimum number if type is number ##### `max` integer [​](#spec-versions-parameters-max "Direct link to spec-versions-parameters-max")Max is the maximum number if type is number ##### `required` boolean [​](#spec-versions-parameters-required "Direct link to spec-versions-parameters-required")Required specifies if this parameter is required ##### `defaultValue` string [​](#spec-versions-parameters-defaultValue "Direct link to spec-versions-parameters-defaultValue")DefaultValue is the default value if none is specified ##### `placeholder` string [​](#spec-versions-parameters-placeholder "Direct link to spec-versions-parameters-placeholder")Placeholder shown in the UI ##### `invalidation` string [​](#spec-versions-parameters-invalidation "Direct link to spec-versions-parameters-invalidation")Invalidation regex that if matched will reject the input ##### `validation` string [​](#spec-versions-parameters-validation "Direct link to spec-versions-parameters-validation")Validation regex that if matched will allow the input ##### `section` string [​](#spec-versions-parameters-section "Direct link to spec-versions-parameters-section")Section where this app should be displayed. Apps with the same section name will be grouped together #### `streamContainer` object [​](#spec-versions-streamContainer "Direct link to spec-versions-streamContainer")DEPRECATED: Use config.bash instead StreamContainer can be used to stream a containers logs instead of the helm output. ##### `selector` object [​](#spec-versions-streamContainer-selector "Direct link to spec-versions-streamContainer-selector")Label selector for pods. The newest matching pod will be used to stream logs from ##### `matchLabels` object [​](#spec-versions-streamContainer-selector-matchLabels "Direct link to spec-versions-streamContainer-selector-matchLabels")matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. ##### `matchExpressions` object\[] [​](#spec-versions-streamContainer-selector-matchExpressions "Direct link to spec-versions-streamContainer-selector-matchExpressions")matchExpressions is a list of label selector requirements. The requirements are ANDed. ##### `key` required string [​](#spec-versions-streamContainer-selector-matchExpressions-key "Direct link to spec-versions-streamContainer-selector-matchExpressions-key")key is the label key that the selector applies to. ##### `operator` required string [​](#spec-versions-streamContainer-selector-matchExpressions-operator "Direct link to spec-versions-streamContainer-selector-matchExpressions-operator")operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. ##### `values` string\[] [​](#spec-versions-streamContainer-selector-matchExpressions-values "Direct link to spec-versions-streamContainer-selector-matchExpressions-values")values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. ##### `container` string [​](#spec-versions-streamContainer-container "Direct link to spec-versions-streamContainer-container")Container is the container name to use #### `version` string [​](#spec-versions-version "Direct link to spec-versions-version")Version is the version. Needs to be in X.X.X format. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ### `manifests` string [​](#spec-manifests "Direct link to spec-manifests")DEPRECATED: Use config instead manifest represents kubernetes resources that will be deployed into the target namespace ### `helm` object [​](#spec-helm "Direct link to spec-helm")DEPRECATED: Use config instead helm defines the configuration for a helm deployment #### `name` required string [​](#spec-helm-name "Direct link to spec-helm-name")Name of the chart to deploy #### `values` string [​](#spec-helm-values "Direct link to spec-helm-values")The additional helm values to use. Expected block string #### `version` string [​](#spec-helm-version "Direct link to spec-helm-version")Version is the version of the chart to deploy #### `repoUrl` string [​](#spec-helm-repoUrl "Direct link to spec-helm-repoUrl")The repo url to use #### `username` string [​](#spec-helm-username "Direct link to spec-helm-username")The username to use for the selected repository #### `password` string [​](#spec-helm-password "Direct link to spec-helm-password")The password to use for the selected repository #### `insecure` boolean [​](#spec-helm-insecure "Direct link to spec-helm-insecure")Determines if the remote location uses an insecure TLS certificate. ## `status` object [​](#status "Direct link to status") ## Retrieve: Apps[​](#retrieve-apps "Direct link to Retrieve: Apps") You can either use curl or kubectl to retrieve Apps. * kubectl * curl ### Retrieve a list of Apps[​](#retrieve-a-list-of-apps "Direct link to Retrieve a list of Apps") Run the following command to list all Apps: ``` kubectl get apps.management.loft.sh -o yaml ``` ### Retrieve a single App by name[​](#retrieve-a-single-app-by-name "Direct link to Retrieve a single App by name") Run the following kubectl command to get App `my-app`: ``` kubectl get apps.management.loft.sh my-app -o yaml ``` ### Retrieve a list of Apps[​](#retrieve-a-list-of-apps-1 "Direct link to Retrieve a list of Apps") Run the following curl command to list all Apps: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/apps" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single App by name[​](#get-a-single-app-by-name "Direct link to Get a single App by name") Run the following curl command to get App `my-app`: ``` # Exchange my-app in the url below with the name of the App curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/apps/my-app" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: App[​](#create-app "Direct link to Create: App") You can either use curl or kubectl to create a new App. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: App metadata: name: my-app spec: access: - users: - '*' verbs: - get config: chart: name: argo-cd repoURL: https://argoproj.github.io/argo-helm description: Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes displayName: ArgoCD icon: https://argo-cd.readthedocs.io/en/stable/assets/logo.png recommendedApp: - cluster status: {} ``` Then create the App `my-app` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: App metadata: name: my-app spec: access: - users: - '*' verbs: - get config: chart: name: argo-cd repoURL: https://argoproj.github.io/argo-helm description: Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes displayName: ArgoCD icon: https://argo-cd.readthedocs.io/en/stable/assets/logo.png recommendedApp: - cluster status: {} ``` Run the following curl command to create a new App `my-app`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/apps" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: App[​](#update-app "Direct link to Update: App") You can either use curl or kubectl to update Apps. * kubectl * curl ### Update App[​](#update-app "Direct link to Update App") Run the following command to update App `my-app`: ``` kubectl edit apps.management.loft.sh my-app ``` Then edit the object and upon save, kubectl will update the resource. ### Patch App[​](#patch-app "Direct link to Patch App") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the App `my-app` via a patch: ``` kubectl patch apps.management.loft.sh my-app \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update App[​](#update-app-1 "Direct link to Update App") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: App metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-app resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: access: - users: - '*' verbs: - get config: chart: name: argo-cd repoURL: https://argoproj.github.io/argo-helm description: Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes displayName: ArgoCD icon: https://argo-cd.readthedocs.io/en/stable/assets/logo.png recommendedApp: - cluster status: {} ``` Run the following curl command to update a single App `my-app`: ``` # Replace the my-app in the url below with the name of the App you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/apps/my-app" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch App[​](#patch-app-1 "Direct link to Patch App") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the App `my-app` via a patch: ``` # Replace the my-app in the url below with the name of the App you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/apps/my-app" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: App[​](#delete-app "Direct link to Delete: App") You can either use curl or kubectl to delete Apps. * kubectl * curl Run the following command to delete App `my-app`: ``` kubectl delete apps.management.loft.sh my-app ``` Run the following curl command to delete App `my-app`: ``` # Replace the my-app in the url below with the name of the App you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/apps/my-app" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # ArgoCD Application ArgoCDApplication declares an Argo CD Application that the platform reconciles and syncs to a connected Argo CD instance. ## ArgoCDApplication example[​](#argocdapplication-example "Direct link to ArgoCDApplication example") An example ArgoCDApplication: ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplication metadata: name: shared-infra namespace: p-my-project spec: destination: cluster: name: loft-cluster displayName: Shared Infrastructure templateRef: name: infra-template status: {} ``` ## ArgoCDApplication reference[​](#argocdapplication-reference "Direct link to ArgoCDApplication reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes the Argo CD application ### `templateRef` object [​](#spec-templateRef "Direct link to spec-templateRef")TemplateRef holds the Argo CD application template reference #### `name` string [​](#spec-templateRef-name "Direct link to spec-templateRef-name")Name holds the name of the blueprint template to reference. ### `template` object [​](#spec-template "Direct link to spec-template")Template is the Argo CD application template definition #### `metadata` object [​](#spec-template-metadata "Direct link to spec-template-metadata") ##### `labels` object [​](#spec-template-metadata-labels "Direct link to spec-template-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-template-metadata-annotations "Direct link to spec-template-metadata-annotations")Annotations are annotations on the object #### `spec` object [​](#spec-template-spec "Direct link to spec-template-spec")Spec is the spec of the Argo CD application ##### `source` object [​](#spec-template-spec-source "Direct link to spec-template-spec-source") ##### `repoURL` required string [​](#spec-template-spec-source-repoURL "Direct link to spec-template-spec-source-repoURL") ##### `path` string [​](#spec-template-spec-source-path "Direct link to spec-template-spec-source-path") ##### `targetRevision` string [​](#spec-template-spec-source-targetRevision "Direct link to spec-template-spec-source-targetRevision") ##### `helm` object [​](#spec-template-spec-source-helm "Direct link to spec-template-spec-source-helm") ##### `valueFiles` string\[] [​](#spec-template-spec-source-helm-valueFiles "Direct link to spec-template-spec-source-helm-valueFiles") ##### `parameters` object\[] [​](#spec-template-spec-source-helm-parameters "Direct link to spec-template-spec-source-helm-parameters") ##### `name` string [​](#spec-template-spec-source-helm-parameters-name "Direct link to spec-template-spec-source-helm-parameters-name") ##### `value` string [​](#spec-template-spec-source-helm-parameters-value "Direct link to spec-template-spec-source-helm-parameters-value") ##### `forceString` boolean [​](#spec-template-spec-source-helm-parameters-forceString "Direct link to spec-template-spec-source-helm-parameters-forceString") ##### `releaseName` string [​](#spec-template-spec-source-helm-releaseName "Direct link to spec-template-spec-source-helm-releaseName") ##### `values` string [​](#spec-template-spec-source-helm-values "Direct link to spec-template-spec-source-helm-values") ##### `fileParameters` object\[] [​](#spec-template-spec-source-helm-fileParameters "Direct link to spec-template-spec-source-helm-fileParameters") ##### `name` string [​](#spec-template-spec-source-helm-fileParameters-name "Direct link to spec-template-spec-source-helm-fileParameters-name") ##### `path` string [​](#spec-template-spec-source-helm-fileParameters-path "Direct link to spec-template-spec-source-helm-fileParameters-path") ##### `version` string [​](#spec-template-spec-source-helm-version "Direct link to spec-template-spec-source-helm-version") ##### `passCredentials` boolean [​](#spec-template-spec-source-helm-passCredentials "Direct link to spec-template-spec-source-helm-passCredentials") ##### `ignoreMissingValueFiles` boolean [​](#spec-template-spec-source-helm-ignoreMissingValueFiles "Direct link to spec-template-spec-source-helm-ignoreMissingValueFiles") ##### `skipCrds` boolean [​](#spec-template-spec-source-helm-skipCrds "Direct link to spec-template-spec-source-helm-skipCrds") ##### `valuesObject` object [​](#spec-template-spec-source-helm-valuesObject "Direct link to spec-template-spec-source-helm-valuesObject") ##### `namespace` string [​](#spec-template-spec-source-helm-namespace "Direct link to spec-template-spec-source-helm-namespace") ##### `kubeVersion` string [​](#spec-template-spec-source-helm-kubeVersion "Direct link to spec-template-spec-source-helm-kubeVersion") ##### `apiVersions` string\[] [​](#spec-template-spec-source-helm-apiVersions "Direct link to spec-template-spec-source-helm-apiVersions") ##### `skipTests` boolean [​](#spec-template-spec-source-helm-skipTests "Direct link to spec-template-spec-source-helm-skipTests") ##### `skipSchemaValidation` boolean [​](#spec-template-spec-source-helm-skipSchemaValidation "Direct link to spec-template-spec-source-helm-skipSchemaValidation") ##### `kustomize` object [​](#spec-template-spec-source-kustomize "Direct link to spec-template-spec-source-kustomize") ##### `namePrefix` string [​](#spec-template-spec-source-kustomize-namePrefix "Direct link to spec-template-spec-source-kustomize-namePrefix") ##### `nameSuffix` string [​](#spec-template-spec-source-kustomize-nameSuffix "Direct link to spec-template-spec-source-kustomize-nameSuffix") ##### `images` object [​](#spec-template-spec-source-kustomize-images "Direct link to spec-template-spec-source-kustomize-images") ##### `commonLabels` object [​](#spec-template-spec-source-kustomize-commonLabels "Direct link to spec-template-spec-source-kustomize-commonLabels") ##### `version` string [​](#spec-template-spec-source-kustomize-version "Direct link to spec-template-spec-source-kustomize-version") ##### `commonAnnotations` object [​](#spec-template-spec-source-kustomize-commonAnnotations "Direct link to spec-template-spec-source-kustomize-commonAnnotations") ##### `forceCommonLabels` boolean [​](#spec-template-spec-source-kustomize-forceCommonLabels "Direct link to spec-template-spec-source-kustomize-forceCommonLabels") ##### `forceCommonAnnotations` boolean [​](#spec-template-spec-source-kustomize-forceCommonAnnotations "Direct link to spec-template-spec-source-kustomize-forceCommonAnnotations") ##### `namespace` string [​](#spec-template-spec-source-kustomize-namespace "Direct link to spec-template-spec-source-kustomize-namespace") ##### `commonAnnotationsEnvsubst` boolean [​](#spec-template-spec-source-kustomize-commonAnnotationsEnvsubst "Direct link to spec-template-spec-source-kustomize-commonAnnotationsEnvsubst") ##### `replicas` object [​](#spec-template-spec-source-kustomize-replicas "Direct link to spec-template-spec-source-kustomize-replicas") ##### `patches` object [​](#spec-template-spec-source-kustomize-patches "Direct link to spec-template-spec-source-kustomize-patches") ##### `components` string\[] [​](#spec-template-spec-source-kustomize-components "Direct link to spec-template-spec-source-kustomize-components") ##### `ignoreMissingComponents` boolean [​](#spec-template-spec-source-kustomize-ignoreMissingComponents "Direct link to spec-template-spec-source-kustomize-ignoreMissingComponents") ##### `labelWithoutSelector` boolean [​](#spec-template-spec-source-kustomize-labelWithoutSelector "Direct link to spec-template-spec-source-kustomize-labelWithoutSelector") ##### `kubeVersion` string [​](#spec-template-spec-source-kustomize-kubeVersion "Direct link to spec-template-spec-source-kustomize-kubeVersion") ##### `apiVersions` string\[] [​](#spec-template-spec-source-kustomize-apiVersions "Direct link to spec-template-spec-source-kustomize-apiVersions") ##### `labelIncludeTemplates` boolean [​](#spec-template-spec-source-kustomize-labelIncludeTemplates "Direct link to spec-template-spec-source-kustomize-labelIncludeTemplates") ##### `directory` object [​](#spec-template-spec-source-directory "Direct link to spec-template-spec-source-directory") ##### `recurse` boolean [​](#spec-template-spec-source-directory-recurse "Direct link to spec-template-spec-source-directory-recurse") ##### `jsonnet` object [​](#spec-template-spec-source-directory-jsonnet "Direct link to spec-template-spec-source-directory-jsonnet") ##### `extVars` object\[] [​](#spec-template-spec-source-directory-jsonnet-extVars "Direct link to spec-template-spec-source-directory-jsonnet-extVars") ##### `name` required string [​](#spec-template-spec-source-directory-jsonnet-extVars-name "Direct link to spec-template-spec-source-directory-jsonnet-extVars-name") ##### `value` required string [​](#spec-template-spec-source-directory-jsonnet-extVars-value "Direct link to spec-template-spec-source-directory-jsonnet-extVars-value") ##### `code` boolean [​](#spec-template-spec-source-directory-jsonnet-extVars-code "Direct link to spec-template-spec-source-directory-jsonnet-extVars-code") ##### `tlas` object\[] [​](#spec-template-spec-source-directory-jsonnet-tlas "Direct link to spec-template-spec-source-directory-jsonnet-tlas") ##### `name` required string [​](#spec-template-spec-source-directory-jsonnet-tlas-name "Direct link to spec-template-spec-source-directory-jsonnet-tlas-name") ##### `value` required string [​](#spec-template-spec-source-directory-jsonnet-tlas-value "Direct link to spec-template-spec-source-directory-jsonnet-tlas-value") ##### `code` boolean [​](#spec-template-spec-source-directory-jsonnet-tlas-code "Direct link to spec-template-spec-source-directory-jsonnet-tlas-code") ##### `libs` string\[] [​](#spec-template-spec-source-directory-jsonnet-libs "Direct link to spec-template-spec-source-directory-jsonnet-libs") ##### `exclude` string [​](#spec-template-spec-source-directory-exclude "Direct link to spec-template-spec-source-directory-exclude") ##### `include` string [​](#spec-template-spec-source-directory-include "Direct link to spec-template-spec-source-directory-include") ##### `plugin` object [​](#spec-template-spec-source-plugin "Direct link to spec-template-spec-source-plugin") ##### `name` string [​](#spec-template-spec-source-plugin-name "Direct link to spec-template-spec-source-plugin-name") ##### `env` object [​](#spec-template-spec-source-plugin-env "Direct link to spec-template-spec-source-plugin-env") ##### `parameters` object [​](#spec-template-spec-source-plugin-parameters "Direct link to spec-template-spec-source-plugin-parameters") ##### `chart` string [​](#spec-template-spec-source-chart "Direct link to spec-template-spec-source-chart") ##### `ref` string [​](#spec-template-spec-source-ref "Direct link to spec-template-spec-source-ref") ##### `name` string [​](#spec-template-spec-source-name "Direct link to spec-template-spec-source-name") ##### `destination` required object [​](#spec-template-spec-destination "Direct link to spec-template-spec-destination") ##### `server` string [​](#spec-template-spec-destination-server "Direct link to spec-template-spec-destination-server") ##### `namespace` string [​](#spec-template-spec-destination-namespace "Direct link to spec-template-spec-destination-namespace") ##### `name` string [​](#spec-template-spec-destination-name "Direct link to spec-template-spec-destination-name") ##### `project` required string [​](#spec-template-spec-project "Direct link to spec-template-spec-project") ##### `syncPolicy` object [​](#spec-template-spec-syncPolicy "Direct link to spec-template-spec-syncPolicy") ##### `automated` object [​](#spec-template-spec-syncPolicy-automated "Direct link to spec-template-spec-syncPolicy-automated") ##### `prune` boolean [​](#spec-template-spec-syncPolicy-automated-prune "Direct link to spec-template-spec-syncPolicy-automated-prune") ##### `selfHeal` boolean [​](#spec-template-spec-syncPolicy-automated-selfHeal "Direct link to spec-template-spec-syncPolicy-automated-selfHeal") ##### `allowEmpty` boolean [​](#spec-template-spec-syncPolicy-automated-allowEmpty "Direct link to spec-template-spec-syncPolicy-automated-allowEmpty") ##### `enabled` boolean [​](#spec-template-spec-syncPolicy-automated-enabled "Direct link to spec-template-spec-syncPolicy-automated-enabled") ##### `syncOptions` object [​](#spec-template-spec-syncPolicy-syncOptions "Direct link to spec-template-spec-syncPolicy-syncOptions") ##### `retry` object [​](#spec-template-spec-syncPolicy-retry "Direct link to spec-template-spec-syncPolicy-retry") ##### `limit` integer [​](#spec-template-spec-syncPolicy-retry-limit "Direct link to spec-template-spec-syncPolicy-retry-limit") ##### `backoff` object [​](#spec-template-spec-syncPolicy-retry-backoff "Direct link to spec-template-spec-syncPolicy-retry-backoff") ##### `duration` string [​](#spec-template-spec-syncPolicy-retry-backoff-duration "Direct link to spec-template-spec-syncPolicy-retry-backoff-duration") ##### `factor` integer [​](#spec-template-spec-syncPolicy-retry-backoff-factor "Direct link to spec-template-spec-syncPolicy-retry-backoff-factor") ##### `maxDuration` string [​](#spec-template-spec-syncPolicy-retry-backoff-maxDuration "Direct link to spec-template-spec-syncPolicy-retry-backoff-maxDuration") ##### `refresh` boolean [​](#spec-template-spec-syncPolicy-retry-refresh "Direct link to spec-template-spec-syncPolicy-retry-refresh") ##### `managedNamespaceMetadata` object [​](#spec-template-spec-syncPolicy-managedNamespaceMetadata "Direct link to spec-template-spec-syncPolicy-managedNamespaceMetadata") ##### `labels` object [​](#spec-template-spec-syncPolicy-managedNamespaceMetadata-labels "Direct link to spec-template-spec-syncPolicy-managedNamespaceMetadata-labels") ##### `annotations` object [​](#spec-template-spec-syncPolicy-managedNamespaceMetadata-annotations "Direct link to spec-template-spec-syncPolicy-managedNamespaceMetadata-annotations") ##### `ignoreDifferences` object [​](#spec-template-spec-ignoreDifferences "Direct link to spec-template-spec-ignoreDifferences") ##### `info` object\[] [​](#spec-template-spec-info "Direct link to spec-template-spec-info") ##### `name` required string [​](#spec-template-spec-info-name "Direct link to spec-template-spec-info-name") ##### `value` required string [​](#spec-template-spec-info-value "Direct link to spec-template-spec-info-value") ##### `revisionHistoryLimit` integer [​](#spec-template-spec-revisionHistoryLimit "Direct link to spec-template-spec-revisionHistoryLimit") ##### `sources` object [​](#spec-template-spec-sources "Direct link to spec-template-spec-sources") ##### `sourceHydrator` object [​](#spec-template-spec-sourceHydrator "Direct link to spec-template-spec-sourceHydrator") ##### `drySource` required object [​](#spec-template-spec-sourceHydrator-drySource "Direct link to spec-template-spec-sourceHydrator-drySource") ##### `repoURL` required string [​](#spec-template-spec-sourceHydrator-drySource-repoURL "Direct link to spec-template-spec-sourceHydrator-drySource-repoURL") ##### `targetRevision` required string [​](#spec-template-spec-sourceHydrator-drySource-targetRevision "Direct link to spec-template-spec-sourceHydrator-drySource-targetRevision") ##### `path` required string [​](#spec-template-spec-sourceHydrator-drySource-path "Direct link to spec-template-spec-sourceHydrator-drySource-path") ##### `helm` object [​](#spec-template-spec-sourceHydrator-drySource-helm "Direct link to spec-template-spec-sourceHydrator-drySource-helm") ##### `valueFiles` string\[] [​](#spec-template-spec-sourceHydrator-drySource-helm-valueFiles "Direct link to spec-template-spec-sourceHydrator-drySource-helm-valueFiles") ##### `parameters` object\[] [​](#spec-template-spec-sourceHydrator-drySource-helm-parameters "Direct link to spec-template-spec-sourceHydrator-drySource-helm-parameters") ##### `name` string [​](#spec-template-spec-sourceHydrator-drySource-helm-parameters-name "Direct link to spec-template-spec-sourceHydrator-drySource-helm-parameters-name") ##### `value` string [​](#spec-template-spec-sourceHydrator-drySource-helm-parameters-value "Direct link to spec-template-spec-sourceHydrator-drySource-helm-parameters-value") ##### `forceString` boolean [​](#spec-template-spec-sourceHydrator-drySource-helm-parameters-forceString "Direct link to spec-template-spec-sourceHydrator-drySource-helm-parameters-forceString") ##### `releaseName` string [​](#spec-template-spec-sourceHydrator-drySource-helm-releaseName "Direct link to spec-template-spec-sourceHydrator-drySource-helm-releaseName") ##### `values` string [​](#spec-template-spec-sourceHydrator-drySource-helm-values "Direct link to spec-template-spec-sourceHydrator-drySource-helm-values") ##### `fileParameters` object\[] [​](#spec-template-spec-sourceHydrator-drySource-helm-fileParameters "Direct link to spec-template-spec-sourceHydrator-drySource-helm-fileParameters") ##### `name` string [​](#spec-template-spec-sourceHydrator-drySource-helm-fileParameters-name "Direct link to spec-template-spec-sourceHydrator-drySource-helm-fileParameters-name") ##### `path` string [​](#spec-template-spec-sourceHydrator-drySource-helm-fileParameters-path "Direct link to spec-template-spec-sourceHydrator-drySource-helm-fileParameters-path") ##### `version` string [​](#spec-template-spec-sourceHydrator-drySource-helm-version "Direct link to spec-template-spec-sourceHydrator-drySource-helm-version") ##### `passCredentials` boolean [​](#spec-template-spec-sourceHydrator-drySource-helm-passCredentials "Direct link to spec-template-spec-sourceHydrator-drySource-helm-passCredentials") ##### `ignoreMissingValueFiles` boolean [​](#spec-template-spec-sourceHydrator-drySource-helm-ignoreMissingValueFiles "Direct link to spec-template-spec-sourceHydrator-drySource-helm-ignoreMissingValueFiles") ##### `skipCrds` boolean [​](#spec-template-spec-sourceHydrator-drySource-helm-skipCrds "Direct link to spec-template-spec-sourceHydrator-drySource-helm-skipCrds") ##### `valuesObject` object [​](#spec-template-spec-sourceHydrator-drySource-helm-valuesObject "Direct link to spec-template-spec-sourceHydrator-drySource-helm-valuesObject") ##### `namespace` string [​](#spec-template-spec-sourceHydrator-drySource-helm-namespace "Direct link to spec-template-spec-sourceHydrator-drySource-helm-namespace") ##### `kubeVersion` string [​](#spec-template-spec-sourceHydrator-drySource-helm-kubeVersion "Direct link to spec-template-spec-sourceHydrator-drySource-helm-kubeVersion") ##### `apiVersions` string\[] [​](#spec-template-spec-sourceHydrator-drySource-helm-apiVersions "Direct link to spec-template-spec-sourceHydrator-drySource-helm-apiVersions") ##### `skipTests` boolean [​](#spec-template-spec-sourceHydrator-drySource-helm-skipTests "Direct link to spec-template-spec-sourceHydrator-drySource-helm-skipTests") ##### `skipSchemaValidation` boolean [​](#spec-template-spec-sourceHydrator-drySource-helm-skipSchemaValidation "Direct link to spec-template-spec-sourceHydrator-drySource-helm-skipSchemaValidation") ##### `kustomize` object [​](#spec-template-spec-sourceHydrator-drySource-kustomize "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize") ##### `namePrefix` string [​](#spec-template-spec-sourceHydrator-drySource-kustomize-namePrefix "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-namePrefix") ##### `nameSuffix` string [​](#spec-template-spec-sourceHydrator-drySource-kustomize-nameSuffix "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-nameSuffix") ##### `images` object [​](#spec-template-spec-sourceHydrator-drySource-kustomize-images "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-images") ##### `commonLabels` object [​](#spec-template-spec-sourceHydrator-drySource-kustomize-commonLabels "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-commonLabels") ##### `version` string [​](#spec-template-spec-sourceHydrator-drySource-kustomize-version "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-version") ##### `commonAnnotations` object [​](#spec-template-spec-sourceHydrator-drySource-kustomize-commonAnnotations "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-commonAnnotations") ##### `forceCommonLabels` boolean [​](#spec-template-spec-sourceHydrator-drySource-kustomize-forceCommonLabels "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-forceCommonLabels") ##### `forceCommonAnnotations` boolean [​](#spec-template-spec-sourceHydrator-drySource-kustomize-forceCommonAnnotations "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-forceCommonAnnotations") ##### `namespace` string [​](#spec-template-spec-sourceHydrator-drySource-kustomize-namespace "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-namespace") ##### `commonAnnotationsEnvsubst` boolean [​](#spec-template-spec-sourceHydrator-drySource-kustomize-commonAnnotationsEnvsubst "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-commonAnnotationsEnvsubst") ##### `replicas` object [​](#spec-template-spec-sourceHydrator-drySource-kustomize-replicas "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-replicas") ##### `patches` object [​](#spec-template-spec-sourceHydrator-drySource-kustomize-patches "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-patches") ##### `components` string\[] [​](#spec-template-spec-sourceHydrator-drySource-kustomize-components "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-components") ##### `ignoreMissingComponents` boolean [​](#spec-template-spec-sourceHydrator-drySource-kustomize-ignoreMissingComponents "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-ignoreMissingComponents") ##### `labelWithoutSelector` boolean [​](#spec-template-spec-sourceHydrator-drySource-kustomize-labelWithoutSelector "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-labelWithoutSelector") ##### `kubeVersion` string [​](#spec-template-spec-sourceHydrator-drySource-kustomize-kubeVersion "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-kubeVersion") ##### `apiVersions` string\[] [​](#spec-template-spec-sourceHydrator-drySource-kustomize-apiVersions "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-apiVersions") ##### `labelIncludeTemplates` boolean [​](#spec-template-spec-sourceHydrator-drySource-kustomize-labelIncludeTemplates "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-labelIncludeTemplates") ##### `directory` object [​](#spec-template-spec-sourceHydrator-drySource-directory "Direct link to spec-template-spec-sourceHydrator-drySource-directory") ##### `recurse` boolean [​](#spec-template-spec-sourceHydrator-drySource-directory-recurse "Direct link to spec-template-spec-sourceHydrator-drySource-directory-recurse") ##### `jsonnet` object [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet") ##### `extVars` object\[] [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars") ##### `name` required string [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-name "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-name") ##### `value` required string [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-value "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-value") ##### `code` boolean [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-code "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-code") ##### `tlas` object\[] [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas") ##### `name` required string [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-name "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-name") ##### `value` required string [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-value "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-value") ##### `code` boolean [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-code "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-code") ##### `libs` string\[] [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-libs "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-libs") ##### `exclude` string [​](#spec-template-spec-sourceHydrator-drySource-directory-exclude "Direct link to spec-template-spec-sourceHydrator-drySource-directory-exclude") ##### `include` string [​](#spec-template-spec-sourceHydrator-drySource-directory-include "Direct link to spec-template-spec-sourceHydrator-drySource-directory-include") ##### `plugin` object [​](#spec-template-spec-sourceHydrator-drySource-plugin "Direct link to spec-template-spec-sourceHydrator-drySource-plugin") ##### `name` string [​](#spec-template-spec-sourceHydrator-drySource-plugin-name "Direct link to spec-template-spec-sourceHydrator-drySource-plugin-name") ##### `env` object [​](#spec-template-spec-sourceHydrator-drySource-plugin-env "Direct link to spec-template-spec-sourceHydrator-drySource-plugin-env") ##### `parameters` object [​](#spec-template-spec-sourceHydrator-drySource-plugin-parameters "Direct link to spec-template-spec-sourceHydrator-drySource-plugin-parameters") ##### `syncSource` required object [​](#spec-template-spec-sourceHydrator-syncSource "Direct link to spec-template-spec-sourceHydrator-syncSource") ##### `targetBranch` required string [​](#spec-template-spec-sourceHydrator-syncSource-targetBranch "Direct link to spec-template-spec-sourceHydrator-syncSource-targetBranch") ##### `path` required string [​](#spec-template-spec-sourceHydrator-syncSource-path "Direct link to spec-template-spec-sourceHydrator-syncSource-path") ##### `hydrateTo` object [​](#spec-template-spec-sourceHydrator-hydrateTo "Direct link to spec-template-spec-sourceHydrator-hydrateTo") ##### `targetBranch` required string [​](#spec-template-spec-sourceHydrator-hydrateTo-targetBranch "Direct link to spec-template-spec-sourceHydrator-hydrateTo-targetBranch") ### `parameters` object [​](#spec-parameters "Direct link to spec-parameters")Parameters are values to pass to the template. The values should be encoded as YAML string where each parameter is represented as a top-level field key. ### `destination` object [​](#spec-destination "Direct link to spec-destination")Destination holds the Argo CD target reference #### `virtualCluster` object [​](#spec-destination-virtualCluster "Direct link to spec-destination-virtualCluster")VirtualCluster name. Mutually exclusive with Cluster. ##### `name` string [​](#spec-destination-virtualCluster-name "Direct link to spec-destination-virtualCluster-name")Name of the tenant cluster ##### `target` string [​](#spec-destination-virtualCluster-target "Direct link to spec-destination-virtualCluster-target")Target of the tenant cluster #### `cluster` object [​](#spec-destination-cluster "Direct link to spec-destination-cluster")Cluster name. Mutually exclusive with VirtualCluster. ##### `name` string [​](#spec-destination-cluster-name "Direct link to spec-destination-cluster-name")Name of the cluster ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ## `status` object [​](#status "Direct link to status") ### `conditions` object [​](#status-conditions "Direct link to status-conditions")Conditions holds several conditions the tenant cluster might be in ### `host` string [​](#status-host "Direct link to status-host")Host of the Argo CD server ### `application` object [​](#status-application "Direct link to status-application")Application holds the status of the Argo CD application #### `resources` object\[] [​](#status-application-resources "Direct link to status-application-resources") ##### `group` string [​](#status-application-resources-group "Direct link to status-application-resources-group") ##### `version` string [​](#status-application-resources-version "Direct link to status-application-resources-version") ##### `kind` string [​](#status-application-resources-kind "Direct link to status-application-resources-kind") ##### `namespace` string [​](#status-application-resources-namespace "Direct link to status-application-resources-namespace") ##### `name` string [​](#status-application-resources-name "Direct link to status-application-resources-name") ##### `status` string [​](#status-application-resources-status "Direct link to status-application-resources-status") ##### `health` object [​](#status-application-resources-health "Direct link to status-application-resources-health") ##### `status` string [​](#status-application-resources-health-status "Direct link to status-application-resources-health-status") ##### `message` string [​](#status-application-resources-health-message "Direct link to status-application-resources-health-message") ##### `lastTransitionTime` object [​](#status-application-resources-health-lastTransitionTime "Direct link to status-application-resources-health-lastTransitionTime") ##### `hook` boolean [​](#status-application-resources-hook "Direct link to status-application-resources-hook") ##### `requiresPruning` boolean [​](#status-application-resources-requiresPruning "Direct link to status-application-resources-requiresPruning") ##### `syncWave` integer [​](#status-application-resources-syncWave "Direct link to status-application-resources-syncWave") ##### `requiresDeletionConfirmation` boolean [​](#status-application-resources-requiresDeletionConfirmation "Direct link to status-application-resources-requiresDeletionConfirmation") #### `sync` object [​](#status-application-sync "Direct link to status-application-sync") ##### `status` required string [​](#status-application-sync-status "Direct link to status-application-sync-status") ##### `comparedTo` object [​](#status-application-sync-comparedTo "Direct link to status-application-sync-comparedTo") ##### `source` object [​](#status-application-sync-comparedTo-source "Direct link to status-application-sync-comparedTo-source") ##### `repoURL` required string [​](#status-application-sync-comparedTo-source-repoURL "Direct link to status-application-sync-comparedTo-source-repoURL") ##### `path` string [​](#status-application-sync-comparedTo-source-path "Direct link to status-application-sync-comparedTo-source-path") ##### `targetRevision` string [​](#status-application-sync-comparedTo-source-targetRevision "Direct link to status-application-sync-comparedTo-source-targetRevision") ##### `helm` object [​](#status-application-sync-comparedTo-source-helm "Direct link to status-application-sync-comparedTo-source-helm") ##### `valueFiles` string\[] [​](#status-application-sync-comparedTo-source-helm-valueFiles "Direct link to status-application-sync-comparedTo-source-helm-valueFiles") ##### `parameters` object\[] [​](#status-application-sync-comparedTo-source-helm-parameters "Direct link to status-application-sync-comparedTo-source-helm-parameters") ##### `name` string [​](#status-application-sync-comparedTo-source-helm-parameters-name "Direct link to status-application-sync-comparedTo-source-helm-parameters-name") ##### `value` string [​](#status-application-sync-comparedTo-source-helm-parameters-value "Direct link to status-application-sync-comparedTo-source-helm-parameters-value") ##### `forceString` boolean [​](#status-application-sync-comparedTo-source-helm-parameters-forceString "Direct link to status-application-sync-comparedTo-source-helm-parameters-forceString") ##### `releaseName` string [​](#status-application-sync-comparedTo-source-helm-releaseName "Direct link to status-application-sync-comparedTo-source-helm-releaseName") ##### `values` string [​](#status-application-sync-comparedTo-source-helm-values "Direct link to status-application-sync-comparedTo-source-helm-values") ##### `fileParameters` object\[] [​](#status-application-sync-comparedTo-source-helm-fileParameters "Direct link to status-application-sync-comparedTo-source-helm-fileParameters") ##### `name` string [​](#status-application-sync-comparedTo-source-helm-fileParameters-name "Direct link to status-application-sync-comparedTo-source-helm-fileParameters-name") ##### `path` string [​](#status-application-sync-comparedTo-source-helm-fileParameters-path "Direct link to status-application-sync-comparedTo-source-helm-fileParameters-path") ##### `version` string [​](#status-application-sync-comparedTo-source-helm-version "Direct link to status-application-sync-comparedTo-source-helm-version") ##### `passCredentials` boolean [​](#status-application-sync-comparedTo-source-helm-passCredentials "Direct link to status-application-sync-comparedTo-source-helm-passCredentials") ##### `ignoreMissingValueFiles` boolean [​](#status-application-sync-comparedTo-source-helm-ignoreMissingValueFiles "Direct link to status-application-sync-comparedTo-source-helm-ignoreMissingValueFiles") ##### `skipCrds` boolean [​](#status-application-sync-comparedTo-source-helm-skipCrds "Direct link to status-application-sync-comparedTo-source-helm-skipCrds") ##### `valuesObject` object [​](#status-application-sync-comparedTo-source-helm-valuesObject "Direct link to status-application-sync-comparedTo-source-helm-valuesObject") ##### `namespace` string [​](#status-application-sync-comparedTo-source-helm-namespace "Direct link to status-application-sync-comparedTo-source-helm-namespace") ##### `kubeVersion` string [​](#status-application-sync-comparedTo-source-helm-kubeVersion "Direct link to status-application-sync-comparedTo-source-helm-kubeVersion") ##### `apiVersions` string\[] [​](#status-application-sync-comparedTo-source-helm-apiVersions "Direct link to status-application-sync-comparedTo-source-helm-apiVersions") ##### `skipTests` boolean [​](#status-application-sync-comparedTo-source-helm-skipTests "Direct link to status-application-sync-comparedTo-source-helm-skipTests") ##### `skipSchemaValidation` boolean [​](#status-application-sync-comparedTo-source-helm-skipSchemaValidation "Direct link to status-application-sync-comparedTo-source-helm-skipSchemaValidation") ##### `kustomize` object [​](#status-application-sync-comparedTo-source-kustomize "Direct link to status-application-sync-comparedTo-source-kustomize") ##### `namePrefix` string [​](#status-application-sync-comparedTo-source-kustomize-namePrefix "Direct link to status-application-sync-comparedTo-source-kustomize-namePrefix") ##### `nameSuffix` string [​](#status-application-sync-comparedTo-source-kustomize-nameSuffix "Direct link to status-application-sync-comparedTo-source-kustomize-nameSuffix") ##### `images` object [​](#status-application-sync-comparedTo-source-kustomize-images "Direct link to status-application-sync-comparedTo-source-kustomize-images") ##### `commonLabels` object [​](#status-application-sync-comparedTo-source-kustomize-commonLabels "Direct link to status-application-sync-comparedTo-source-kustomize-commonLabels") ##### `version` string [​](#status-application-sync-comparedTo-source-kustomize-version "Direct link to status-application-sync-comparedTo-source-kustomize-version") ##### `commonAnnotations` object [​](#status-application-sync-comparedTo-source-kustomize-commonAnnotations "Direct link to status-application-sync-comparedTo-source-kustomize-commonAnnotations") ##### `forceCommonLabels` boolean [​](#status-application-sync-comparedTo-source-kustomize-forceCommonLabels "Direct link to status-application-sync-comparedTo-source-kustomize-forceCommonLabels") ##### `forceCommonAnnotations` boolean [​](#status-application-sync-comparedTo-source-kustomize-forceCommonAnnotations "Direct link to status-application-sync-comparedTo-source-kustomize-forceCommonAnnotations") ##### `namespace` string [​](#status-application-sync-comparedTo-source-kustomize-namespace "Direct link to status-application-sync-comparedTo-source-kustomize-namespace") ##### `commonAnnotationsEnvsubst` boolean [​](#status-application-sync-comparedTo-source-kustomize-commonAnnotationsEnvsubst "Direct link to status-application-sync-comparedTo-source-kustomize-commonAnnotationsEnvsubst") ##### `replicas` object [​](#status-application-sync-comparedTo-source-kustomize-replicas "Direct link to status-application-sync-comparedTo-source-kustomize-replicas") ##### `patches` object [​](#status-application-sync-comparedTo-source-kustomize-patches "Direct link to status-application-sync-comparedTo-source-kustomize-patches") ##### `components` string\[] [​](#status-application-sync-comparedTo-source-kustomize-components "Direct link to status-application-sync-comparedTo-source-kustomize-components") ##### `ignoreMissingComponents` boolean [​](#status-application-sync-comparedTo-source-kustomize-ignoreMissingComponents "Direct link to status-application-sync-comparedTo-source-kustomize-ignoreMissingComponents") ##### `labelWithoutSelector` boolean [​](#status-application-sync-comparedTo-source-kustomize-labelWithoutSelector "Direct link to status-application-sync-comparedTo-source-kustomize-labelWithoutSelector") ##### `kubeVersion` string [​](#status-application-sync-comparedTo-source-kustomize-kubeVersion "Direct link to status-application-sync-comparedTo-source-kustomize-kubeVersion") ##### `apiVersions` string\[] [​](#status-application-sync-comparedTo-source-kustomize-apiVersions "Direct link to status-application-sync-comparedTo-source-kustomize-apiVersions") ##### `labelIncludeTemplates` boolean [​](#status-application-sync-comparedTo-source-kustomize-labelIncludeTemplates "Direct link to status-application-sync-comparedTo-source-kustomize-labelIncludeTemplates") ##### `directory` object [​](#status-application-sync-comparedTo-source-directory "Direct link to status-application-sync-comparedTo-source-directory") ##### `recurse` boolean [​](#status-application-sync-comparedTo-source-directory-recurse "Direct link to status-application-sync-comparedTo-source-directory-recurse") ##### `jsonnet` object [​](#status-application-sync-comparedTo-source-directory-jsonnet "Direct link to status-application-sync-comparedTo-source-directory-jsonnet") ##### `extVars` object\[] [​](#status-application-sync-comparedTo-source-directory-jsonnet-extVars "Direct link to status-application-sync-comparedTo-source-directory-jsonnet-extVars") ##### `name` required string [​](#status-application-sync-comparedTo-source-directory-jsonnet-extVars-name "Direct link to status-application-sync-comparedTo-source-directory-jsonnet-extVars-name") ##### `value` required string [​](#status-application-sync-comparedTo-source-directory-jsonnet-extVars-value "Direct link to status-application-sync-comparedTo-source-directory-jsonnet-extVars-value") ##### `code` boolean [​](#status-application-sync-comparedTo-source-directory-jsonnet-extVars-code "Direct link to status-application-sync-comparedTo-source-directory-jsonnet-extVars-code") ##### `tlas` object\[] [​](#status-application-sync-comparedTo-source-directory-jsonnet-tlas "Direct link to status-application-sync-comparedTo-source-directory-jsonnet-tlas") ##### `name` required string [​](#status-application-sync-comparedTo-source-directory-jsonnet-tlas-name "Direct link to status-application-sync-comparedTo-source-directory-jsonnet-tlas-name") ##### `value` required string [​](#status-application-sync-comparedTo-source-directory-jsonnet-tlas-value "Direct link to status-application-sync-comparedTo-source-directory-jsonnet-tlas-value") ##### `code` boolean [​](#status-application-sync-comparedTo-source-directory-jsonnet-tlas-code "Direct link to status-application-sync-comparedTo-source-directory-jsonnet-tlas-code") ##### `libs` string\[] [​](#status-application-sync-comparedTo-source-directory-jsonnet-libs "Direct link to status-application-sync-comparedTo-source-directory-jsonnet-libs") ##### `exclude` string [​](#status-application-sync-comparedTo-source-directory-exclude "Direct link to status-application-sync-comparedTo-source-directory-exclude") ##### `include` string [​](#status-application-sync-comparedTo-source-directory-include "Direct link to status-application-sync-comparedTo-source-directory-include") ##### `plugin` object [​](#status-application-sync-comparedTo-source-plugin "Direct link to status-application-sync-comparedTo-source-plugin") ##### `name` string [​](#status-application-sync-comparedTo-source-plugin-name "Direct link to status-application-sync-comparedTo-source-plugin-name") ##### `env` object [​](#status-application-sync-comparedTo-source-plugin-env "Direct link to status-application-sync-comparedTo-source-plugin-env") ##### `parameters` object [​](#status-application-sync-comparedTo-source-plugin-parameters "Direct link to status-application-sync-comparedTo-source-plugin-parameters") ##### `chart` string [​](#status-application-sync-comparedTo-source-chart "Direct link to status-application-sync-comparedTo-source-chart") ##### `ref` string [​](#status-application-sync-comparedTo-source-ref "Direct link to status-application-sync-comparedTo-source-ref") ##### `name` string [​](#status-application-sync-comparedTo-source-name "Direct link to status-application-sync-comparedTo-source-name") ##### `destination` required object [​](#status-application-sync-comparedTo-destination "Direct link to status-application-sync-comparedTo-destination") ##### `server` string [​](#status-application-sync-comparedTo-destination-server "Direct link to status-application-sync-comparedTo-destination-server") ##### `namespace` string [​](#status-application-sync-comparedTo-destination-namespace "Direct link to status-application-sync-comparedTo-destination-namespace") ##### `name` string [​](#status-application-sync-comparedTo-destination-name "Direct link to status-application-sync-comparedTo-destination-name") ##### `sources` object [​](#status-application-sync-comparedTo-sources "Direct link to status-application-sync-comparedTo-sources") ##### `ignoreDifferences` object [​](#status-application-sync-comparedTo-ignoreDifferences "Direct link to status-application-sync-comparedTo-ignoreDifferences") ##### `revision` string [​](#status-application-sync-revision "Direct link to status-application-sync-revision") ##### `revisions` string\[] [​](#status-application-sync-revisions "Direct link to status-application-sync-revisions") #### `health` object [​](#status-application-health "Direct link to status-application-health") ##### `status` string [​](#status-application-health-status "Direct link to status-application-health-status") ##### `message` string [​](#status-application-health-message "Direct link to status-application-health-message") ##### `lastTransitionTime` object [​](#status-application-health-lastTransitionTime "Direct link to status-application-health-lastTransitionTime") #### `history` object [​](#status-application-history "Direct link to status-application-history") #### `conditions` object\[] [​](#status-application-conditions "Direct link to status-application-conditions") ##### `type` required string [​](#status-application-conditions-type "Direct link to status-application-conditions-type") ##### `message` required string [​](#status-application-conditions-message "Direct link to status-application-conditions-message") ##### `lastTransitionTime` object [​](#status-application-conditions-lastTransitionTime "Direct link to status-application-conditions-lastTransitionTime") #### `reconciledAt` object [​](#status-application-reconciledAt "Direct link to status-application-reconciledAt") #### `operationState` object [​](#status-application-operationState "Direct link to status-application-operationState") ##### `operation` required object [​](#status-application-operationState-operation "Direct link to status-application-operationState-operation") ##### `sync` object [​](#status-application-operationState-operation-sync "Direct link to status-application-operationState-operation-sync") ##### `revision` string [​](#status-application-operationState-operation-sync-revision "Direct link to status-application-operationState-operation-sync-revision") ##### `prune` boolean [​](#status-application-operationState-operation-sync-prune "Direct link to status-application-operationState-operation-sync-prune") ##### `dryRun` boolean [​](#status-application-operationState-operation-sync-dryRun "Direct link to status-application-operationState-operation-sync-dryRun") ##### `syncStrategy` object [​](#status-application-operationState-operation-sync-syncStrategy "Direct link to status-application-operationState-operation-sync-syncStrategy") ##### `apply` object [​](#status-application-operationState-operation-sync-syncStrategy-apply "Direct link to status-application-operationState-operation-sync-syncStrategy-apply") ##### `force` boolean [​](#status-application-operationState-operation-sync-syncStrategy-apply-force "Direct link to status-application-operationState-operation-sync-syncStrategy-apply-force") ##### `hook` object [​](#status-application-operationState-operation-sync-syncStrategy-hook "Direct link to status-application-operationState-operation-sync-syncStrategy-hook") ##### `force` boolean [​](#status-application-operationState-operation-sync-syncStrategy-hook-force "Direct link to status-application-operationState-operation-sync-syncStrategy-hook-force") ##### `resources` object\[] [​](#status-application-operationState-operation-sync-resources "Direct link to status-application-operationState-operation-sync-resources") ##### `group` string [​](#status-application-operationState-operation-sync-resources-group "Direct link to status-application-operationState-operation-sync-resources-group") ##### `kind` required string [​](#status-application-operationState-operation-sync-resources-kind "Direct link to status-application-operationState-operation-sync-resources-kind") ##### `name` required string [​](#status-application-operationState-operation-sync-resources-name "Direct link to status-application-operationState-operation-sync-resources-name") ##### `namespace` string [​](#status-application-operationState-operation-sync-resources-namespace "Direct link to status-application-operationState-operation-sync-resources-namespace") ##### `source` object [​](#status-application-operationState-operation-sync-source "Direct link to status-application-operationState-operation-sync-source") ##### `repoURL` required string [​](#status-application-operationState-operation-sync-source-repoURL "Direct link to status-application-operationState-operation-sync-source-repoURL") ##### `path` string [​](#status-application-operationState-operation-sync-source-path "Direct link to status-application-operationState-operation-sync-source-path") ##### `targetRevision` string [​](#status-application-operationState-operation-sync-source-targetRevision "Direct link to status-application-operationState-operation-sync-source-targetRevision") ##### `helm` object [​](#status-application-operationState-operation-sync-source-helm "Direct link to status-application-operationState-operation-sync-source-helm") ##### `valueFiles` string\[] [​](#status-application-operationState-operation-sync-source-helm-valueFiles "Direct link to status-application-operationState-operation-sync-source-helm-valueFiles") ##### `parameters` object\[] [​](#status-application-operationState-operation-sync-source-helm-parameters "Direct link to status-application-operationState-operation-sync-source-helm-parameters") ##### `name` string [​](#status-application-operationState-operation-sync-source-helm-parameters-name "Direct link to status-application-operationState-operation-sync-source-helm-parameters-name") ##### `value` string [​](#status-application-operationState-operation-sync-source-helm-parameters-value "Direct link to status-application-operationState-operation-sync-source-helm-parameters-value") ##### `forceString` boolean [​](#status-application-operationState-operation-sync-source-helm-parameters-forceString "Direct link to status-application-operationState-operation-sync-source-helm-parameters-forceString") ##### `releaseName` string [​](#status-application-operationState-operation-sync-source-helm-releaseName "Direct link to status-application-operationState-operation-sync-source-helm-releaseName") ##### `values` string [​](#status-application-operationState-operation-sync-source-helm-values "Direct link to status-application-operationState-operation-sync-source-helm-values") ##### `fileParameters` object\[] [​](#status-application-operationState-operation-sync-source-helm-fileParameters "Direct link to status-application-operationState-operation-sync-source-helm-fileParameters") ##### `name` string [​](#status-application-operationState-operation-sync-source-helm-fileParameters-name "Direct link to status-application-operationState-operation-sync-source-helm-fileParameters-name") ##### `path` string [​](#status-application-operationState-operation-sync-source-helm-fileParameters-path "Direct link to status-application-operationState-operation-sync-source-helm-fileParameters-path") ##### `version` string [​](#status-application-operationState-operation-sync-source-helm-version "Direct link to status-application-operationState-operation-sync-source-helm-version") ##### `passCredentials` boolean [​](#status-application-operationState-operation-sync-source-helm-passCredentials "Direct link to status-application-operationState-operation-sync-source-helm-passCredentials") ##### `ignoreMissingValueFiles` boolean [​](#status-application-operationState-operation-sync-source-helm-ignoreMissingValueFiles "Direct link to status-application-operationState-operation-sync-source-helm-ignoreMissingValueFiles") ##### `skipCrds` boolean [​](#status-application-operationState-operation-sync-source-helm-skipCrds "Direct link to status-application-operationState-operation-sync-source-helm-skipCrds") ##### `valuesObject` object [​](#status-application-operationState-operation-sync-source-helm-valuesObject "Direct link to status-application-operationState-operation-sync-source-helm-valuesObject") ##### `namespace` string [​](#status-application-operationState-operation-sync-source-helm-namespace "Direct link to status-application-operationState-operation-sync-source-helm-namespace") ##### `kubeVersion` string [​](#status-application-operationState-operation-sync-source-helm-kubeVersion "Direct link to status-application-operationState-operation-sync-source-helm-kubeVersion") ##### `apiVersions` string\[] [​](#status-application-operationState-operation-sync-source-helm-apiVersions "Direct link to status-application-operationState-operation-sync-source-helm-apiVersions") ##### `skipTests` boolean [​](#status-application-operationState-operation-sync-source-helm-skipTests "Direct link to status-application-operationState-operation-sync-source-helm-skipTests") ##### `skipSchemaValidation` boolean [​](#status-application-operationState-operation-sync-source-helm-skipSchemaValidation "Direct link to status-application-operationState-operation-sync-source-helm-skipSchemaValidation") ##### `kustomize` object [​](#status-application-operationState-operation-sync-source-kustomize "Direct link to status-application-operationState-operation-sync-source-kustomize") ##### `namePrefix` string [​](#status-application-operationState-operation-sync-source-kustomize-namePrefix "Direct link to status-application-operationState-operation-sync-source-kustomize-namePrefix") ##### `nameSuffix` string [​](#status-application-operationState-operation-sync-source-kustomize-nameSuffix "Direct link to status-application-operationState-operation-sync-source-kustomize-nameSuffix") ##### `images` object [​](#status-application-operationState-operation-sync-source-kustomize-images "Direct link to status-application-operationState-operation-sync-source-kustomize-images") ##### `commonLabels` object [​](#status-application-operationState-operation-sync-source-kustomize-commonLabels "Direct link to status-application-operationState-operation-sync-source-kustomize-commonLabels") ##### `version` string [​](#status-application-operationState-operation-sync-source-kustomize-version "Direct link to status-application-operationState-operation-sync-source-kustomize-version") ##### `commonAnnotations` object [​](#status-application-operationState-operation-sync-source-kustomize-commonAnnotations "Direct link to status-application-operationState-operation-sync-source-kustomize-commonAnnotations") ##### `forceCommonLabels` boolean [​](#status-application-operationState-operation-sync-source-kustomize-forceCommonLabels "Direct link to status-application-operationState-operation-sync-source-kustomize-forceCommonLabels") ##### `forceCommonAnnotations` boolean [​](#status-application-operationState-operation-sync-source-kustomize-forceCommonAnnotations "Direct link to status-application-operationState-operation-sync-source-kustomize-forceCommonAnnotations") ##### `namespace` string [​](#status-application-operationState-operation-sync-source-kustomize-namespace "Direct link to status-application-operationState-operation-sync-source-kustomize-namespace") ##### `commonAnnotationsEnvsubst` boolean [​](#status-application-operationState-operation-sync-source-kustomize-commonAnnotationsEnvsubst "Direct link to status-application-operationState-operation-sync-source-kustomize-commonAnnotationsEnvsubst") ##### `replicas` object [​](#status-application-operationState-operation-sync-source-kustomize-replicas "Direct link to status-application-operationState-operation-sync-source-kustomize-replicas") ##### `patches` object [​](#status-application-operationState-operation-sync-source-kustomize-patches "Direct link to status-application-operationState-operation-sync-source-kustomize-patches") ##### `components` string\[] [​](#status-application-operationState-operation-sync-source-kustomize-components "Direct link to status-application-operationState-operation-sync-source-kustomize-components") ##### `ignoreMissingComponents` boolean [​](#status-application-operationState-operation-sync-source-kustomize-ignoreMissingComponents "Direct link to status-application-operationState-operation-sync-source-kustomize-ignoreMissingComponents") ##### `labelWithoutSelector` boolean [​](#status-application-operationState-operation-sync-source-kustomize-labelWithoutSelector "Direct link to status-application-operationState-operation-sync-source-kustomize-labelWithoutSelector") ##### `kubeVersion` string [​](#status-application-operationState-operation-sync-source-kustomize-kubeVersion "Direct link to status-application-operationState-operation-sync-source-kustomize-kubeVersion") ##### `apiVersions` string\[] [​](#status-application-operationState-operation-sync-source-kustomize-apiVersions "Direct link to status-application-operationState-operation-sync-source-kustomize-apiVersions") ##### `labelIncludeTemplates` boolean [​](#status-application-operationState-operation-sync-source-kustomize-labelIncludeTemplates "Direct link to status-application-operationState-operation-sync-source-kustomize-labelIncludeTemplates") ##### `directory` object [​](#status-application-operationState-operation-sync-source-directory "Direct link to status-application-operationState-operation-sync-source-directory") ##### `recurse` boolean [​](#status-application-operationState-operation-sync-source-directory-recurse "Direct link to status-application-operationState-operation-sync-source-directory-recurse") ##### `jsonnet` object [​](#status-application-operationState-operation-sync-source-directory-jsonnet "Direct link to status-application-operationState-operation-sync-source-directory-jsonnet") ##### `extVars` object\[] [​](#status-application-operationState-operation-sync-source-directory-jsonnet-extVars "Direct link to status-application-operationState-operation-sync-source-directory-jsonnet-extVars") ##### `name` required string [​](#status-application-operationState-operation-sync-source-directory-jsonnet-extVars-name "Direct link to status-application-operationState-operation-sync-source-directory-jsonnet-extVars-name") ##### `value` required string [​](#status-application-operationState-operation-sync-source-directory-jsonnet-extVars-value "Direct link to status-application-operationState-operation-sync-source-directory-jsonnet-extVars-value") ##### `code` boolean [​](#status-application-operationState-operation-sync-source-directory-jsonnet-extVars-code "Direct link to status-application-operationState-operation-sync-source-directory-jsonnet-extVars-code") ##### `tlas` object\[] [​](#status-application-operationState-operation-sync-source-directory-jsonnet-tlas "Direct link to status-application-operationState-operation-sync-source-directory-jsonnet-tlas") ##### `name` required string [​](#status-application-operationState-operation-sync-source-directory-jsonnet-tlas-name "Direct link to status-application-operationState-operation-sync-source-directory-jsonnet-tlas-name") ##### `value` required string [​](#status-application-operationState-operation-sync-source-directory-jsonnet-tlas-value "Direct link to status-application-operationState-operation-sync-source-directory-jsonnet-tlas-value") ##### `code` boolean [​](#status-application-operationState-operation-sync-source-directory-jsonnet-tlas-code "Direct link to status-application-operationState-operation-sync-source-directory-jsonnet-tlas-code") ##### `libs` string\[] [​](#status-application-operationState-operation-sync-source-directory-jsonnet-libs "Direct link to status-application-operationState-operation-sync-source-directory-jsonnet-libs") ##### `exclude` string [​](#status-application-operationState-operation-sync-source-directory-exclude "Direct link to status-application-operationState-operation-sync-source-directory-exclude") ##### `include` string [​](#status-application-operationState-operation-sync-source-directory-include "Direct link to status-application-operationState-operation-sync-source-directory-include") ##### `plugin` object [​](#status-application-operationState-operation-sync-source-plugin "Direct link to status-application-operationState-operation-sync-source-plugin") ##### `name` string [​](#status-application-operationState-operation-sync-source-plugin-name "Direct link to status-application-operationState-operation-sync-source-plugin-name") ##### `env` object [​](#status-application-operationState-operation-sync-source-plugin-env "Direct link to status-application-operationState-operation-sync-source-plugin-env") ##### `parameters` object [​](#status-application-operationState-operation-sync-source-plugin-parameters "Direct link to status-application-operationState-operation-sync-source-plugin-parameters") ##### `chart` string [​](#status-application-operationState-operation-sync-source-chart "Direct link to status-application-operationState-operation-sync-source-chart") ##### `ref` string [​](#status-application-operationState-operation-sync-source-ref "Direct link to status-application-operationState-operation-sync-source-ref") ##### `name` string [​](#status-application-operationState-operation-sync-source-name "Direct link to status-application-operationState-operation-sync-source-name") ##### `manifests` string\[] [​](#status-application-operationState-operation-sync-manifests "Direct link to status-application-operationState-operation-sync-manifests") ##### `syncOptions` object [​](#status-application-operationState-operation-sync-syncOptions "Direct link to status-application-operationState-operation-sync-syncOptions") ##### `sources` object [​](#status-application-operationState-operation-sync-sources "Direct link to status-application-operationState-operation-sync-sources") ##### `revisions` string\[] [​](#status-application-operationState-operation-sync-revisions "Direct link to status-application-operationState-operation-sync-revisions") ##### `autoHealAttemptsCount` integer [​](#status-application-operationState-operation-sync-autoHealAttemptsCount "Direct link to status-application-operationState-operation-sync-autoHealAttemptsCount") ##### `initiatedBy` object [​](#status-application-operationState-operation-initiatedBy "Direct link to status-application-operationState-operation-initiatedBy") ##### `username` string [​](#status-application-operationState-operation-initiatedBy-username "Direct link to status-application-operationState-operation-initiatedBy-username") ##### `automated` boolean [​](#status-application-operationState-operation-initiatedBy-automated "Direct link to status-application-operationState-operation-initiatedBy-automated") ##### `info` object\[] [​](#status-application-operationState-operation-info "Direct link to status-application-operationState-operation-info") ##### `name` required string [​](#status-application-operationState-operation-info-name "Direct link to status-application-operationState-operation-info-name") ##### `value` required string [​](#status-application-operationState-operation-info-value "Direct link to status-application-operationState-operation-info-value") ##### `retry` object [​](#status-application-operationState-operation-retry "Direct link to status-application-operationState-operation-retry") ##### `limit` integer [​](#status-application-operationState-operation-retry-limit "Direct link to status-application-operationState-operation-retry-limit") ##### `backoff` object [​](#status-application-operationState-operation-retry-backoff "Direct link to status-application-operationState-operation-retry-backoff") ##### `duration` string [​](#status-application-operationState-operation-retry-backoff-duration "Direct link to status-application-operationState-operation-retry-backoff-duration") ##### `factor` integer [​](#status-application-operationState-operation-retry-backoff-factor "Direct link to status-application-operationState-operation-retry-backoff-factor") ##### `maxDuration` string [​](#status-application-operationState-operation-retry-backoff-maxDuration "Direct link to status-application-operationState-operation-retry-backoff-maxDuration") ##### `refresh` boolean [​](#status-application-operationState-operation-retry-refresh "Direct link to status-application-operationState-operation-retry-refresh") ##### `phase` required string [​](#status-application-operationState-phase "Direct link to status-application-operationState-phase") ##### `message` string [​](#status-application-operationState-message "Direct link to status-application-operationState-message") ##### `syncResult` object [​](#status-application-operationState-syncResult "Direct link to status-application-operationState-syncResult") ##### `resources` object [​](#status-application-operationState-syncResult-resources "Direct link to status-application-operationState-syncResult-resources") ##### `revision` required string [​](#status-application-operationState-syncResult-revision "Direct link to status-application-operationState-syncResult-revision") ##### `source` object [​](#status-application-operationState-syncResult-source "Direct link to status-application-operationState-syncResult-source") ##### `repoURL` required string [​](#status-application-operationState-syncResult-source-repoURL "Direct link to status-application-operationState-syncResult-source-repoURL") ##### `path` string [​](#status-application-operationState-syncResult-source-path "Direct link to status-application-operationState-syncResult-source-path") ##### `targetRevision` string [​](#status-application-operationState-syncResult-source-targetRevision "Direct link to status-application-operationState-syncResult-source-targetRevision") ##### `helm` object [​](#status-application-operationState-syncResult-source-helm "Direct link to status-application-operationState-syncResult-source-helm") ##### `valueFiles` string\[] [​](#status-application-operationState-syncResult-source-helm-valueFiles "Direct link to status-application-operationState-syncResult-source-helm-valueFiles") ##### `parameters` object\[] [​](#status-application-operationState-syncResult-source-helm-parameters "Direct link to status-application-operationState-syncResult-source-helm-parameters") ##### `name` string [​](#status-application-operationState-syncResult-source-helm-parameters-name "Direct link to status-application-operationState-syncResult-source-helm-parameters-name") ##### `value` string [​](#status-application-operationState-syncResult-source-helm-parameters-value "Direct link to status-application-operationState-syncResult-source-helm-parameters-value") ##### `forceString` boolean [​](#status-application-operationState-syncResult-source-helm-parameters-forceString "Direct link to status-application-operationState-syncResult-source-helm-parameters-forceString") ##### `releaseName` string [​](#status-application-operationState-syncResult-source-helm-releaseName "Direct link to status-application-operationState-syncResult-source-helm-releaseName") ##### `values` string [​](#status-application-operationState-syncResult-source-helm-values "Direct link to status-application-operationState-syncResult-source-helm-values") ##### `fileParameters` object\[] [​](#status-application-operationState-syncResult-source-helm-fileParameters "Direct link to status-application-operationState-syncResult-source-helm-fileParameters") ##### `name` string [​](#status-application-operationState-syncResult-source-helm-fileParameters-name "Direct link to status-application-operationState-syncResult-source-helm-fileParameters-name") ##### `path` string [​](#status-application-operationState-syncResult-source-helm-fileParameters-path "Direct link to status-application-operationState-syncResult-source-helm-fileParameters-path") ##### `version` string [​](#status-application-operationState-syncResult-source-helm-version "Direct link to status-application-operationState-syncResult-source-helm-version") ##### `passCredentials` boolean [​](#status-application-operationState-syncResult-source-helm-passCredentials "Direct link to status-application-operationState-syncResult-source-helm-passCredentials") ##### `ignoreMissingValueFiles` boolean [​](#status-application-operationState-syncResult-source-helm-ignoreMissingValueFiles "Direct link to status-application-operationState-syncResult-source-helm-ignoreMissingValueFiles") ##### `skipCrds` boolean [​](#status-application-operationState-syncResult-source-helm-skipCrds "Direct link to status-application-operationState-syncResult-source-helm-skipCrds") ##### `valuesObject` object [​](#status-application-operationState-syncResult-source-helm-valuesObject "Direct link to status-application-operationState-syncResult-source-helm-valuesObject") ##### `namespace` string [​](#status-application-operationState-syncResult-source-helm-namespace "Direct link to status-application-operationState-syncResult-source-helm-namespace") ##### `kubeVersion` string [​](#status-application-operationState-syncResult-source-helm-kubeVersion "Direct link to status-application-operationState-syncResult-source-helm-kubeVersion") ##### `apiVersions` string\[] [​](#status-application-operationState-syncResult-source-helm-apiVersions "Direct link to status-application-operationState-syncResult-source-helm-apiVersions") ##### `skipTests` boolean [​](#status-application-operationState-syncResult-source-helm-skipTests "Direct link to status-application-operationState-syncResult-source-helm-skipTests") ##### `skipSchemaValidation` boolean [​](#status-application-operationState-syncResult-source-helm-skipSchemaValidation "Direct link to status-application-operationState-syncResult-source-helm-skipSchemaValidation") ##### `kustomize` object [​](#status-application-operationState-syncResult-source-kustomize "Direct link to status-application-operationState-syncResult-source-kustomize") ##### `namePrefix` string [​](#status-application-operationState-syncResult-source-kustomize-namePrefix "Direct link to status-application-operationState-syncResult-source-kustomize-namePrefix") ##### `nameSuffix` string [​](#status-application-operationState-syncResult-source-kustomize-nameSuffix "Direct link to status-application-operationState-syncResult-source-kustomize-nameSuffix") ##### `images` object [​](#status-application-operationState-syncResult-source-kustomize-images "Direct link to status-application-operationState-syncResult-source-kustomize-images") ##### `commonLabels` object [​](#status-application-operationState-syncResult-source-kustomize-commonLabels "Direct link to status-application-operationState-syncResult-source-kustomize-commonLabels") ##### `version` string [​](#status-application-operationState-syncResult-source-kustomize-version "Direct link to status-application-operationState-syncResult-source-kustomize-version") ##### `commonAnnotations` object [​](#status-application-operationState-syncResult-source-kustomize-commonAnnotations "Direct link to status-application-operationState-syncResult-source-kustomize-commonAnnotations") ##### `forceCommonLabels` boolean [​](#status-application-operationState-syncResult-source-kustomize-forceCommonLabels "Direct link to status-application-operationState-syncResult-source-kustomize-forceCommonLabels") ##### `forceCommonAnnotations` boolean [​](#status-application-operationState-syncResult-source-kustomize-forceCommonAnnotations "Direct link to status-application-operationState-syncResult-source-kustomize-forceCommonAnnotations") ##### `namespace` string [​](#status-application-operationState-syncResult-source-kustomize-namespace "Direct link to status-application-operationState-syncResult-source-kustomize-namespace") ##### `commonAnnotationsEnvsubst` boolean [​](#status-application-operationState-syncResult-source-kustomize-commonAnnotationsEnvsubst "Direct link to status-application-operationState-syncResult-source-kustomize-commonAnnotationsEnvsubst") ##### `replicas` object [​](#status-application-operationState-syncResult-source-kustomize-replicas "Direct link to status-application-operationState-syncResult-source-kustomize-replicas") ##### `patches` object [​](#status-application-operationState-syncResult-source-kustomize-patches "Direct link to status-application-operationState-syncResult-source-kustomize-patches") ##### `components` string\[] [​](#status-application-operationState-syncResult-source-kustomize-components "Direct link to status-application-operationState-syncResult-source-kustomize-components") ##### `ignoreMissingComponents` boolean [​](#status-application-operationState-syncResult-source-kustomize-ignoreMissingComponents "Direct link to status-application-operationState-syncResult-source-kustomize-ignoreMissingComponents") ##### `labelWithoutSelector` boolean [​](#status-application-operationState-syncResult-source-kustomize-labelWithoutSelector "Direct link to status-application-operationState-syncResult-source-kustomize-labelWithoutSelector") ##### `kubeVersion` string [​](#status-application-operationState-syncResult-source-kustomize-kubeVersion "Direct link to status-application-operationState-syncResult-source-kustomize-kubeVersion") ##### `apiVersions` string\[] [​](#status-application-operationState-syncResult-source-kustomize-apiVersions "Direct link to status-application-operationState-syncResult-source-kustomize-apiVersions") ##### `labelIncludeTemplates` boolean [​](#status-application-operationState-syncResult-source-kustomize-labelIncludeTemplates "Direct link to status-application-operationState-syncResult-source-kustomize-labelIncludeTemplates") ##### `directory` object [​](#status-application-operationState-syncResult-source-directory "Direct link to status-application-operationState-syncResult-source-directory") ##### `recurse` boolean [​](#status-application-operationState-syncResult-source-directory-recurse "Direct link to status-application-operationState-syncResult-source-directory-recurse") ##### `jsonnet` object [​](#status-application-operationState-syncResult-source-directory-jsonnet "Direct link to status-application-operationState-syncResult-source-directory-jsonnet") ##### `extVars` object\[] [​](#status-application-operationState-syncResult-source-directory-jsonnet-extVars "Direct link to status-application-operationState-syncResult-source-directory-jsonnet-extVars") ##### `name` required string [​](#status-application-operationState-syncResult-source-directory-jsonnet-extVars-name "Direct link to status-application-operationState-syncResult-source-directory-jsonnet-extVars-name") ##### `value` required string [​](#status-application-operationState-syncResult-source-directory-jsonnet-extVars-value "Direct link to status-application-operationState-syncResult-source-directory-jsonnet-extVars-value") ##### `code` boolean [​](#status-application-operationState-syncResult-source-directory-jsonnet-extVars-code "Direct link to status-application-operationState-syncResult-source-directory-jsonnet-extVars-code") ##### `tlas` object\[] [​](#status-application-operationState-syncResult-source-directory-jsonnet-tlas "Direct link to status-application-operationState-syncResult-source-directory-jsonnet-tlas") ##### `name` required string [​](#status-application-operationState-syncResult-source-directory-jsonnet-tlas-name "Direct link to status-application-operationState-syncResult-source-directory-jsonnet-tlas-name") ##### `value` required string [​](#status-application-operationState-syncResult-source-directory-jsonnet-tlas-value "Direct link to status-application-operationState-syncResult-source-directory-jsonnet-tlas-value") ##### `code` boolean [​](#status-application-operationState-syncResult-source-directory-jsonnet-tlas-code "Direct link to status-application-operationState-syncResult-source-directory-jsonnet-tlas-code") ##### `libs` string\[] [​](#status-application-operationState-syncResult-source-directory-jsonnet-libs "Direct link to status-application-operationState-syncResult-source-directory-jsonnet-libs") ##### `exclude` string [​](#status-application-operationState-syncResult-source-directory-exclude "Direct link to status-application-operationState-syncResult-source-directory-exclude") ##### `include` string [​](#status-application-operationState-syncResult-source-directory-include "Direct link to status-application-operationState-syncResult-source-directory-include") ##### `plugin` object [​](#status-application-operationState-syncResult-source-plugin "Direct link to status-application-operationState-syncResult-source-plugin") ##### `name` string [​](#status-application-operationState-syncResult-source-plugin-name "Direct link to status-application-operationState-syncResult-source-plugin-name") ##### `env` object [​](#status-application-operationState-syncResult-source-plugin-env "Direct link to status-application-operationState-syncResult-source-plugin-env") ##### `parameters` object [​](#status-application-operationState-syncResult-source-plugin-parameters "Direct link to status-application-operationState-syncResult-source-plugin-parameters") ##### `chart` string [​](#status-application-operationState-syncResult-source-chart "Direct link to status-application-operationState-syncResult-source-chart") ##### `ref` string [​](#status-application-operationState-syncResult-source-ref "Direct link to status-application-operationState-syncResult-source-ref") ##### `name` string [​](#status-application-operationState-syncResult-source-name "Direct link to status-application-operationState-syncResult-source-name") ##### `sources` object [​](#status-application-operationState-syncResult-sources "Direct link to status-application-operationState-syncResult-sources") ##### `revisions` string\[] [​](#status-application-operationState-syncResult-revisions "Direct link to status-application-operationState-syncResult-revisions") ##### `managedNamespaceMetadata` object [​](#status-application-operationState-syncResult-managedNamespaceMetadata "Direct link to status-application-operationState-syncResult-managedNamespaceMetadata") ##### `labels` object [​](#status-application-operationState-syncResult-managedNamespaceMetadata-labels "Direct link to status-application-operationState-syncResult-managedNamespaceMetadata-labels") ##### `annotations` object [​](#status-application-operationState-syncResult-managedNamespaceMetadata-annotations "Direct link to status-application-operationState-syncResult-managedNamespaceMetadata-annotations") ##### `startedAt` required object [​](#status-application-operationState-startedAt "Direct link to status-application-operationState-startedAt") ##### `finishedAt` object [​](#status-application-operationState-finishedAt "Direct link to status-application-operationState-finishedAt") ##### `retryCount` integer [​](#status-application-operationState-retryCount "Direct link to status-application-operationState-retryCount") #### `observedAt` object [​](#status-application-observedAt "Direct link to status-application-observedAt") #### `sourceType` string [​](#status-application-sourceType "Direct link to status-application-sourceType") #### `summary` object [​](#status-application-summary "Direct link to status-application-summary") ##### `externalURLs` string\[] [​](#status-application-summary-externalURLs "Direct link to status-application-summary-externalURLs") ##### `images` string\[] [​](#status-application-summary-images "Direct link to status-application-summary-images") #### `resourceHealthSource` string [​](#status-application-resourceHealthSource "Direct link to status-application-resourceHealthSource") #### `sourceTypes` string\[] [​](#status-application-sourceTypes "Direct link to status-application-sourceTypes") #### `controllerNamespace` string [​](#status-application-controllerNamespace "Direct link to status-application-controllerNamespace") #### `sourceHydrator` object [​](#status-application-sourceHydrator "Direct link to status-application-sourceHydrator") ##### `lastSuccessfulOperation` object [​](#status-application-sourceHydrator-lastSuccessfulOperation "Direct link to status-application-sourceHydrator-lastSuccessfulOperation") ##### `drySHA` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-drySHA "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-drySHA") ##### `hydratedSHA` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-hydratedSHA "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-hydratedSHA") ##### `sourceHydrator` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator") ##### `drySource` required object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource") ##### `repoURL` required string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-repoURL "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-repoURL") ##### `targetRevision` required string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-targetRevision "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-targetRevision") ##### `path` required string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-path "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-path") ##### `helm` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm") ##### `valueFiles` string\[] [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-valueFiles "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-valueFiles") ##### `parameters` object\[] [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-parameters "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-parameters") ##### `name` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-parameters-name "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-parameters-name") ##### `value` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-parameters-value "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-parameters-value") ##### `forceString` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-parameters-forceString "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-parameters-forceString") ##### `releaseName` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-releaseName "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-releaseName") ##### `values` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-values "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-values") ##### `fileParameters` object\[] [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-fileParameters "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-fileParameters") ##### `name` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-fileParameters-name "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-fileParameters-name") ##### `path` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-fileParameters-path "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-fileParameters-path") ##### `version` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-version "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-version") ##### `passCredentials` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-passCredentials "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-passCredentials") ##### `ignoreMissingValueFiles` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-ignoreMissingValueFiles "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-ignoreMissingValueFiles") ##### `skipCrds` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-skipCrds "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-skipCrds") ##### `valuesObject` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-valuesObject "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-valuesObject") ##### `namespace` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-namespace "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-namespace") ##### `kubeVersion` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-kubeVersion "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-kubeVersion") ##### `apiVersions` string\[] [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-apiVersions "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-apiVersions") ##### `skipTests` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-skipTests "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-skipTests") ##### `skipSchemaValidation` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-skipSchemaValidation "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-helm-skipSchemaValidation") ##### `kustomize` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize") ##### `namePrefix` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-namePrefix "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-namePrefix") ##### `nameSuffix` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-nameSuffix "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-nameSuffix") ##### `images` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-images "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-images") ##### `commonLabels` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-commonLabels "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-commonLabels") ##### `version` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-version "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-version") ##### `commonAnnotations` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-commonAnnotations "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-commonAnnotations") ##### `forceCommonLabels` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-forceCommonLabels "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-forceCommonLabels") ##### `forceCommonAnnotations` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-forceCommonAnnotations "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-forceCommonAnnotations") ##### `namespace` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-namespace "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-namespace") ##### `commonAnnotationsEnvsubst` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-commonAnnotationsEnvsubst "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-commonAnnotationsEnvsubst") ##### `replicas` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-replicas "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-replicas") ##### `patches` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-patches "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-patches") ##### `components` string\[] [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-components "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-components") ##### `ignoreMissingComponents` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-ignoreMissingComponents "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-ignoreMissingComponents") ##### `labelWithoutSelector` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-labelWithoutSelector "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-labelWithoutSelector") ##### `kubeVersion` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-kubeVersion "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-kubeVersion") ##### `apiVersions` string\[] [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-apiVersions "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-apiVersions") ##### `labelIncludeTemplates` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-labelIncludeTemplates "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-kustomize-labelIncludeTemplates") ##### `directory` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory") ##### `recurse` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-recurse "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-recurse") ##### `jsonnet` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet") ##### `extVars` object\[] [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-extVars "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-extVars") ##### `name` required string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-extVars-name "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-extVars-name") ##### `value` required string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-extVars-value "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-extVars-value") ##### `code` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-extVars-code "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-extVars-code") ##### `tlas` object\[] [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-tlas "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-tlas") ##### `name` required string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-tlas-name "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-tlas-name") ##### `value` required string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-tlas-value "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-tlas-value") ##### `code` boolean [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-tlas-code "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-tlas-code") ##### `libs` string\[] [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-libs "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-jsonnet-libs") ##### `exclude` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-exclude "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-exclude") ##### `include` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-include "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-directory-include") ##### `plugin` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-plugin "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-plugin") ##### `name` string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-plugin-name "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-plugin-name") ##### `env` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-plugin-env "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-plugin-env") ##### `parameters` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-plugin-parameters "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-drySource-plugin-parameters") ##### `syncSource` required object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-syncSource "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-syncSource") ##### `targetBranch` required string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-syncSource-targetBranch "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-syncSource-targetBranch") ##### `path` required string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-syncSource-path "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-syncSource-path") ##### `hydrateTo` object [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-hydrateTo "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-hydrateTo") ##### `targetBranch` required string [​](#status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-hydrateTo-targetBranch "Direct link to status-application-sourceHydrator-lastSuccessfulOperation-sourceHydrator-hydrateTo-targetBranch") ##### `currentOperation` object [​](#status-application-sourceHydrator-currentOperation "Direct link to status-application-sourceHydrator-currentOperation") ##### `startedAt` object [​](#status-application-sourceHydrator-currentOperation-startedAt "Direct link to status-application-sourceHydrator-currentOperation-startedAt") ##### `finishedAt` object [​](#status-application-sourceHydrator-currentOperation-finishedAt "Direct link to status-application-sourceHydrator-currentOperation-finishedAt") ##### `phase` required string [​](#status-application-sourceHydrator-currentOperation-phase "Direct link to status-application-sourceHydrator-currentOperation-phase") ##### `message` required string [​](#status-application-sourceHydrator-currentOperation-message "Direct link to status-application-sourceHydrator-currentOperation-message") ##### `drySHA` string [​](#status-application-sourceHydrator-currentOperation-drySHA "Direct link to status-application-sourceHydrator-currentOperation-drySHA") ##### `hydratedSHA` string [​](#status-application-sourceHydrator-currentOperation-hydratedSHA "Direct link to status-application-sourceHydrator-currentOperation-hydratedSHA") ##### `sourceHydrator` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator") ##### `drySource` required object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource") ##### `repoURL` required string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-repoURL "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-repoURL") ##### `targetRevision` required string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-targetRevision "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-targetRevision") ##### `path` required string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-path "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-path") ##### `helm` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm") ##### `valueFiles` string\[] [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-valueFiles "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-valueFiles") ##### `parameters` object\[] [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-parameters "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-parameters") ##### `name` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-parameters-name "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-parameters-name") ##### `value` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-parameters-value "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-parameters-value") ##### `forceString` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-parameters-forceString "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-parameters-forceString") ##### `releaseName` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-releaseName "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-releaseName") ##### `values` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-values "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-values") ##### `fileParameters` object\[] [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-fileParameters "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-fileParameters") ##### `name` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-fileParameters-name "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-fileParameters-name") ##### `path` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-fileParameters-path "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-fileParameters-path") ##### `version` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-version "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-version") ##### `passCredentials` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-passCredentials "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-passCredentials") ##### `ignoreMissingValueFiles` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-ignoreMissingValueFiles "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-ignoreMissingValueFiles") ##### `skipCrds` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-skipCrds "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-skipCrds") ##### `valuesObject` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-valuesObject "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-valuesObject") ##### `namespace` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-namespace "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-namespace") ##### `kubeVersion` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-kubeVersion "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-kubeVersion") ##### `apiVersions` string\[] [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-apiVersions "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-apiVersions") ##### `skipTests` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-skipTests "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-skipTests") ##### `skipSchemaValidation` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-skipSchemaValidation "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-helm-skipSchemaValidation") ##### `kustomize` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize") ##### `namePrefix` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-namePrefix "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-namePrefix") ##### `nameSuffix` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-nameSuffix "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-nameSuffix") ##### `images` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-images "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-images") ##### `commonLabels` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-commonLabels "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-commonLabels") ##### `version` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-version "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-version") ##### `commonAnnotations` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-commonAnnotations "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-commonAnnotations") ##### `forceCommonLabels` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-forceCommonLabels "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-forceCommonLabels") ##### `forceCommonAnnotations` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-forceCommonAnnotations "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-forceCommonAnnotations") ##### `namespace` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-namespace "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-namespace") ##### `commonAnnotationsEnvsubst` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-commonAnnotationsEnvsubst "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-commonAnnotationsEnvsubst") ##### `replicas` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-replicas "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-replicas") ##### `patches` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-patches "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-patches") ##### `components` string\[] [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-components "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-components") ##### `ignoreMissingComponents` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-ignoreMissingComponents "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-ignoreMissingComponents") ##### `labelWithoutSelector` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-labelWithoutSelector "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-labelWithoutSelector") ##### `kubeVersion` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-kubeVersion "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-kubeVersion") ##### `apiVersions` string\[] [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-apiVersions "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-apiVersions") ##### `labelIncludeTemplates` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-labelIncludeTemplates "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-kustomize-labelIncludeTemplates") ##### `directory` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory") ##### `recurse` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-recurse "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-recurse") ##### `jsonnet` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet") ##### `extVars` object\[] [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-extVars "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-extVars") ##### `name` required string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-extVars-name "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-extVars-name") ##### `value` required string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-extVars-value "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-extVars-value") ##### `code` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-extVars-code "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-extVars-code") ##### `tlas` object\[] [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-tlas "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-tlas") ##### `name` required string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-tlas-name "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-tlas-name") ##### `value` required string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-tlas-value "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-tlas-value") ##### `code` boolean [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-tlas-code "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-tlas-code") ##### `libs` string\[] [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-libs "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-jsonnet-libs") ##### `exclude` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-exclude "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-exclude") ##### `include` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-include "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-directory-include") ##### `plugin` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-plugin "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-plugin") ##### `name` string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-plugin-name "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-plugin-name") ##### `env` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-plugin-env "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-plugin-env") ##### `parameters` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-plugin-parameters "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-drySource-plugin-parameters") ##### `syncSource` required object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-syncSource "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-syncSource") ##### `targetBranch` required string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-syncSource-targetBranch "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-syncSource-targetBranch") ##### `path` required string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-syncSource-path "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-syncSource-path") ##### `hydrateTo` object [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-hydrateTo "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-hydrateTo") ##### `targetBranch` required string [​](#status-application-sourceHydrator-currentOperation-sourceHydrator-hydrateTo-targetBranch "Direct link to status-application-sourceHydrator-currentOperation-sourceHydrator-hydrateTo-targetBranch") ## Retrieve: ArgoCDApplications[​](#retrieve-argocdapplications "Direct link to Retrieve: ArgoCDApplications") You can either use curl or kubectl to retrieve ArgoCDApplications. * kubectl * curl ### Retrieve a list of ArgoCDApplications[​](#retrieve-a-list-of-argocdapplications "Direct link to Retrieve a list of ArgoCDApplications") Run the following command to list all ArgoCDApplications in project `my-project`: ``` kubectl get argocdapplications.management.loft.sh -n p-my-project -o yaml ``` ### Retrieve a single ArgoCDApplication by name[​](#retrieve-a-single-argocdapplication-by-name "Direct link to Retrieve a single ArgoCDApplication by name") Run the following kubectl command to get ArgoCDApplication `shared-infra` in project `my-project`: ``` kubectl get argocdapplications.management.loft.sh shared-infra -n p-my-project -o yaml ``` ### Retrieve a list of ArgoCDApplications[​](#retrieve-a-list-of-argocdapplications-1 "Direct link to Retrieve a list of ArgoCDApplications") Run the following curl command to list all ArgoCDApplications in project `my-project`: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/argocdapplications" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single ArgoCDApplication by name[​](#get-a-single-argocdapplication-by-name "Direct link to Get a single ArgoCDApplication by name") Run the following curl command to get ArgoCDApplication `shared-infra` in project `my-project`: ``` # Exchange shared-infra in the url below with the name of the ArgoCDApplication curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/argocdapplications/shared-infra" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: ArgoCDApplication[​](#create-argocdapplication "Direct link to Create: ArgoCDApplication") You can either use curl or kubectl to create a new ArgoCDApplication. Make sure to set the project in the `metadata.namespace` field you want to create the ArgoCDApplication in. If your project has the id `my-project`, the corresponding namespace would be `p-my-project`. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplication metadata: name: shared-infra namespace: p-my-project spec: destination: cluster: name: loft-cluster displayName: Shared Infrastructure templateRef: name: infra-template status: {} ``` Then create the ArgoCDApplication `shared-infra` in project `my-project` with: ``` kubectl create -f object.yaml -n p-my-project ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplication metadata: name: shared-infra namespace: p-my-project spec: destination: cluster: name: loft-cluster displayName: Shared Infrastructure templateRef: name: infra-template status: {} ``` Run the following curl command to create a new ArgoCDApplication `shared-infra` in project `my-project`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/argocdapplications" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: ArgoCDApplication[​](#update-argocdapplication "Direct link to Update: ArgoCDApplication") You can either use curl or kubectl to update ArgoCDApplications. * kubectl * curl ### Update ArgoCDApplication[​](#update-argocdapplication "Direct link to Update ArgoCDApplication") Run the following command to update ArgoCDApplication `shared-infra` in project `my-project`: ``` kubectl edit argocdapplications.management.loft.sh shared-infra -n p-my-project ``` Then edit the object and upon save, kubectl will update the resource. ### Patch ArgoCDApplication[​](#patch-argocdapplication "Direct link to Patch ArgoCDApplication") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the ArgoCDApplication `shared-infra` in project `my-project` via a patch: ``` kubectl patch argocdapplications.management.loft.sh shared-infra -n p-my-project \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update ArgoCDApplication[​](#update-argocdapplication-1 "Direct link to Update ArgoCDApplication") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplication metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: shared-infra namespace: p-my-project resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: destination: cluster: name: loft-cluster displayName: Shared Infrastructure templateRef: name: infra-template status: {} ``` Run the following curl command to update a single ArgoCDApplication `shared-infra` in project `my-project`: ``` # Replace the shared-infra in the url below with the name of the ArgoCDApplication you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/argocdapplications/shared-infra" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch ArgoCDApplication[​](#patch-argocdapplication-1 "Direct link to Patch ArgoCDApplication") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the ArgoCDApplication `shared-infra` in project `my-project` via a patch: ``` # Replace the shared-infra in the url below with the name of the ArgoCDApplication you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/argocdapplications/shared-infra" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: ArgoCDApplication[​](#delete-argocdapplication "Direct link to Delete: ArgoCDApplication") You can either use curl or kubectl to delete ArgoCDApplications. * kubectl * curl Run the following command to delete ArgoCDApplication `shared-infra` in project `my-project`: ``` kubectl delete argocdapplications.management.loft.sh shared-infra -n p-my-project ``` Run the following curl command to delete ArgoCDApplication `shared-infra` in project `my-project`: ``` # Replace the shared-infra in the url below with the name of the ArgoCDApplication you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/argocdapplications/shared-infra" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # ArgoCD Application Template ArgoCDApplicationTemplate is a reusable Argo CD ApplicationSpec that can be referenced by name from an ArgoCDApplication to avoid duplicating configuration across tenant clusters. ## ArgoCDApplicationTemplate example[​](#argocdapplicationtemplate-example "Direct link to ArgoCDApplicationTemplate example") An example ArgoCDApplicationTemplate: ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplicationTemplate metadata: name: app-template spec: template: metadata: {} spec: destination: {} project: default source: path: app repoURL: https://github.com/acme/app-charts targetRevision: main syncPolicy: automated: prune: true selfHeal: true status: {} ``` ## ArgoCDApplicationTemplate reference[​](#argocdapplicationtemplate-reference "Direct link to ArgoCDApplicationTemplate reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes an OS image ### `template` object [​](#spec-template "Direct link to spec-template")Template is the blueprint template definition #### `metadata` object [​](#spec-template-metadata "Direct link to spec-template-metadata") ##### `labels` object [​](#spec-template-metadata-labels "Direct link to spec-template-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-template-metadata-annotations "Direct link to spec-template-metadata-annotations")Annotations are annotations on the object #### `spec` object [​](#spec-template-spec "Direct link to spec-template-spec")Spec is the spec of the Argo CD application ##### `source` object [​](#spec-template-spec-source "Direct link to spec-template-spec-source") ##### `repoURL` required string [​](#spec-template-spec-source-repoURL "Direct link to spec-template-spec-source-repoURL") ##### `path` string [​](#spec-template-spec-source-path "Direct link to spec-template-spec-source-path") ##### `targetRevision` string [​](#spec-template-spec-source-targetRevision "Direct link to spec-template-spec-source-targetRevision") ##### `helm` object [​](#spec-template-spec-source-helm "Direct link to spec-template-spec-source-helm") ##### `valueFiles` string\[] [​](#spec-template-spec-source-helm-valueFiles "Direct link to spec-template-spec-source-helm-valueFiles") ##### `parameters` object\[] [​](#spec-template-spec-source-helm-parameters "Direct link to spec-template-spec-source-helm-parameters") ##### `name` string [​](#spec-template-spec-source-helm-parameters-name "Direct link to spec-template-spec-source-helm-parameters-name") ##### `value` string [​](#spec-template-spec-source-helm-parameters-value "Direct link to spec-template-spec-source-helm-parameters-value") ##### `forceString` boolean [​](#spec-template-spec-source-helm-parameters-forceString "Direct link to spec-template-spec-source-helm-parameters-forceString") ##### `releaseName` string [​](#spec-template-spec-source-helm-releaseName "Direct link to spec-template-spec-source-helm-releaseName") ##### `values` string [​](#spec-template-spec-source-helm-values "Direct link to spec-template-spec-source-helm-values") ##### `fileParameters` object\[] [​](#spec-template-spec-source-helm-fileParameters "Direct link to spec-template-spec-source-helm-fileParameters") ##### `name` string [​](#spec-template-spec-source-helm-fileParameters-name "Direct link to spec-template-spec-source-helm-fileParameters-name") ##### `path` string [​](#spec-template-spec-source-helm-fileParameters-path "Direct link to spec-template-spec-source-helm-fileParameters-path") ##### `version` string [​](#spec-template-spec-source-helm-version "Direct link to spec-template-spec-source-helm-version") ##### `passCredentials` boolean [​](#spec-template-spec-source-helm-passCredentials "Direct link to spec-template-spec-source-helm-passCredentials") ##### `ignoreMissingValueFiles` boolean [​](#spec-template-spec-source-helm-ignoreMissingValueFiles "Direct link to spec-template-spec-source-helm-ignoreMissingValueFiles") ##### `skipCrds` boolean [​](#spec-template-spec-source-helm-skipCrds "Direct link to spec-template-spec-source-helm-skipCrds") ##### `valuesObject` object [​](#spec-template-spec-source-helm-valuesObject "Direct link to spec-template-spec-source-helm-valuesObject") ##### `namespace` string [​](#spec-template-spec-source-helm-namespace "Direct link to spec-template-spec-source-helm-namespace") ##### `kubeVersion` string [​](#spec-template-spec-source-helm-kubeVersion "Direct link to spec-template-spec-source-helm-kubeVersion") ##### `apiVersions` string\[] [​](#spec-template-spec-source-helm-apiVersions "Direct link to spec-template-spec-source-helm-apiVersions") ##### `skipTests` boolean [​](#spec-template-spec-source-helm-skipTests "Direct link to spec-template-spec-source-helm-skipTests") ##### `skipSchemaValidation` boolean [​](#spec-template-spec-source-helm-skipSchemaValidation "Direct link to spec-template-spec-source-helm-skipSchemaValidation") ##### `kustomize` object [​](#spec-template-spec-source-kustomize "Direct link to spec-template-spec-source-kustomize") ##### `namePrefix` string [​](#spec-template-spec-source-kustomize-namePrefix "Direct link to spec-template-spec-source-kustomize-namePrefix") ##### `nameSuffix` string [​](#spec-template-spec-source-kustomize-nameSuffix "Direct link to spec-template-spec-source-kustomize-nameSuffix") ##### `images` object [​](#spec-template-spec-source-kustomize-images "Direct link to spec-template-spec-source-kustomize-images") ##### `commonLabels` object [​](#spec-template-spec-source-kustomize-commonLabels "Direct link to spec-template-spec-source-kustomize-commonLabels") ##### `version` string [​](#spec-template-spec-source-kustomize-version "Direct link to spec-template-spec-source-kustomize-version") ##### `commonAnnotations` object [​](#spec-template-spec-source-kustomize-commonAnnotations "Direct link to spec-template-spec-source-kustomize-commonAnnotations") ##### `forceCommonLabels` boolean [​](#spec-template-spec-source-kustomize-forceCommonLabels "Direct link to spec-template-spec-source-kustomize-forceCommonLabels") ##### `forceCommonAnnotations` boolean [​](#spec-template-spec-source-kustomize-forceCommonAnnotations "Direct link to spec-template-spec-source-kustomize-forceCommonAnnotations") ##### `namespace` string [​](#spec-template-spec-source-kustomize-namespace "Direct link to spec-template-spec-source-kustomize-namespace") ##### `commonAnnotationsEnvsubst` boolean [​](#spec-template-spec-source-kustomize-commonAnnotationsEnvsubst "Direct link to spec-template-spec-source-kustomize-commonAnnotationsEnvsubst") ##### `replicas` object [​](#spec-template-spec-source-kustomize-replicas "Direct link to spec-template-spec-source-kustomize-replicas") ##### `patches` object [​](#spec-template-spec-source-kustomize-patches "Direct link to spec-template-spec-source-kustomize-patches") ##### `components` string\[] [​](#spec-template-spec-source-kustomize-components "Direct link to spec-template-spec-source-kustomize-components") ##### `ignoreMissingComponents` boolean [​](#spec-template-spec-source-kustomize-ignoreMissingComponents "Direct link to spec-template-spec-source-kustomize-ignoreMissingComponents") ##### `labelWithoutSelector` boolean [​](#spec-template-spec-source-kustomize-labelWithoutSelector "Direct link to spec-template-spec-source-kustomize-labelWithoutSelector") ##### `kubeVersion` string [​](#spec-template-spec-source-kustomize-kubeVersion "Direct link to spec-template-spec-source-kustomize-kubeVersion") ##### `apiVersions` string\[] [​](#spec-template-spec-source-kustomize-apiVersions "Direct link to spec-template-spec-source-kustomize-apiVersions") ##### `labelIncludeTemplates` boolean [​](#spec-template-spec-source-kustomize-labelIncludeTemplates "Direct link to spec-template-spec-source-kustomize-labelIncludeTemplates") ##### `directory` object [​](#spec-template-spec-source-directory "Direct link to spec-template-spec-source-directory") ##### `recurse` boolean [​](#spec-template-spec-source-directory-recurse "Direct link to spec-template-spec-source-directory-recurse") ##### `jsonnet` object [​](#spec-template-spec-source-directory-jsonnet "Direct link to spec-template-spec-source-directory-jsonnet") ##### `extVars` object\[] [​](#spec-template-spec-source-directory-jsonnet-extVars "Direct link to spec-template-spec-source-directory-jsonnet-extVars") ##### `name` required string [​](#spec-template-spec-source-directory-jsonnet-extVars-name "Direct link to spec-template-spec-source-directory-jsonnet-extVars-name") ##### `value` required string [​](#spec-template-spec-source-directory-jsonnet-extVars-value "Direct link to spec-template-spec-source-directory-jsonnet-extVars-value") ##### `code` boolean [​](#spec-template-spec-source-directory-jsonnet-extVars-code "Direct link to spec-template-spec-source-directory-jsonnet-extVars-code") ##### `tlas` object\[] [​](#spec-template-spec-source-directory-jsonnet-tlas "Direct link to spec-template-spec-source-directory-jsonnet-tlas") ##### `name` required string [​](#spec-template-spec-source-directory-jsonnet-tlas-name "Direct link to spec-template-spec-source-directory-jsonnet-tlas-name") ##### `value` required string [​](#spec-template-spec-source-directory-jsonnet-tlas-value "Direct link to spec-template-spec-source-directory-jsonnet-tlas-value") ##### `code` boolean [​](#spec-template-spec-source-directory-jsonnet-tlas-code "Direct link to spec-template-spec-source-directory-jsonnet-tlas-code") ##### `libs` string\[] [​](#spec-template-spec-source-directory-jsonnet-libs "Direct link to spec-template-spec-source-directory-jsonnet-libs") ##### `exclude` string [​](#spec-template-spec-source-directory-exclude "Direct link to spec-template-spec-source-directory-exclude") ##### `include` string [​](#spec-template-spec-source-directory-include "Direct link to spec-template-spec-source-directory-include") ##### `plugin` object [​](#spec-template-spec-source-plugin "Direct link to spec-template-spec-source-plugin") ##### `name` string [​](#spec-template-spec-source-plugin-name "Direct link to spec-template-spec-source-plugin-name") ##### `env` object [​](#spec-template-spec-source-plugin-env "Direct link to spec-template-spec-source-plugin-env") ##### `parameters` object [​](#spec-template-spec-source-plugin-parameters "Direct link to spec-template-spec-source-plugin-parameters") ##### `chart` string [​](#spec-template-spec-source-chart "Direct link to spec-template-spec-source-chart") ##### `ref` string [​](#spec-template-spec-source-ref "Direct link to spec-template-spec-source-ref") ##### `name` string [​](#spec-template-spec-source-name "Direct link to spec-template-spec-source-name") ##### `destination` required object [​](#spec-template-spec-destination "Direct link to spec-template-spec-destination") ##### `server` string [​](#spec-template-spec-destination-server "Direct link to spec-template-spec-destination-server") ##### `namespace` string [​](#spec-template-spec-destination-namespace "Direct link to spec-template-spec-destination-namespace") ##### `name` string [​](#spec-template-spec-destination-name "Direct link to spec-template-spec-destination-name") ##### `project` required string [​](#spec-template-spec-project "Direct link to spec-template-spec-project") ##### `syncPolicy` object [​](#spec-template-spec-syncPolicy "Direct link to spec-template-spec-syncPolicy") ##### `automated` object [​](#spec-template-spec-syncPolicy-automated "Direct link to spec-template-spec-syncPolicy-automated") ##### `prune` boolean [​](#spec-template-spec-syncPolicy-automated-prune "Direct link to spec-template-spec-syncPolicy-automated-prune") ##### `selfHeal` boolean [​](#spec-template-spec-syncPolicy-automated-selfHeal "Direct link to spec-template-spec-syncPolicy-automated-selfHeal") ##### `allowEmpty` boolean [​](#spec-template-spec-syncPolicy-automated-allowEmpty "Direct link to spec-template-spec-syncPolicy-automated-allowEmpty") ##### `enabled` boolean [​](#spec-template-spec-syncPolicy-automated-enabled "Direct link to spec-template-spec-syncPolicy-automated-enabled") ##### `syncOptions` object [​](#spec-template-spec-syncPolicy-syncOptions "Direct link to spec-template-spec-syncPolicy-syncOptions") ##### `retry` object [​](#spec-template-spec-syncPolicy-retry "Direct link to spec-template-spec-syncPolicy-retry") ##### `limit` integer [​](#spec-template-spec-syncPolicy-retry-limit "Direct link to spec-template-spec-syncPolicy-retry-limit") ##### `backoff` object [​](#spec-template-spec-syncPolicy-retry-backoff "Direct link to spec-template-spec-syncPolicy-retry-backoff") ##### `duration` string [​](#spec-template-spec-syncPolicy-retry-backoff-duration "Direct link to spec-template-spec-syncPolicy-retry-backoff-duration") ##### `factor` integer [​](#spec-template-spec-syncPolicy-retry-backoff-factor "Direct link to spec-template-spec-syncPolicy-retry-backoff-factor") ##### `maxDuration` string [​](#spec-template-spec-syncPolicy-retry-backoff-maxDuration "Direct link to spec-template-spec-syncPolicy-retry-backoff-maxDuration") ##### `refresh` boolean [​](#spec-template-spec-syncPolicy-retry-refresh "Direct link to spec-template-spec-syncPolicy-retry-refresh") ##### `managedNamespaceMetadata` object [​](#spec-template-spec-syncPolicy-managedNamespaceMetadata "Direct link to spec-template-spec-syncPolicy-managedNamespaceMetadata") ##### `labels` object [​](#spec-template-spec-syncPolicy-managedNamespaceMetadata-labels "Direct link to spec-template-spec-syncPolicy-managedNamespaceMetadata-labels") ##### `annotations` object [​](#spec-template-spec-syncPolicy-managedNamespaceMetadata-annotations "Direct link to spec-template-spec-syncPolicy-managedNamespaceMetadata-annotations") ##### `ignoreDifferences` object [​](#spec-template-spec-ignoreDifferences "Direct link to spec-template-spec-ignoreDifferences") ##### `info` object\[] [​](#spec-template-spec-info "Direct link to spec-template-spec-info") ##### `name` required string [​](#spec-template-spec-info-name "Direct link to spec-template-spec-info-name") ##### `value` required string [​](#spec-template-spec-info-value "Direct link to spec-template-spec-info-value") ##### `revisionHistoryLimit` integer [​](#spec-template-spec-revisionHistoryLimit "Direct link to spec-template-spec-revisionHistoryLimit") ##### `sources` object [​](#spec-template-spec-sources "Direct link to spec-template-spec-sources") ##### `sourceHydrator` object [​](#spec-template-spec-sourceHydrator "Direct link to spec-template-spec-sourceHydrator") ##### `drySource` required object [​](#spec-template-spec-sourceHydrator-drySource "Direct link to spec-template-spec-sourceHydrator-drySource") ##### `repoURL` required string [​](#spec-template-spec-sourceHydrator-drySource-repoURL "Direct link to spec-template-spec-sourceHydrator-drySource-repoURL") ##### `targetRevision` required string [​](#spec-template-spec-sourceHydrator-drySource-targetRevision "Direct link to spec-template-spec-sourceHydrator-drySource-targetRevision") ##### `path` required string [​](#spec-template-spec-sourceHydrator-drySource-path "Direct link to spec-template-spec-sourceHydrator-drySource-path") ##### `helm` object [​](#spec-template-spec-sourceHydrator-drySource-helm "Direct link to spec-template-spec-sourceHydrator-drySource-helm") ##### `valueFiles` string\[] [​](#spec-template-spec-sourceHydrator-drySource-helm-valueFiles "Direct link to spec-template-spec-sourceHydrator-drySource-helm-valueFiles") ##### `parameters` object\[] [​](#spec-template-spec-sourceHydrator-drySource-helm-parameters "Direct link to spec-template-spec-sourceHydrator-drySource-helm-parameters") ##### `name` string [​](#spec-template-spec-sourceHydrator-drySource-helm-parameters-name "Direct link to spec-template-spec-sourceHydrator-drySource-helm-parameters-name") ##### `value` string [​](#spec-template-spec-sourceHydrator-drySource-helm-parameters-value "Direct link to spec-template-spec-sourceHydrator-drySource-helm-parameters-value") ##### `forceString` boolean [​](#spec-template-spec-sourceHydrator-drySource-helm-parameters-forceString "Direct link to spec-template-spec-sourceHydrator-drySource-helm-parameters-forceString") ##### `releaseName` string [​](#spec-template-spec-sourceHydrator-drySource-helm-releaseName "Direct link to spec-template-spec-sourceHydrator-drySource-helm-releaseName") ##### `values` string [​](#spec-template-spec-sourceHydrator-drySource-helm-values "Direct link to spec-template-spec-sourceHydrator-drySource-helm-values") ##### `fileParameters` object\[] [​](#spec-template-spec-sourceHydrator-drySource-helm-fileParameters "Direct link to spec-template-spec-sourceHydrator-drySource-helm-fileParameters") ##### `name` string [​](#spec-template-spec-sourceHydrator-drySource-helm-fileParameters-name "Direct link to spec-template-spec-sourceHydrator-drySource-helm-fileParameters-name") ##### `path` string [​](#spec-template-spec-sourceHydrator-drySource-helm-fileParameters-path "Direct link to spec-template-spec-sourceHydrator-drySource-helm-fileParameters-path") ##### `version` string [​](#spec-template-spec-sourceHydrator-drySource-helm-version "Direct link to spec-template-spec-sourceHydrator-drySource-helm-version") ##### `passCredentials` boolean [​](#spec-template-spec-sourceHydrator-drySource-helm-passCredentials "Direct link to spec-template-spec-sourceHydrator-drySource-helm-passCredentials") ##### `ignoreMissingValueFiles` boolean [​](#spec-template-spec-sourceHydrator-drySource-helm-ignoreMissingValueFiles "Direct link to spec-template-spec-sourceHydrator-drySource-helm-ignoreMissingValueFiles") ##### `skipCrds` boolean [​](#spec-template-spec-sourceHydrator-drySource-helm-skipCrds "Direct link to spec-template-spec-sourceHydrator-drySource-helm-skipCrds") ##### `valuesObject` object [​](#spec-template-spec-sourceHydrator-drySource-helm-valuesObject "Direct link to spec-template-spec-sourceHydrator-drySource-helm-valuesObject") ##### `namespace` string [​](#spec-template-spec-sourceHydrator-drySource-helm-namespace "Direct link to spec-template-spec-sourceHydrator-drySource-helm-namespace") ##### `kubeVersion` string [​](#spec-template-spec-sourceHydrator-drySource-helm-kubeVersion "Direct link to spec-template-spec-sourceHydrator-drySource-helm-kubeVersion") ##### `apiVersions` string\[] [​](#spec-template-spec-sourceHydrator-drySource-helm-apiVersions "Direct link to spec-template-spec-sourceHydrator-drySource-helm-apiVersions") ##### `skipTests` boolean [​](#spec-template-spec-sourceHydrator-drySource-helm-skipTests "Direct link to spec-template-spec-sourceHydrator-drySource-helm-skipTests") ##### `skipSchemaValidation` boolean [​](#spec-template-spec-sourceHydrator-drySource-helm-skipSchemaValidation "Direct link to spec-template-spec-sourceHydrator-drySource-helm-skipSchemaValidation") ##### `kustomize` object [​](#spec-template-spec-sourceHydrator-drySource-kustomize "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize") ##### `namePrefix` string [​](#spec-template-spec-sourceHydrator-drySource-kustomize-namePrefix "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-namePrefix") ##### `nameSuffix` string [​](#spec-template-spec-sourceHydrator-drySource-kustomize-nameSuffix "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-nameSuffix") ##### `images` object [​](#spec-template-spec-sourceHydrator-drySource-kustomize-images "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-images") ##### `commonLabels` object [​](#spec-template-spec-sourceHydrator-drySource-kustomize-commonLabels "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-commonLabels") ##### `version` string [​](#spec-template-spec-sourceHydrator-drySource-kustomize-version "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-version") ##### `commonAnnotations` object [​](#spec-template-spec-sourceHydrator-drySource-kustomize-commonAnnotations "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-commonAnnotations") ##### `forceCommonLabels` boolean [​](#spec-template-spec-sourceHydrator-drySource-kustomize-forceCommonLabels "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-forceCommonLabels") ##### `forceCommonAnnotations` boolean [​](#spec-template-spec-sourceHydrator-drySource-kustomize-forceCommonAnnotations "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-forceCommonAnnotations") ##### `namespace` string [​](#spec-template-spec-sourceHydrator-drySource-kustomize-namespace "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-namespace") ##### `commonAnnotationsEnvsubst` boolean [​](#spec-template-spec-sourceHydrator-drySource-kustomize-commonAnnotationsEnvsubst "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-commonAnnotationsEnvsubst") ##### `replicas` object [​](#spec-template-spec-sourceHydrator-drySource-kustomize-replicas "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-replicas") ##### `patches` object [​](#spec-template-spec-sourceHydrator-drySource-kustomize-patches "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-patches") ##### `components` string\[] [​](#spec-template-spec-sourceHydrator-drySource-kustomize-components "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-components") ##### `ignoreMissingComponents` boolean [​](#spec-template-spec-sourceHydrator-drySource-kustomize-ignoreMissingComponents "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-ignoreMissingComponents") ##### `labelWithoutSelector` boolean [​](#spec-template-spec-sourceHydrator-drySource-kustomize-labelWithoutSelector "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-labelWithoutSelector") ##### `kubeVersion` string [​](#spec-template-spec-sourceHydrator-drySource-kustomize-kubeVersion "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-kubeVersion") ##### `apiVersions` string\[] [​](#spec-template-spec-sourceHydrator-drySource-kustomize-apiVersions "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-apiVersions") ##### `labelIncludeTemplates` boolean [​](#spec-template-spec-sourceHydrator-drySource-kustomize-labelIncludeTemplates "Direct link to spec-template-spec-sourceHydrator-drySource-kustomize-labelIncludeTemplates") ##### `directory` object [​](#spec-template-spec-sourceHydrator-drySource-directory "Direct link to spec-template-spec-sourceHydrator-drySource-directory") ##### `recurse` boolean [​](#spec-template-spec-sourceHydrator-drySource-directory-recurse "Direct link to spec-template-spec-sourceHydrator-drySource-directory-recurse") ##### `jsonnet` object [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet") ##### `extVars` object\[] [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars") ##### `name` required string [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-name "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-name") ##### `value` required string [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-value "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-value") ##### `code` boolean [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-code "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-code") ##### `tlas` object\[] [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas") ##### `name` required string [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-name "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-name") ##### `value` required string [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-value "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-value") ##### `code` boolean [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-code "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-code") ##### `libs` string\[] [​](#spec-template-spec-sourceHydrator-drySource-directory-jsonnet-libs "Direct link to spec-template-spec-sourceHydrator-drySource-directory-jsonnet-libs") ##### `exclude` string [​](#spec-template-spec-sourceHydrator-drySource-directory-exclude "Direct link to spec-template-spec-sourceHydrator-drySource-directory-exclude") ##### `include` string [​](#spec-template-spec-sourceHydrator-drySource-directory-include "Direct link to spec-template-spec-sourceHydrator-drySource-directory-include") ##### `plugin` object [​](#spec-template-spec-sourceHydrator-drySource-plugin "Direct link to spec-template-spec-sourceHydrator-drySource-plugin") ##### `name` string [​](#spec-template-spec-sourceHydrator-drySource-plugin-name "Direct link to spec-template-spec-sourceHydrator-drySource-plugin-name") ##### `env` object [​](#spec-template-spec-sourceHydrator-drySource-plugin-env "Direct link to spec-template-spec-sourceHydrator-drySource-plugin-env") ##### `parameters` object [​](#spec-template-spec-sourceHydrator-drySource-plugin-parameters "Direct link to spec-template-spec-sourceHydrator-drySource-plugin-parameters") ##### `syncSource` required object [​](#spec-template-spec-sourceHydrator-syncSource "Direct link to spec-template-spec-sourceHydrator-syncSource") ##### `targetBranch` required string [​](#spec-template-spec-sourceHydrator-syncSource-targetBranch "Direct link to spec-template-spec-sourceHydrator-syncSource-targetBranch") ##### `path` required string [​](#spec-template-spec-sourceHydrator-syncSource-path "Direct link to spec-template-spec-sourceHydrator-syncSource-path") ##### `hydrateTo` object [​](#spec-template-spec-sourceHydrator-hydrateTo "Direct link to spec-template-spec-sourceHydrator-hydrateTo") ##### `targetBranch` required string [​](#spec-template-spec-sourceHydrator-hydrateTo-targetBranch "Direct link to spec-template-spec-sourceHydrator-hydrateTo-targetBranch") ### `parameters` object\[] [​](#spec-parameters "Direct link to spec-parameters")Parameters define additional app parameters that will set helm values #### `variable` string [​](#spec-parameters-variable "Direct link to spec-parameters-variable")Variable is the path of the variable. Can be foo or foo.bar for nested objects. #### `label` string [​](#spec-parameters-label "Direct link to spec-parameters-label")Label is the label to show for this parameter #### `description` string [​](#spec-parameters-description "Direct link to spec-parameters-description")Description is the description to show for this parameter #### `type` string [​](#spec-parameters-type "Direct link to spec-parameters-type")Type of the parameter. Can be one of: string, multiline, boolean, number and password #### `options` string\[] [​](#spec-parameters-options "Direct link to spec-parameters-options")Options is a slice of strings, where each string represents a mutually exclusive choice. #### `min` integer [​](#spec-parameters-min "Direct link to spec-parameters-min")Min is the minimum number if type is number #### `max` integer [​](#spec-parameters-max "Direct link to spec-parameters-max")Max is the maximum number if type is number #### `required` boolean [​](#spec-parameters-required "Direct link to spec-parameters-required")Required specifies if this parameter is required #### `defaultValue` string [​](#spec-parameters-defaultValue "Direct link to spec-parameters-defaultValue")DefaultValue is the default value if none is specified #### `placeholder` string [​](#spec-parameters-placeholder "Direct link to spec-parameters-placeholder")Placeholder shown in the UI #### `invalidation` string [​](#spec-parameters-invalidation "Direct link to spec-parameters-invalidation")Invalidation regex that if matched will reject the input #### `validation` string [​](#spec-parameters-validation "Direct link to spec-parameters-validation")Validation regex that if matched will allow the input #### `section` string [​](#spec-parameters-section "Direct link to spec-parameters-section")Section where this app should be displayed. Apps with the same section name will be grouped together ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ## `status` object [​](#status "Direct link to status") ## Retrieve: ArgoCDApplicationTemplates[​](#retrieve-argocdapplicationtemplates "Direct link to Retrieve: ArgoCDApplicationTemplates") You can either use curl or kubectl to retrieve ArgoCDApplicationTemplates. * kubectl * curl ### Retrieve a list of ArgoCDApplicationTemplates[​](#retrieve-a-list-of-argocdapplicationtemplates "Direct link to Retrieve a list of ArgoCDApplicationTemplates") Run the following command to list all ArgoCDApplicationTemplates: ``` kubectl get argocdapplicationtemplates.management.loft.sh -o yaml ``` ### Retrieve a single ArgoCDApplicationTemplate by name[​](#retrieve-a-single-argocdapplicationtemplate-by-name "Direct link to Retrieve a single ArgoCDApplicationTemplate by name") Run the following kubectl command to get ArgoCDApplicationTemplate `app-template`: ``` kubectl get argocdapplicationtemplates.management.loft.sh app-template -o yaml ``` ### Retrieve a list of ArgoCDApplicationTemplates[​](#retrieve-a-list-of-argocdapplicationtemplates-1 "Direct link to Retrieve a list of ArgoCDApplicationTemplates") Run the following curl command to list all ArgoCDApplicationTemplates: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/argocdapplicationtemplates" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single ArgoCDApplicationTemplate by name[​](#get-a-single-argocdapplicationtemplate-by-name "Direct link to Get a single ArgoCDApplicationTemplate by name") Run the following curl command to get ArgoCDApplicationTemplate `app-template`: ``` # Exchange app-template in the url below with the name of the ArgoCDApplicationTemplate curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/argocdapplicationtemplates/app-template" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: ArgoCDApplicationTemplate[​](#create-argocdapplicationtemplate "Direct link to Create: ArgoCDApplicationTemplate") You can either use curl or kubectl to create a new ArgoCDApplicationTemplate. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplicationTemplate metadata: name: app-template spec: template: metadata: {} spec: destination: {} project: default source: path: app repoURL: https://github.com/acme/app-charts targetRevision: main syncPolicy: automated: prune: true selfHeal: true status: {} ``` Then create the ArgoCDApplicationTemplate `app-template` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplicationTemplate metadata: name: app-template spec: template: metadata: {} spec: destination: {} project: default source: path: app repoURL: https://github.com/acme/app-charts targetRevision: main syncPolicy: automated: prune: true selfHeal: true status: {} ``` Run the following curl command to create a new ArgoCDApplicationTemplate `app-template`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/argocdapplicationtemplates" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: ArgoCDApplicationTemplate[​](#update-argocdapplicationtemplate "Direct link to Update: ArgoCDApplicationTemplate") You can either use curl or kubectl to update ArgoCDApplicationTemplates. * kubectl * curl ### Update ArgoCDApplicationTemplate[​](#update-argocdapplicationtemplate "Direct link to Update ArgoCDApplicationTemplate") Run the following command to update ArgoCDApplicationTemplate `app-template`: ``` kubectl edit argocdapplicationtemplates.management.loft.sh app-template ``` Then edit the object and upon save, kubectl will update the resource. ### Patch ArgoCDApplicationTemplate[​](#patch-argocdapplicationtemplate "Direct link to Patch ArgoCDApplicationTemplate") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the ArgoCDApplicationTemplate `app-template` via a patch: ``` kubectl patch argocdapplicationtemplates.management.loft.sh app-template \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update ArgoCDApplicationTemplate[​](#update-argocdapplicationtemplate-1 "Direct link to Update ArgoCDApplicationTemplate") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplicationTemplate metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: app-template resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: template: metadata: {} spec: destination: {} project: default source: path: app repoURL: https://github.com/acme/app-charts targetRevision: main syncPolicy: automated: prune: true selfHeal: true status: {} ``` Run the following curl command to update a single ArgoCDApplicationTemplate `app-template`: ``` # Replace the app-template in the url below with the name of the ArgoCDApplicationTemplate you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/argocdapplicationtemplates/app-template" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch ArgoCDApplicationTemplate[​](#patch-argocdapplicationtemplate-1 "Direct link to Patch ArgoCDApplicationTemplate") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the ArgoCDApplicationTemplate `app-template` via a patch: ``` # Replace the app-template in the url below with the name of the ArgoCDApplicationTemplate you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/argocdapplicationtemplates/app-template" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: ArgoCDApplicationTemplate[​](#delete-argocdapplicationtemplate "Direct link to Delete: ArgoCDApplicationTemplate") You can either use curl or kubectl to delete ArgoCDApplicationTemplates. * kubectl * curl Run the following command to delete ArgoCDApplicationTemplate `app-template`: ``` kubectl delete argocdapplicationtemplates.management.loft.sh app-template ``` Run the following curl command to delete ArgoCDApplicationTemplate `app-template`: ``` # Replace the app-template in the url below with the name of the ArgoCDApplicationTemplate you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/argocdapplicationtemplates/app-template" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Cluster Access Globally defined cluster access. You can allow users or teams to access certain clusters here and define their cluster roles in those clusters. ## Cluster Access example[​](#cluster-access-example "Direct link to Cluster Access example") An example Cluster Access: ``` apiVersion: management.loft.sh/v1 kind: ClusterAccess metadata: name: my-cluster-access spec: clusters: - '*' description: Defines cluster access for the global admins displayName: Global Admins localClusterAccessTemplate: metadata: {} spec: clusterRoles: - name: loft-cluster-admin priority: 1000000 users: - team: loft-admins status: {} ``` ## Cluster Access reference[​](#cluster-access-reference "Direct link to Cluster Access reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes a cluster access object ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `clusters` string\[] [​](#spec-clusters "Direct link to spec-clusters")Clusters are the clusters this template should be applied on. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ### `localClusterAccessTemplate` object [​](#spec-localClusterAccessTemplate "Direct link to spec-localClusterAccessTemplate")LocalClusterAccessTemplate holds the cluster access template #### `metadata` object [​](#spec-localClusterAccessTemplate-metadata "Direct link to spec-localClusterAccessTemplate-metadata")Metadata is the metadata of the cluster access object ##### `name` string [​](#spec-localClusterAccessTemplate-metadata-name "Direct link to spec-localClusterAccessTemplate-metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ##### `generateName` string [​](#spec-localClusterAccessTemplate-metadata-generateName "Direct link to spec-localClusterAccessTemplate-metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ##### `namespace` string [​](#spec-localClusterAccessTemplate-metadata-namespace "Direct link to spec-localClusterAccessTemplate-metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ##### `selfLink` string [​](#spec-localClusterAccessTemplate-metadata-selfLink "Direct link to spec-localClusterAccessTemplate-metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ##### `uid` string [​](#spec-localClusterAccessTemplate-metadata-uid "Direct link to spec-localClusterAccessTemplate-metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ##### `resourceVersion` string [​](#spec-localClusterAccessTemplate-metadata-resourceVersion "Direct link to spec-localClusterAccessTemplate-metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ##### `generation` integer [​](#spec-localClusterAccessTemplate-metadata-generation "Direct link to spec-localClusterAccessTemplate-metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ##### `creationTimestamp` object [​](#spec-localClusterAccessTemplate-metadata-creationTimestamp "Direct link to spec-localClusterAccessTemplate-metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ##### `deletionTimestamp` object [​](#spec-localClusterAccessTemplate-metadata-deletionTimestamp "Direct link to spec-localClusterAccessTemplate-metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ##### `deletionGracePeriodSeconds` integer [​](#spec-localClusterAccessTemplate-metadata-deletionGracePeriodSeconds "Direct link to spec-localClusterAccessTemplate-metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ##### `labels` object [​](#spec-localClusterAccessTemplate-metadata-labels "Direct link to spec-localClusterAccessTemplate-metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ##### `annotations` object [​](#spec-localClusterAccessTemplate-metadata-annotations "Direct link to spec-localClusterAccessTemplate-metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ##### `ownerReferences` object\[] [​](#spec-localClusterAccessTemplate-metadata-ownerReferences "Direct link to spec-localClusterAccessTemplate-metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. ##### `apiVersion` required string [​](#spec-localClusterAccessTemplate-metadata-ownerReferences-apiVersion "Direct link to spec-localClusterAccessTemplate-metadata-ownerReferences-apiVersion")API version of the referent. ##### `kind` required string [​](#spec-localClusterAccessTemplate-metadata-ownerReferences-kind "Direct link to spec-localClusterAccessTemplate-metadata-ownerReferences-kind")Kind of the referent. More info: ##### `name` required string [​](#spec-localClusterAccessTemplate-metadata-ownerReferences-name "Direct link to spec-localClusterAccessTemplate-metadata-ownerReferences-name")Name of the referent. More info: ##### `uid` required string [​](#spec-localClusterAccessTemplate-metadata-ownerReferences-uid "Direct link to spec-localClusterAccessTemplate-metadata-ownerReferences-uid")UID of the referent. More info: ##### `controller` boolean [​](#spec-localClusterAccessTemplate-metadata-ownerReferences-controller "Direct link to spec-localClusterAccessTemplate-metadata-ownerReferences-controller")If true, this reference points to the managing controller. ##### `blockOwnerDeletion` boolean [​](#spec-localClusterAccessTemplate-metadata-ownerReferences-blockOwnerDeletion "Direct link to spec-localClusterAccessTemplate-metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ##### `finalizers` string\[] [​](#spec-localClusterAccessTemplate-metadata-finalizers "Direct link to spec-localClusterAccessTemplate-metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ##### `managedFields` object\[] [​](#spec-localClusterAccessTemplate-metadata-managedFields "Direct link to spec-localClusterAccessTemplate-metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. ##### `manager` string [​](#spec-localClusterAccessTemplate-metadata-managedFields-manager "Direct link to spec-localClusterAccessTemplate-metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. ##### `operation` string [​](#spec-localClusterAccessTemplate-metadata-managedFields-operation "Direct link to spec-localClusterAccessTemplate-metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. ##### `apiVersion` string [​](#spec-localClusterAccessTemplate-metadata-managedFields-apiVersion "Direct link to spec-localClusterAccessTemplate-metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. ##### `time` object [​](#spec-localClusterAccessTemplate-metadata-managedFields-time "Direct link to spec-localClusterAccessTemplate-metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. ##### `fieldsType` string [​](#spec-localClusterAccessTemplate-metadata-managedFields-fieldsType "Direct link to spec-localClusterAccessTemplate-metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" ##### `fieldsV1` object [​](#spec-localClusterAccessTemplate-metadata-managedFields-fieldsV1 "Direct link to spec-localClusterAccessTemplate-metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. ##### `subresource` string [​](#spec-localClusterAccessTemplate-metadata-managedFields-subresource "Direct link to spec-localClusterAccessTemplate-metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. #### `spec` object [​](#spec-localClusterAccessTemplate-spec "Direct link to spec-localClusterAccessTemplate-spec")LocalClusterAccessSpec holds the spec of the cluster access in the cluster ##### `displayName` string [​](#spec-localClusterAccessTemplate-spec-displayName "Direct link to spec-localClusterAccessTemplate-spec-displayName")DisplayName is the name that should be shown in the UI ##### `description` string [​](#spec-localClusterAccessTemplate-spec-description "Direct link to spec-localClusterAccessTemplate-spec-description")Description is the description of this object in human-readable text. ##### `users` object\[] [​](#spec-localClusterAccessTemplate-spec-users "Direct link to spec-localClusterAccessTemplate-spec-users")Users are the users affected by this cluster access object ##### `user` string [​](#spec-localClusterAccessTemplate-spec-users-user "Direct link to spec-localClusterAccessTemplate-spec-users-user")User specifies a Loft user. ##### `team` string [​](#spec-localClusterAccessTemplate-spec-users-team "Direct link to spec-localClusterAccessTemplate-spec-users-team")Team specifies a Loft team. ##### `teams` string\[] [​](#spec-localClusterAccessTemplate-spec-teams "Direct link to spec-localClusterAccessTemplate-spec-teams")Teams are the teams affected by this cluster access object ##### `clusterRoles` object\[] [​](#spec-localClusterAccessTemplate-spec-clusterRoles "Direct link to spec-localClusterAccessTemplate-spec-clusterRoles")ClusterRoles define the cluster roles that the users should have assigned in the cluster. ##### `name` string [​](#spec-localClusterAccessTemplate-spec-clusterRoles-name "Direct link to spec-localClusterAccessTemplate-spec-clusterRoles-name")Name is the cluster role to assign ##### `priority` integer [​](#spec-localClusterAccessTemplate-spec-priority "Direct link to spec-localClusterAccessTemplate-spec-priority")Priority is a unique value that specifies the priority of this cluster access for the space constraints and quota. A higher priority means the cluster access object will override the space constraints of lower priority cluster access objects ## `status` object [​](#status "Direct link to status") ## Retrieve: Cluster Accesses[​](#retrieve-cluster-accesses "Direct link to Retrieve: Cluster Accesses") You can either use curl or kubectl to retrieve Cluster Accesses. * kubectl * curl ### Retrieve a list of Cluster Accesses[​](#retrieve-a-list-of-cluster-accesses "Direct link to Retrieve a list of Cluster Accesses") Run the following command to list all Cluster Accesses: ``` kubectl get clusteraccesses.management.loft.sh -o yaml ``` ### Retrieve a single Cluster Access by name[​](#retrieve-a-single-cluster-access-by-name "Direct link to Retrieve a single Cluster Access by name") Run the following kubectl command to get Cluster Access `my-cluster-access`: ``` kubectl get clusteraccesses.management.loft.sh my-cluster-access -o yaml ``` ### Retrieve a list of Cluster Accesses[​](#retrieve-a-list-of-cluster-accesses-1 "Direct link to Retrieve a list of Cluster Accesses") Run the following curl command to list all Cluster Accesses: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/clusteraccesses" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single Cluster Access by name[​](#get-a-single-cluster-access-by-name "Direct link to Get a single Cluster Access by name") Run the following curl command to get Cluster Access `my-cluster-access`: ``` # Exchange my-cluster-access in the url below with the name of the Cluster Access curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/clusteraccesses/my-cluster-access" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: Cluster Access[​](#create-cluster-access "Direct link to Create: Cluster Access") You can either use curl or kubectl to create a new Cluster Access. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: ClusterAccess metadata: name: my-cluster-access spec: clusters: - '*' description: Defines cluster access for the global admins displayName: Global Admins localClusterAccessTemplate: metadata: {} spec: clusterRoles: - name: loft-cluster-admin priority: 1000000 users: - team: loft-admins status: {} ``` Then create the Cluster Access `my-cluster-access` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: ClusterAccess metadata: name: my-cluster-access spec: clusters: - '*' description: Defines cluster access for the global admins displayName: Global Admins localClusterAccessTemplate: metadata: {} spec: clusterRoles: - name: loft-cluster-admin priority: 1000000 users: - team: loft-admins status: {} ``` Run the following curl command to create a new Cluster Access `my-cluster-access`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/clusteraccesses" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: Cluster Access[​](#update-cluster-access "Direct link to Update: Cluster Access") You can either use curl or kubectl to update Cluster Accesses. * kubectl * curl ### Update Cluster Access[​](#update-cluster-access "Direct link to Update Cluster Access") Run the following command to update Cluster Access `my-cluster-access`: ``` kubectl edit clusteraccesses.management.loft.sh my-cluster-access ``` Then edit the object and upon save, kubectl will update the resource. ### Patch Cluster Access[​](#patch-cluster-access "Direct link to Patch Cluster Access") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the Cluster Access `my-cluster-access` via a patch: ``` kubectl patch clusteraccesses.management.loft.sh my-cluster-access \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update Cluster Access[​](#update-cluster-access-1 "Direct link to Update Cluster Access") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: ClusterAccess metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-cluster-access resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: clusters: - '*' description: Defines cluster access for the global admins displayName: Global Admins localClusterAccessTemplate: metadata: {} spec: clusterRoles: - name: loft-cluster-admin priority: 1000000 users: - team: loft-admins status: {} ``` Run the following curl command to update a single Cluster Access `my-cluster-access`: ``` # Replace the my-cluster-access in the url below with the name of the Cluster Access you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/clusteraccesses/my-cluster-access" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch Cluster Access[​](#patch-cluster-access-1 "Direct link to Patch Cluster Access") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the Cluster Access `my-cluster-access` via a patch: ``` # Replace the my-cluster-access in the url below with the name of the Cluster Access you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/clusteraccesses/my-cluster-access" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: Cluster Access[​](#delete-cluster-access "Direct link to Delete: Cluster Access") You can either use curl or kubectl to delete Cluster Accesses. * kubectl * curl Run the following command to delete Cluster Access `my-cluster-access`: ``` kubectl delete clusteraccesses.management.loft.sh my-cluster-access ``` Run the following curl command to delete Cluster Access `my-cluster-access`: ``` # Replace the my-cluster-access in the url below with the name of the Cluster Access you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/clusteraccesses/my-cluster-access" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Cluster Role Template ClusterRoleTemplate holds the clusterRoleTemplate information ## ClusterRoleTemplate example[​](#clusterroletemplate-example "Direct link to ClusterRoleTemplate example") An example ClusterRoleTemplate: ``` apiVersion: management.loft.sh/v1 kind: ClusterRoleTemplate metadata: {} spec: access: - users: - '*' verbs: - get clusterRoleTemplate: metadata: {} rules: - apiGroups: - management.loft.sh resources: - spaceinstances - virtualclusterinstances verbs: - get - list - update - apiGroups: - management.loft.sh resources: - projectsecrets verbs: - get - list description: |- Allows the user or team to manage the project. Gives only access to existing space and virtual cluster objects but no creation or deletion privileges. displayName: Project User -No Create management: true status: {} ``` ## Policy rules[​](#policy-rules "Direct link to Policy rules") The `rules` field under `clusterRoleTemplate` defines RBAC permissions using standard Kubernetes PolicyRule objects. Each rule specifies which actions (verbs) are allowed on which resources. ### Verbs[​](#verbs "Direct link to Verbs") Verbs define the actions allowed on resources. Standard Kubernetes RBAC verbs include: | Verb | Description | | ------------------ | -------------------------------------------------------- | | `get` | Retrieve a single resource | | `list` | Retrieve a collection of resources | | `watch` | Watch for changes to resources | | `create` | Create a new resource | | `update` | Update an existing resource (replaces the entire object) | | `patch` | Partially modify an existing resource | | `delete` | Delete a single resource | | `deletecollection` | Delete a collection of resources | | `*` | Wildcard representing all verbs | ### API groups[​](#api-groups "Direct link to API groups") API groups define which API the resources belong to. Common API groups include: | API Group | Description | | --------------------------- | ---------------------------------------------------------------- | | `""` | Core API group (pods, services, configmaps, secrets, namespaces) | | `apps` | Deployments, DaemonSets, ReplicaSets, StatefulSets | | `batch` | Jobs, CronJobs | | `networking.k8s.io` | NetworkPolicies, Ingresses | | `rbac.authorization.k8s.io` | Roles, RoleBindings, ClusterRoles, ClusterRoleBindings | | `management.loft.sh` | vCluster Platform resources | | `storage.loft.sh` | vCluster Platform storage resources | | `*` | Wildcard matching all API groups | ### Platform resources[​](#platform-resources "Direct link to Platform resources") vCluster Platform resources in the `management.loft.sh` API group: | Resource | Description | | ------------------------- | --------------------------------- | | `announcements` | Platform announcements | | `apps` | Application configurations | | `backups` | Platform backups | | `clusteraccesses` | Cluster access permissions | | `clusterroletemplates` | Cluster role templates | | `clusters` | Connected clusters | | `configs` | Platform configuration | | `events` | Platform events | | `features` | Platform features | | `licenses` | Platform licenses | | `nodeclaims` | Node claims for auto-provisioning | | `nodeenvironments` | Node environment configurations | | `nodeproviders` | Node provider configurations | | `nodetypes` | Node type definitions | | `ownedaccesskeys` | User-owned access keys | | `projects` | Projects | | `projectsecrets` | Project-scoped secrets | | `selves` | Current user information | | `sharedsecrets` | Shared secrets | | `spaceinstances` | Space instances | | `spacetemplates` | Space templates | | `stackinstances` | Project-scoped Stack instances | | `stackinstances/outputs` | Published Stack outputs | | `stacktemplates` | Cluster-scoped Stack templates | | `tasks` | Platform tasks | | `teams` | Teams | | `users` | Users | | `virtualclusterinstances` | Tenant cluster instances | | `virtualclustertemplates` | Tenant cluster templates | Common subresources include `projects/members`, `projects/templates`, `clusters/members`, `stackinstances/outputs`, `virtualclusterinstances/kubeconfig`, and `virtualclusterinstances/log`. ### Resource names[​](#resource-names "Direct link to Resource names") The `resourceNames` field optionally restricts a rule to specific named resources. When empty, the rule applies to all resources of the specified type. ### Non-resource URLs[​](#non-resource-urls "Direct link to Non-resource URLs") The `nonResourceURLs` field specifies access to non-resource endpoints like `/healthz`, `/api`, `/apis`, and `/version`. Use `*` as a suffix to match paths (for example, `/healthz/*`). ## ClusterRoleTemplate reference[​](#clusterroletemplate-reference "Direct link to ClusterRoleTemplate reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes a cluster role template object ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `clusters` string\[] [​](#spec-clusters "Direct link to spec-clusters")Clusters are the clusters this template should be applied on. ### `management` boolean [​](#spec-management "Direct link to spec-management")Management defines if this cluster role should be created in the management instance. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ### `clusterRoleTemplate` object [​](#spec-clusterRoleTemplate "Direct link to spec-clusterRoleTemplate")ClusterRoleTemplate holds the cluster role template #### `metadata` object [​](#spec-clusterRoleTemplate-metadata "Direct link to spec-clusterRoleTemplate-metadata") ##### `name` string [​](#spec-clusterRoleTemplate-metadata-name "Direct link to spec-clusterRoleTemplate-metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ##### `generateName` string [​](#spec-clusterRoleTemplate-metadata-generateName "Direct link to spec-clusterRoleTemplate-metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ##### `namespace` string [​](#spec-clusterRoleTemplate-metadata-namespace "Direct link to spec-clusterRoleTemplate-metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ##### `selfLink` string [​](#spec-clusterRoleTemplate-metadata-selfLink "Direct link to spec-clusterRoleTemplate-metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ##### `uid` string [​](#spec-clusterRoleTemplate-metadata-uid "Direct link to spec-clusterRoleTemplate-metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ##### `resourceVersion` string [​](#spec-clusterRoleTemplate-metadata-resourceVersion "Direct link to spec-clusterRoleTemplate-metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ##### `generation` integer [​](#spec-clusterRoleTemplate-metadata-generation "Direct link to spec-clusterRoleTemplate-metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ##### `creationTimestamp` object [​](#spec-clusterRoleTemplate-metadata-creationTimestamp "Direct link to spec-clusterRoleTemplate-metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ##### `deletionTimestamp` object [​](#spec-clusterRoleTemplate-metadata-deletionTimestamp "Direct link to spec-clusterRoleTemplate-metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ##### `deletionGracePeriodSeconds` integer [​](#spec-clusterRoleTemplate-metadata-deletionGracePeriodSeconds "Direct link to spec-clusterRoleTemplate-metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ##### `labels` object [​](#spec-clusterRoleTemplate-metadata-labels "Direct link to spec-clusterRoleTemplate-metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ##### `annotations` object [​](#spec-clusterRoleTemplate-metadata-annotations "Direct link to spec-clusterRoleTemplate-metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ##### `ownerReferences` object\[] [​](#spec-clusterRoleTemplate-metadata-ownerReferences "Direct link to spec-clusterRoleTemplate-metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. ##### `apiVersion` required string [​](#spec-clusterRoleTemplate-metadata-ownerReferences-apiVersion "Direct link to spec-clusterRoleTemplate-metadata-ownerReferences-apiVersion")API version of the referent. ##### `kind` required string [​](#spec-clusterRoleTemplate-metadata-ownerReferences-kind "Direct link to spec-clusterRoleTemplate-metadata-ownerReferences-kind")Kind of the referent. More info: ##### `name` required string [​](#spec-clusterRoleTemplate-metadata-ownerReferences-name "Direct link to spec-clusterRoleTemplate-metadata-ownerReferences-name")Name of the referent. More info: ##### `uid` required string [​](#spec-clusterRoleTemplate-metadata-ownerReferences-uid "Direct link to spec-clusterRoleTemplate-metadata-ownerReferences-uid")UID of the referent. More info: ##### `controller` boolean [​](#spec-clusterRoleTemplate-metadata-ownerReferences-controller "Direct link to spec-clusterRoleTemplate-metadata-ownerReferences-controller")If true, this reference points to the managing controller. ##### `blockOwnerDeletion` boolean [​](#spec-clusterRoleTemplate-metadata-ownerReferences-blockOwnerDeletion "Direct link to spec-clusterRoleTemplate-metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ##### `finalizers` string\[] [​](#spec-clusterRoleTemplate-metadata-finalizers "Direct link to spec-clusterRoleTemplate-metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ##### `managedFields` object\[] [​](#spec-clusterRoleTemplate-metadata-managedFields "Direct link to spec-clusterRoleTemplate-metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. ##### `manager` string [​](#spec-clusterRoleTemplate-metadata-managedFields-manager "Direct link to spec-clusterRoleTemplate-metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. ##### `operation` string [​](#spec-clusterRoleTemplate-metadata-managedFields-operation "Direct link to spec-clusterRoleTemplate-metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. ##### `apiVersion` string [​](#spec-clusterRoleTemplate-metadata-managedFields-apiVersion "Direct link to spec-clusterRoleTemplate-metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. ##### `time` object [​](#spec-clusterRoleTemplate-metadata-managedFields-time "Direct link to spec-clusterRoleTemplate-metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. ##### `fieldsType` string [​](#spec-clusterRoleTemplate-metadata-managedFields-fieldsType "Direct link to spec-clusterRoleTemplate-metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" ##### `fieldsV1` object [​](#spec-clusterRoleTemplate-metadata-managedFields-fieldsV1 "Direct link to spec-clusterRoleTemplate-metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. ##### `subresource` string [​](#spec-clusterRoleTemplate-metadata-managedFields-subresource "Direct link to spec-clusterRoleTemplate-metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. #### `rules` object\[] [​](#spec-clusterRoleTemplate-rules "Direct link to spec-clusterRoleTemplate-rules")Rules holds all the PolicyRules for this ClusterRole ##### `verbs` required string\[] [​](#spec-clusterRoleTemplate-rules-verbs "Direct link to spec-clusterRoleTemplate-rules-verbs") ##### `apiGroups` string\[] [​](#spec-clusterRoleTemplate-rules-apiGroups "Direct link to spec-clusterRoleTemplate-rules-apiGroups") ##### `resources` string\[] [​](#spec-clusterRoleTemplate-rules-resources "Direct link to spec-clusterRoleTemplate-rules-resources") ##### `resourceNames` string\[] [​](#spec-clusterRoleTemplate-rules-resourceNames "Direct link to spec-clusterRoleTemplate-rules-resourceNames") ##### `nonResourceURLs` string\[] [​](#spec-clusterRoleTemplate-rules-nonResourceURLs "Direct link to spec-clusterRoleTemplate-rules-nonResourceURLs") #### `aggregationRule` object [​](#spec-clusterRoleTemplate-aggregationRule "Direct link to spec-clusterRoleTemplate-aggregationRule")AggregationRule is an optional field that describes how to build the Rules for this ClusterRole. If AggregationRule is set, then the Rules are controller managed and direct changes to Rules will be stomped by the controller. ##### `clusterRoleSelectors` object\[] [​](#spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors "Direct link to spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors") ##### `matchLabels` object [​](#spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchLabels "Direct link to spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchLabels")matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. ##### `matchExpressions` object\[] [​](#spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions "Direct link to spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions")matchExpressions is a list of label selector requirements. The requirements are ANDed. ##### `key` required string [​](#spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions-key "Direct link to spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions-key")key is the label key that the selector applies to. ##### `operator` required string [​](#spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions-operator "Direct link to spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions-operator")operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. ##### `values` string\[] [​](#spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions-values "Direct link to spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions-values")values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. ### `localClusterRoleTemplate` object [​](#spec-localClusterRoleTemplate "Direct link to spec-localClusterRoleTemplate")DEPRECATED: Use ClusterRoleTemplate instead LocalClusterRoleTemplate holds the cluster role template #### `metadata` object [​](#spec-localClusterRoleTemplate-metadata "Direct link to spec-localClusterRoleTemplate-metadata")Metadata is the metadata of the cluster role template object ##### `name` string [​](#spec-localClusterRoleTemplate-metadata-name "Direct link to spec-localClusterRoleTemplate-metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ##### `generateName` string [​](#spec-localClusterRoleTemplate-metadata-generateName "Direct link to spec-localClusterRoleTemplate-metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ##### `namespace` string [​](#spec-localClusterRoleTemplate-metadata-namespace "Direct link to spec-localClusterRoleTemplate-metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ##### `selfLink` string [​](#spec-localClusterRoleTemplate-metadata-selfLink "Direct link to spec-localClusterRoleTemplate-metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ##### `uid` string [​](#spec-localClusterRoleTemplate-metadata-uid "Direct link to spec-localClusterRoleTemplate-metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ##### `resourceVersion` string [​](#spec-localClusterRoleTemplate-metadata-resourceVersion "Direct link to spec-localClusterRoleTemplate-metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ##### `generation` integer [​](#spec-localClusterRoleTemplate-metadata-generation "Direct link to spec-localClusterRoleTemplate-metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ##### `creationTimestamp` object [​](#spec-localClusterRoleTemplate-metadata-creationTimestamp "Direct link to spec-localClusterRoleTemplate-metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ##### `deletionTimestamp` object [​](#spec-localClusterRoleTemplate-metadata-deletionTimestamp "Direct link to spec-localClusterRoleTemplate-metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ##### `deletionGracePeriodSeconds` integer [​](#spec-localClusterRoleTemplate-metadata-deletionGracePeriodSeconds "Direct link to spec-localClusterRoleTemplate-metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ##### `labels` object [​](#spec-localClusterRoleTemplate-metadata-labels "Direct link to spec-localClusterRoleTemplate-metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ##### `annotations` object [​](#spec-localClusterRoleTemplate-metadata-annotations "Direct link to spec-localClusterRoleTemplate-metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ##### `ownerReferences` object\[] [​](#spec-localClusterRoleTemplate-metadata-ownerReferences "Direct link to spec-localClusterRoleTemplate-metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. ##### `apiVersion` required string [​](#spec-localClusterRoleTemplate-metadata-ownerReferences-apiVersion "Direct link to spec-localClusterRoleTemplate-metadata-ownerReferences-apiVersion")API version of the referent. ##### `kind` required string [​](#spec-localClusterRoleTemplate-metadata-ownerReferences-kind "Direct link to spec-localClusterRoleTemplate-metadata-ownerReferences-kind")Kind of the referent. More info: ##### `name` required string [​](#spec-localClusterRoleTemplate-metadata-ownerReferences-name "Direct link to spec-localClusterRoleTemplate-metadata-ownerReferences-name")Name of the referent. More info: ##### `uid` required string [​](#spec-localClusterRoleTemplate-metadata-ownerReferences-uid "Direct link to spec-localClusterRoleTemplate-metadata-ownerReferences-uid")UID of the referent. More info: ##### `controller` boolean [​](#spec-localClusterRoleTemplate-metadata-ownerReferences-controller "Direct link to spec-localClusterRoleTemplate-metadata-ownerReferences-controller")If true, this reference points to the managing controller. ##### `blockOwnerDeletion` boolean [​](#spec-localClusterRoleTemplate-metadata-ownerReferences-blockOwnerDeletion "Direct link to spec-localClusterRoleTemplate-metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ##### `finalizers` string\[] [​](#spec-localClusterRoleTemplate-metadata-finalizers "Direct link to spec-localClusterRoleTemplate-metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ##### `managedFields` object\[] [​](#spec-localClusterRoleTemplate-metadata-managedFields "Direct link to spec-localClusterRoleTemplate-metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. ##### `manager` string [​](#spec-localClusterRoleTemplate-metadata-managedFields-manager "Direct link to spec-localClusterRoleTemplate-metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. ##### `operation` string [​](#spec-localClusterRoleTemplate-metadata-managedFields-operation "Direct link to spec-localClusterRoleTemplate-metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. ##### `apiVersion` string [​](#spec-localClusterRoleTemplate-metadata-managedFields-apiVersion "Direct link to spec-localClusterRoleTemplate-metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. ##### `time` object [​](#spec-localClusterRoleTemplate-metadata-managedFields-time "Direct link to spec-localClusterRoleTemplate-metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. ##### `fieldsType` string [​](#spec-localClusterRoleTemplate-metadata-managedFields-fieldsType "Direct link to spec-localClusterRoleTemplate-metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" ##### `fieldsV1` object [​](#spec-localClusterRoleTemplate-metadata-managedFields-fieldsV1 "Direct link to spec-localClusterRoleTemplate-metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. ##### `subresource` string [​](#spec-localClusterRoleTemplate-metadata-managedFields-subresource "Direct link to spec-localClusterRoleTemplate-metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. #### `spec` object [​](#spec-localClusterRoleTemplate-spec "Direct link to spec-localClusterRoleTemplate-spec")LocalClusterRoleTemplateSpec holds the spec of the cluster role template in the cluster ##### `displayName` string [​](#spec-localClusterRoleTemplate-spec-displayName "Direct link to spec-localClusterRoleTemplate-spec-displayName")DisplayName is the name that should be shown in the UI ##### `description` string [​](#spec-localClusterRoleTemplate-spec-description "Direct link to spec-localClusterRoleTemplate-spec-description")Description is the description of this object in human-readable text. ##### `clusterRoleTemplate` object [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate")ClusterRoleTemplate holds the cluster role template ##### `metadata` object [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata") ##### `name` string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-name "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ##### `generateName` string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-generateName "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ##### `namespace` string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-namespace "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ##### `selfLink` string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-selfLink "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ##### `uid` string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-uid "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ##### `resourceVersion` string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-resourceVersion "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ##### `generation` integer [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-generation "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ##### `creationTimestamp` object [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-creationTimestamp "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ##### `deletionTimestamp` object [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-deletionTimestamp "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ##### `deletionGracePeriodSeconds` integer [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-deletionGracePeriodSeconds "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ##### `labels` object [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-labels "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ##### `annotations` object [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-annotations "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ##### `ownerReferences` object\[] [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-ownerReferences "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. ##### `apiVersion` required string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-ownerReferences-apiVersion "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-ownerReferences-apiVersion")API version of the referent. ##### `kind` required string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-ownerReferences-kind "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-ownerReferences-kind")Kind of the referent. More info: ##### `name` required string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-ownerReferences-name "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-ownerReferences-name")Name of the referent. More info: ##### `uid` required string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-ownerReferences-uid "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-ownerReferences-uid")UID of the referent. More info: ##### `controller` boolean [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-ownerReferences-controller "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-ownerReferences-controller")If true, this reference points to the managing controller. ##### `blockOwnerDeletion` boolean [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-ownerReferences-blockOwnerDeletion "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ##### `finalizers` string\[] [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-finalizers "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ##### `managedFields` object\[] [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. ##### `manager` string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields-manager "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. ##### `operation` string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields-operation "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. ##### `apiVersion` string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields-apiVersion "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. ##### `time` object [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields-time "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. ##### `fieldsType` string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields-fieldsType "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" ##### `fieldsV1` object [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields-fieldsV1 "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. ##### `subresource` string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields-subresource "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ##### `rules` object\[] [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-rules "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-rules")Rules holds all the PolicyRules for this ClusterRole ##### `verbs` required string\[] [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-rules-verbs "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-rules-verbs") ##### `apiGroups` string\[] [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-rules-apiGroups "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-rules-apiGroups") ##### `resources` string\[] [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-rules-resources "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-rules-resources") ##### `resourceNames` string\[] [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-rules-resourceNames "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-rules-resourceNames") ##### `nonResourceURLs` string\[] [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-rules-nonResourceURLs "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-rules-nonResourceURLs") ##### `aggregationRule` object [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-aggregationRule "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-aggregationRule")AggregationRule is an optional field that describes how to build the Rules for this ClusterRole. If AggregationRule is set, then the Rules are controller managed and direct changes to Rules will be stomped by the controller. ##### `clusterRoleSelectors` object\[] [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors") ##### `matchLabels` object [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchLabels "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchLabels")matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. ##### `matchExpressions` object\[] [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions")matchExpressions is a list of label selector requirements. The requirements are ANDed. ##### `key` required string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions-key "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions-key")key is the label key that the selector applies to. ##### `operator` required string [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions-operator "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions-operator")operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. ##### `values` string\[] [​](#spec-localClusterRoleTemplate-spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions-values "Direct link to spec-localClusterRoleTemplate-spec-clusterRoleTemplate-aggregationRule-clusterRoleSelectors-matchExpressions-values")values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. ## `status` object [​](#status "Direct link to status") ### `clusters` object\[] [​](#status-clusters "Direct link to status-clusters") #### `name` string [​](#status-clusters-name "Direct link to status-clusters-name")Name is the kubernetes name of the object #### `displayName` string [​](#status-clusters-displayName "Direct link to status-clusters-displayName")The display name shown in the UI #### `icon` string [​](#status-clusters-icon "Direct link to status-clusters-icon")Icon is the icon of the user / team #### `username` string [​](#status-clusters-username "Direct link to status-clusters-username")The username that is used to login #### `email` string [​](#status-clusters-email "Direct link to status-clusters-email")The users email address #### `subject` string [​](#status-clusters-subject "Direct link to status-clusters-subject")The user subject ## Create: ClusterRoleTemplate[​](#create-clusterroletemplate "Direct link to Create: ClusterRoleTemplate") You can either use curl or kubectl to create a new ClusterRoleTemplate. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: ClusterRoleTemplate metadata: {} spec: access: - users: - '*' verbs: - get clusterRoleTemplate: metadata: {} rules: - apiGroups: - management.loft.sh resources: - spaceinstances - virtualclusterinstances verbs: - get - list - update - apiGroups: - management.loft.sh resources: - projectsecrets verbs: - get - list description: |- Allows the user or team to manage the project. Gives only access to existing space and virtual cluster objects but no creation or deletion privileges. displayName: Project User -No Create management: true status: {} ``` Then create the ClusterRoleTemplate `my-object` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: ClusterRoleTemplate metadata: {} spec: access: - users: - '*' verbs: - get clusterRoleTemplate: metadata: {} rules: - apiGroups: - management.loft.sh resources: - spaceinstances - virtualclusterinstances verbs: - get - list - update - apiGroups: - management.loft.sh resources: - projectsecrets verbs: - get - list description: |- Allows the user or team to manage the project. Gives only access to existing space and virtual cluster objects but no creation or deletion privileges. displayName: Project User -No Create management: true status: {} ``` Run the following curl command to create a new ClusterRoleTemplate `my-object`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/clusterroletemplates" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Cluster Connected Kubernetes clusters that can be managed through the platform. You can allow users and teams to access those clusters and they can create new spaces and virtual clusters inside them. ## Platform-specific verbs[​](#platform-specific-verbs "Direct link to Platform-specific verbs") In addition to standard Kubernetes RBAC verbs (`get`, `list`, `create`, `update`, `patch`, `delete`), the Cluster resource supports these platform-specific verbs for access control: | Verb | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `bind` | Required when adding a cluster to ClusterAccess rules. Allows granting users or teams permission to create spaces and tenant clusters on the cluster. | | `connectlocal` | Required when setting `spec.local: true` on a cluster. Allows connecting the local cluster (where the platform is installed) to the platform for management. | ### Example access rule with bind verb[​](#example-access-rule-with-bind-verb "Direct link to Example access rule with bind verb") ``` apiVersion: management.loft.sh/v1 kind: Team metadata: name: my-team spec: access: - name: cluster-access verbs: - get - bind subresources: - clusters teams: - my-team ``` ## Cluster example[​](#cluster-example "Direct link to Cluster example") An example Cluster: ``` apiVersion: management.loft.sh/v1 kind: Cluster metadata: name: my-cluster spec: config: secretName: my-kube-config-secret secretNamespace: my-kube-config-secret-namespace description: My AWS Cluster displayName: My Cluster status: {} ``` ## Cluster reference[​](#cluster-reference "Direct link to Cluster reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")If specified this name is displayed in the UI instead of the metadata name ### `description` string [​](#spec-description "Direct link to spec-description")Description describes a cluster access object ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `config` object [​](#spec-config "Direct link to spec-config")Holds a reference to a secret that holds the kube config to access this cluster #### `secretName` string [​](#spec-config-secretName "Direct link to spec-config-secretName") #### `secretNamespace` string [​](#spec-config-secretNamespace "Direct link to spec-config-secretNamespace") #### `key` string [​](#spec-config-key "Direct link to spec-config-key") ### `local` boolean [​](#spec-local "Direct link to spec-local")Local specifies if it is the local cluster that should be connected, when this is specified, config is optional ### `networkPeer` boolean [​](#spec-networkPeer "Direct link to spec-networkPeer")NetworkPeer specifies if the cluster is connected via tailscale, when this is specified, config is optional ### `managementNamespace` string [​](#spec-managementNamespace "Direct link to spec-managementNamespace")The namespace where the cluster components will be installed in ### `unusable` boolean [​](#spec-unusable "Direct link to spec-unusable")If unusable is true, no spaces or tenant clusters can be scheduled on this cluster. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ### `metrics` object [​](#spec-metrics "Direct link to spec-metrics")Metrics holds the cluster's metrics backend configuration #### `replicas` integer [​](#spec-metrics-replicas "Direct link to spec-metrics-replicas")Replicas is the number of desired replicas. #### `resources` object [​](#spec-metrics-resources "Direct link to spec-metrics-resources")Resources are compute resource required by the metrics backend ##### `limits` object [​](#spec-metrics-resources-limits "Direct link to spec-metrics-resources-limits")Limits describes the maximum amount of compute resources allowed. More info: ##### `requests` object [​](#spec-metrics-resources-requests "Direct link to spec-metrics-resources-requests")Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: ##### `claims` object\[] [​](#spec-metrics-resources-claims "Direct link to spec-metrics-resources-claims")Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.This field depends on the DynamicResourceAllocation feature gate.This field is immutable. It can only be set for containers. ##### `name` required string [​](#spec-metrics-resources-claims-name "Direct link to spec-metrics-resources-claims-name")Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. ##### `request` string [​](#spec-metrics-resources-claims-request "Direct link to spec-metrics-resources-claims-request")Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. #### `retention` string [​](#spec-metrics-retention "Direct link to spec-metrics-retention")Retention is the metrics data retention period. Default is 1y #### `storage` object [​](#spec-metrics-storage "Direct link to spec-metrics-storage") ##### `storageClass` string [​](#spec-metrics-storage-storageClass "Direct link to spec-metrics-storage-storageClass")StorageClass the storage class to use when provisioning the metrics backend's persistent volume If set to "-" or "" dynamic provisioning is disabled If set to undefined or null (the default), the cluster's default storage class is used for provisioning ##### `size` string [​](#spec-metrics-storage-size "Direct link to spec-metrics-storage-size")Size the size of the metrics backend's persistent volume ### `opencost` object [​](#spec-opencost "Direct link to spec-opencost")OpenCost holds the cluster's OpenCost backend configuration #### `replicas` integer [​](#spec-opencost-replicas "Direct link to spec-opencost-replicas")Replicas is the number of desired replicas. #### `resources` object [​](#spec-opencost-resources "Direct link to spec-opencost-resources")Resources are compute resource required by the OpenCost backend ##### `limits` object [​](#spec-opencost-resources-limits "Direct link to spec-opencost-resources-limits")Limits describes the maximum amount of compute resources allowed. More info: ##### `requests` object [​](#spec-opencost-resources-requests "Direct link to spec-opencost-resources-requests")Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: ##### `claims` object\[] [​](#spec-opencost-resources-claims "Direct link to spec-opencost-resources-claims")Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.This field depends on the DynamicResourceAllocation feature gate.This field is immutable. It can only be set for containers. ##### `name` required string [​](#spec-opencost-resources-claims-name "Direct link to spec-opencost-resources-claims-name")Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. ##### `request` string [​](#spec-opencost-resources-claims-request "Direct link to spec-opencost-resources-claims-request")Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. ### `argoCD` object [​](#spec-argoCD "Direct link to spec-argoCD")ArgoCD holds the cluster's argo cd configuration #### `enabled` boolean [​](#spec-argoCD-enabled "Direct link to spec-argoCD-enabled")Enabled defines if argo cd is enabled #### `connector` string [​](#spec-argoCD-connector "Direct link to spec-argoCD-connector")Connector specifies the argo cd connector name #### `cluster` object [​](#spec-argoCD-cluster "Direct link to spec-argoCD-cluster")Cluster holds settings for the cluster entry that is registered in argo cd ##### `metadata` object [​](#spec-argoCD-cluster-metadata "Direct link to spec-argoCD-cluster-metadata")Metadata is set on the argo cd cluster, so an ApplicationSet can select it. The platform's own loft.sh/cluster label wins over it. On Akuity, a key removed here stays on the cluster until it is removed in Akuity. ##### `labels` object [​](#spec-argoCD-cluster-metadata-labels "Direct link to spec-argoCD-cluster-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-argoCD-cluster-metadata-annotations "Direct link to spec-argoCD-cluster-metadata-annotations")Annotations are annotations on the object ## `status` object [​](#status "Direct link to status") ### `phase` string [​](#status-phase "Direct link to status-phase") ### `reason` string [​](#status-reason "Direct link to status-reason") ### `message` string [​](#status-message "Direct link to status-message") ### `conditions` object [​](#status-conditions "Direct link to status-conditions")Conditions holds several conditions the cluster might be in ## Retrieve: Clusters[​](#retrieve-clusters "Direct link to Retrieve: Clusters") You can either use curl or kubectl to retrieve Clusters. * kubectl * curl ### Retrieve a list of Clusters[​](#retrieve-a-list-of-clusters "Direct link to Retrieve a list of Clusters") Run the following command to list all Clusters: ``` kubectl get clusters.management.loft.sh -o yaml ``` ### Retrieve a single Cluster by name[​](#retrieve-a-single-cluster-by-name "Direct link to Retrieve a single Cluster by name") Run the following kubectl command to get Cluster `my-cluster`: ``` kubectl get clusters.management.loft.sh my-cluster -o yaml ``` ### Retrieve a list of Clusters[​](#retrieve-a-list-of-clusters-1 "Direct link to Retrieve a list of Clusters") Run the following curl command to list all Clusters: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/clusters" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single Cluster by name[​](#get-a-single-cluster-by-name "Direct link to Get a single Cluster by name") Run the following curl command to get Cluster `my-cluster`: ``` # Exchange my-cluster in the url below with the name of the Cluster curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/clusters/my-cluster" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: Cluster[​](#create-cluster "Direct link to Create: Cluster") You can either use curl or kubectl to create a new Cluster. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: Cluster metadata: name: my-cluster spec: config: secretName: my-kube-config-secret secretNamespace: my-kube-config-secret-namespace description: My AWS Cluster displayName: My Cluster status: {} ``` Then create the Cluster `my-cluster` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: Cluster metadata: name: my-cluster spec: config: secretName: my-kube-config-secret secretNamespace: my-kube-config-secret-namespace description: My AWS Cluster displayName: My Cluster status: {} ``` Run the following curl command to create a new Cluster `my-cluster`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/clusters" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: Cluster[​](#update-cluster "Direct link to Update: Cluster") You can either use curl or kubectl to update Clusters. * kubectl * curl ### Update Cluster[​](#update-cluster "Direct link to Update Cluster") Run the following command to update Cluster `my-cluster`: ``` kubectl edit clusters.management.loft.sh my-cluster ``` Then edit the object and upon save, kubectl will update the resource. ### Patch Cluster[​](#patch-cluster "Direct link to Patch Cluster") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the Cluster `my-cluster` via a patch: ``` kubectl patch clusters.management.loft.sh my-cluster \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update Cluster[​](#update-cluster-1 "Direct link to Update Cluster") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: Cluster metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-cluster resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: config: secretName: my-kube-config-secret secretNamespace: my-kube-config-secret-namespace description: My AWS Cluster displayName: My Cluster status: {} ``` Run the following curl command to update a single Cluster `my-cluster`: ``` # Replace the my-cluster in the url below with the name of the Cluster you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/clusters/my-cluster" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch Cluster[​](#patch-cluster-1 "Direct link to Patch Cluster") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the Cluster `my-cluster` via a patch: ``` # Replace the my-cluster in the url below with the name of the Cluster you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/clusters/my-cluster" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: Cluster[​](#delete-cluster "Direct link to Delete: Cluster") You can either use curl or kubectl to delete Clusters. * kubectl * curl Run the following command to delete Cluster `my-cluster`: ``` kubectl delete clusters.management.loft.sh my-cluster ``` Run the following curl command to delete Cluster `my-cluster`: ``` # Replace the my-cluster in the url below with the name of the Cluster you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/clusters/my-cluster" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Loft Config You can retrieve the platform config through this API. ## Config example[​](#config-example "Direct link to Config example") An example Config: ``` apiVersion: management.loft.sh/v1 kind: Config metadata: {} spec: {} status: auth: accessKeyMaxTTLSeconds: 36000 customHttpHeaders: X-My-Header: my-value github: clientId: my-client-id clientSecret: my-client-secret redirectURI: https://my-redirect-uri loginAccessKeyTTLSeconds: 60 ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `raw` string [​](#spec-raw "Direct link to spec-raw")Raw holds the raw config ## `status` object [​](#status "Direct link to status") ### `auth` object [​](#status-auth "Direct link to status-auth")Authentication holds the information for authentication #### `oidc` object [​](#status-auth-oidc "Direct link to status-auth-oidc")OIDC holds oidc authentication configuration ##### `issuerUrl` string [​](#status-auth-oidc-issuerUrl "Direct link to status-auth-oidc-issuerUrl")IssuerURL is the URL the provider signs ID Tokens as. This will be the "iss" field of all tokens produced by the provider and is used for configuration discovery.The URL is usually the provider's URL without a path, for example "" or "".The provider must implement configuration discovery. See: ##### `clientId` string [​](#status-auth-oidc-clientId "Direct link to status-auth-oidc-clientId")ClientID the JWT must be issued for, the "sub" field. This plugin only trusts a single client to ensure the plugin can be used with public providers.The plugin supports the "authorized party" OpenID Connect claim, which allows specialized providers to issue tokens to a client for a different client. See: ##### `clientSecret` string [​](#status-auth-oidc-clientSecret "Direct link to status-auth-oidc-clientSecret")ClientSecret to issue tokens from the OIDC provider ##### `redirectURI` string [​](#status-auth-oidc-redirectURI "Direct link to status-auth-oidc-redirectURI")loft redirect uri. E.g. ##### `postLogoutRedirectURI` string [​](#status-auth-oidc-postLogoutRedirectURI "Direct link to status-auth-oidc-postLogoutRedirectURI")Loft URI to be redirected to after successful logout by OIDC Provider ##### `caFile` string [​](#status-auth-oidc-caFile "Direct link to status-auth-oidc-caFile")Path to a PEM encoded root certificate of the provider. Optional ##### `insecureCa` boolean [​](#status-auth-oidc-insecureCa "Direct link to status-auth-oidc-insecureCa")Specify whether to communicate without validating SSL certificates ##### `preferredUsername` string [​](#status-auth-oidc-preferredUsername "Direct link to status-auth-oidc-preferredUsername")Configurable key which contains the preferred username claims ##### `loftUsernameClaim` string [​](#status-auth-oidc-loftUsernameClaim "Direct link to status-auth-oidc-loftUsernameClaim")LoftUsernameClaim is the JWT field to use as the user's username. ##### `usernameClaim` string [​](#status-auth-oidc-usernameClaim "Direct link to status-auth-oidc-usernameClaim")UsernameClaim is the JWT field to use as the user's id. ##### `emailClaim` string [​](#status-auth-oidc-emailClaim "Direct link to status-auth-oidc-emailClaim")EmailClaim is the JWT field to use as the user's email. ##### `allowedExtraClaims` string\[] [​](#status-auth-oidc-allowedExtraClaims "Direct link to status-auth-oidc-allowedExtraClaims")AllowedExtraClaims are claims of interest that are not part of User by default but may be provided by the OIDC provider. ##### `usernamePrefix` string [​](#status-auth-oidc-usernamePrefix "Direct link to status-auth-oidc-usernamePrefix")UsernamePrefix, if specified, causes claims mapping to username to be prefix with the provided value. A value "oidc:" would result in usernames like "oidc:john". ##### `groupsClaim` string [​](#status-auth-oidc-groupsClaim "Direct link to status-auth-oidc-groupsClaim")GroupsClaim, if specified, causes the OIDCAuthenticator to try to populate the user's groups with an ID Token field. If the GroupsClaim field is present in an ID Token the value must be a string or list of strings. ##### `groups` string\[] [​](#status-auth-oidc-groups "Direct link to status-auth-oidc-groups")If required groups is non empty, access is denied if the user is not part of at least one of the specified groups. ##### `scopes` string\[] [​](#status-auth-oidc-scopes "Direct link to status-auth-oidc-scopes")Scopes that should be sent to the server. If empty, defaults to "email" and "profile". ##### `getUserInfo` boolean [​](#status-auth-oidc-getUserInfo "Direct link to status-auth-oidc-getUserInfo")GetUserInfo, if specified, tells the OIDCAuthenticator to try to populate the user's information from the UserInfo. ##### `groupsPrefix` string [​](#status-auth-oidc-groupsPrefix "Direct link to status-auth-oidc-groupsPrefix")GroupsPrefix, if specified, causes claims mapping to group names to be prefixed with the value. A value "oidc:" would result in groups like "oidc:engineering" and "oidc:marketing". ##### `type` string [​](#status-auth-oidc-type "Direct link to status-auth-oidc-type")Type of the OIDC to show in the UI. Only for displaying purposes ##### `resource` string [​](#status-auth-oidc-resource "Direct link to status-auth-oidc-resource")Resource, if specified, is the value that is set for the "resource" URL parameter when making a request to the /token endpoint of the OIDC provider. #### `github` object [​](#status-auth-github "Direct link to status-auth-github")Github holds github authentication configuration ##### `clientId` string [​](#status-auth-github-clientId "Direct link to status-auth-github-clientId")ClientID holds the github client id ##### `clientSecret` required string [​](#status-auth-github-clientSecret "Direct link to status-auth-github-clientSecret")ClientID holds the github client secret ##### `redirectURI` required string [​](#status-auth-github-redirectURI "Direct link to status-auth-github-redirectURI")RedirectURI holds the redirect URI. Should be ##### `orgs` object\[] [​](#status-auth-github-orgs "Direct link to status-auth-github-orgs")Loft queries the following organizations for group information. Group claims are formatted as "(org):(team)". For example if a user is part of the "engineering" team of the "coreos" org, the group claim would include "coreos:engineering".If orgs are specified in the config then user MUST be a member of at least one of the specified orgs to authenticate with loft. ##### `name` string [​](#status-auth-github-orgs-name "Direct link to status-auth-github-orgs-name")Organization name in github (not slug, full name). Only users in this github organization can authenticate. ##### `teams` string\[] [​](#status-auth-github-orgs-teams "Direct link to status-auth-github-orgs-teams")Names of teams in a github organization. A user will be able to authenticate if they are members of at least one of these teams. Users in the organization can authenticate if this field is omitted from the config file. ##### `hostName` string [​](#status-auth-github-hostName "Direct link to status-auth-github-hostName")Required ONLY for GitHub Enterprise. This is the Hostname of the GitHub Enterprise account listed on the management console. Ensure this domain is routable on your network. ##### `rootCA` string [​](#status-auth-github-rootCA "Direct link to status-auth-github-rootCA")ONLY for GitHub Enterprise. Optional field. Used to support self-signed or untrusted CA root certificates. #### `gitlab` object [​](#status-auth-gitlab "Direct link to status-auth-gitlab")Gitlab holds gitlab authentication configuration ##### `clientId` required string [​](#status-auth-gitlab-clientId "Direct link to status-auth-gitlab-clientId")Gitlab client id ##### `clientSecret` required string [​](#status-auth-gitlab-clientSecret "Direct link to status-auth-gitlab-clientSecret")Gitlab client secret ##### `redirectURI` required string [​](#status-auth-gitlab-redirectURI "Direct link to status-auth-gitlab-redirectURI")Redirect URI ##### `baseURL` string [​](#status-auth-gitlab-baseURL "Direct link to status-auth-gitlab-baseURL")BaseURL is optional, default = ##### `groups` string\[] [​](#status-auth-gitlab-groups "Direct link to status-auth-gitlab-groups")Optional groups whitelist, communicated through the "groups" scope. If `groups` is omitted, all of the user's GitLab groups are returned. If `groups` is provided, this acts as a whitelist - only the user's GitLab groups that are in the configured `groups` below will go into the groups claim. Conversely, if the user is not in any of the configured `groups`, the user will not be authenticated. #### `google` object [​](#status-auth-google "Direct link to status-auth-google")Google holds google authentication configuration ##### `clientId` required string [​](#status-auth-google-clientId "Direct link to status-auth-google-clientId")Google client id ##### `clientSecret` required string [​](#status-auth-google-clientSecret "Direct link to status-auth-google-clientSecret")Google client secret ##### `redirectURI` required string [​](#status-auth-google-redirectURI "Direct link to status-auth-google-redirectURI")loft redirect uri. E.g. ##### `scopes` string\[] [​](#status-auth-google-scopes "Direct link to status-auth-google-scopes")defaults to "profile" and "email" ##### `hostedDomains` string\[] [​](#status-auth-google-hostedDomains "Direct link to status-auth-google-hostedDomains")Optional list of whitelisted domains If this field is nonempty, only users from a listed domain will be allowed to log in ##### `groups` string\[] [​](#status-auth-google-groups "Direct link to status-auth-google-groups")Optional list of whitelisted groups If this field is nonempty, only users from a listed group will be allowed to log in ##### `serviceAccountFilePath` string [​](#status-auth-google-serviceAccountFilePath "Direct link to status-auth-google-serviceAccountFilePath")Optional path to service account json If nonempty, and groups claim is made, will use authentication from file to check groups with the admin directory api ##### `adminEmail` string [​](#status-auth-google-adminEmail "Direct link to status-auth-google-adminEmail")Required if ServiceAccountFilePath The email of a GSuite super user which the service account will impersonate when listing groups #### `microsoft` object [​](#status-auth-microsoft "Direct link to status-auth-microsoft")Microsoft holds microsoft authentication configuration ##### `clientId` required string [​](#status-auth-microsoft-clientId "Direct link to status-auth-microsoft-clientId")Microsoft client id ##### `clientSecret` required string [​](#status-auth-microsoft-clientSecret "Direct link to status-auth-microsoft-clientSecret")Microsoft client secret ##### `redirectURI` required string [​](#status-auth-microsoft-redirectURI "Direct link to status-auth-microsoft-redirectURI")loft redirect uri. Usually ##### `tenant` string [​](#status-auth-microsoft-tenant "Direct link to status-auth-microsoft-tenant")tenant configuration parameter controls what kinds of accounts may be authenticated in loft. By default, all types of Microsoft accounts (consumers and organizations) can authenticate in loft via Microsoft. To change this, set the tenant parameter to one of the following:common - both personal and business/school accounts can authenticate in loft via Microsoft (default) consumers - only personal accounts can authenticate in loft organizations - only business/school accounts can authenticate in loft tenant uuid or tenant name - only accounts belonging to specific tenant identified by either tenant uuid or tenant name can authenticate in loft ##### `groups` string\[] [​](#status-auth-microsoft-groups "Direct link to status-auth-microsoft-groups")It is possible to require a user to be a member of a particular group in order to be successfully authenticated in loft. ##### `onlySecurityGroups` boolean [​](#status-auth-microsoft-onlySecurityGroups "Direct link to status-auth-microsoft-onlySecurityGroups")configuration option restricts the list to include only security groups. By default all groups (security, Office 365, mailing lists) are included. ##### `useGroupsAsWhitelist` boolean [​](#status-auth-microsoft-useGroupsAsWhitelist "Direct link to status-auth-microsoft-useGroupsAsWhitelist")Restrict the groups claims to include only the user’s groups that are in the configured groups #### `saml` object [​](#status-auth-saml "Direct link to status-auth-saml")SAML holds saml authentication configuration ##### `redirectURI` string [​](#status-auth-saml-redirectURI "Direct link to status-auth-saml-redirectURI")If the response assertion status value contains a Destination element, it must match this value exactly. Usually looks like ##### `ssoURL` string [​](#status-auth-saml-ssoURL "Direct link to status-auth-saml-ssoURL")SSO URL used for POST value. ##### `caData` string [​](#status-auth-saml-caData "Direct link to status-auth-saml-caData")CAData is a base64 encoded string that holds the ca certificate for validating the signature of the SAML response. Either CAData, CA or InsecureSkipSignatureValidation needs to be defined. ##### `usernameAttr` string [​](#status-auth-saml-usernameAttr "Direct link to status-auth-saml-usernameAttr")Name of attribute in the returned assertions to map to username ##### `emailAttr` string [​](#status-auth-saml-emailAttr "Direct link to status-auth-saml-emailAttr")Name of attribute in the returned assertions to map to email ##### `groupsAttr` string [​](#status-auth-saml-groupsAttr "Direct link to status-auth-saml-groupsAttr")Name of attribute in the returned assertions to map to groups ##### `ca` string [​](#status-auth-saml-ca "Direct link to status-auth-saml-ca")CA to use when validating the signature of the SAML response. ##### `insecureSkipSignatureValidation` boolean [​](#status-auth-saml-insecureSkipSignatureValidation "Direct link to status-auth-saml-insecureSkipSignatureValidation")Ignore the ca cert ##### `entityIssuer` string [​](#status-auth-saml-entityIssuer "Direct link to status-auth-saml-entityIssuer")When provided Loft will include this as the Issuer value during AuthnRequest. It will also override the redirectURI as the required audience when evaluating AudienceRestriction elements in the response. ##### `ssoIssuer` string [​](#status-auth-saml-ssoIssuer "Direct link to status-auth-saml-ssoIssuer")Issuer value expected in the SAML response. Optional. ##### `groupsDelim` string [​](#status-auth-saml-groupsDelim "Direct link to status-auth-saml-groupsDelim")If GroupsDelim is supplied the connector assumes groups are returned as a single string instead of multiple attribute values. This delimiter will be used split the groups string. ##### `allowedGroups` string\[] [​](#status-auth-saml-allowedGroups "Direct link to status-auth-saml-allowedGroups")List of groups to filter access based on membership ##### `filterGroups` boolean [​](#status-auth-saml-filterGroups "Direct link to status-auth-saml-filterGroups")If used with allowed groups, only forwards the allowed groups and not all groups specified. ##### `nameIDPolicyFormat` string [​](#status-auth-saml-nameIDPolicyFormat "Direct link to status-auth-saml-nameIDPolicyFormat")Requested format of the NameID. The NameID value is is mapped to the ID Token 'sub' claim.This can be an abbreviated form of the full URI with just the last component. For example, if this value is set to "emailAddress" the format will resolve to:urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddressIf no value is specified, this value defaults to:urn:oasis:names:tc:SAML:2.0:nameid-format:persistent #### `password` object [​](#status-auth-password "Direct link to status-auth-password")Password holds password authentication relevant information ##### `disabled` boolean [​](#status-auth-password-disabled "Direct link to status-auth-password-disabled")If true login via password is disabled #### `connectors` object\[] [​](#status-auth-connectors "Direct link to status-auth-connectors")Connectors are optional additional connectors for Loft. ##### `id` string [​](#status-auth-connectors-id "Direct link to status-auth-connectors-id")ID is the id that should show up in the url ##### `displayName` string [​](#status-auth-connectors-displayName "Direct link to status-auth-connectors-displayName")DisplayName is the name that should show up in the ui ##### `oidc` object [​](#status-auth-connectors-oidc "Direct link to status-auth-connectors-oidc")OIDC holds oidc authentication configuration ##### `issuerUrl` string [​](#status-auth-connectors-oidc-issuerUrl "Direct link to status-auth-connectors-oidc-issuerUrl")IssuerURL is the URL the provider signs ID Tokens as. This will be the "iss" field of all tokens produced by the provider and is used for configuration discovery.The URL is usually the provider's URL without a path, for example "" or "".The provider must implement configuration discovery. See: ##### `clientId` string [​](#status-auth-connectors-oidc-clientId "Direct link to status-auth-connectors-oidc-clientId")ClientID the JWT must be issued for, the "sub" field. This plugin only trusts a single client to ensure the plugin can be used with public providers.The plugin supports the "authorized party" OpenID Connect claim, which allows specialized providers to issue tokens to a client for a different client. See: ##### `clientSecret` string [​](#status-auth-connectors-oidc-clientSecret "Direct link to status-auth-connectors-oidc-clientSecret")ClientSecret to issue tokens from the OIDC provider ##### `redirectURI` string [​](#status-auth-connectors-oidc-redirectURI "Direct link to status-auth-connectors-oidc-redirectURI")loft redirect uri. E.g. ##### `postLogoutRedirectURI` string [​](#status-auth-connectors-oidc-postLogoutRedirectURI "Direct link to status-auth-connectors-oidc-postLogoutRedirectURI")Loft URI to be redirected to after successful logout by OIDC Provider ##### `caFile` string [​](#status-auth-connectors-oidc-caFile "Direct link to status-auth-connectors-oidc-caFile")Path to a PEM encoded root certificate of the provider. Optional ##### `insecureCa` boolean [​](#status-auth-connectors-oidc-insecureCa "Direct link to status-auth-connectors-oidc-insecureCa")Specify whether to communicate without validating SSL certificates ##### `preferredUsername` string [​](#status-auth-connectors-oidc-preferredUsername "Direct link to status-auth-connectors-oidc-preferredUsername")Configurable key which contains the preferred username claims ##### `loftUsernameClaim` string [​](#status-auth-connectors-oidc-loftUsernameClaim "Direct link to status-auth-connectors-oidc-loftUsernameClaim")LoftUsernameClaim is the JWT field to use as the user's username. ##### `usernameClaim` string [​](#status-auth-connectors-oidc-usernameClaim "Direct link to status-auth-connectors-oidc-usernameClaim")UsernameClaim is the JWT field to use as the user's id. ##### `emailClaim` string [​](#status-auth-connectors-oidc-emailClaim "Direct link to status-auth-connectors-oidc-emailClaim")EmailClaim is the JWT field to use as the user's email. ##### `allowedExtraClaims` string\[] [​](#status-auth-connectors-oidc-allowedExtraClaims "Direct link to status-auth-connectors-oidc-allowedExtraClaims")AllowedExtraClaims are claims of interest that are not part of User by default but may be provided by the OIDC provider. ##### `usernamePrefix` string [​](#status-auth-connectors-oidc-usernamePrefix "Direct link to status-auth-connectors-oidc-usernamePrefix")UsernamePrefix, if specified, causes claims mapping to username to be prefix with the provided value. A value "oidc:" would result in usernames like "oidc:john". ##### `groupsClaim` string [​](#status-auth-connectors-oidc-groupsClaim "Direct link to status-auth-connectors-oidc-groupsClaim")GroupsClaim, if specified, causes the OIDCAuthenticator to try to populate the user's groups with an ID Token field. If the GroupsClaim field is present in an ID Token the value must be a string or list of strings. ##### `groups` string\[] [​](#status-auth-connectors-oidc-groups "Direct link to status-auth-connectors-oidc-groups")If required groups is non empty, access is denied if the user is not part of at least one of the specified groups. ##### `scopes` string\[] [​](#status-auth-connectors-oidc-scopes "Direct link to status-auth-connectors-oidc-scopes")Scopes that should be sent to the server. If empty, defaults to "email" and "profile". ##### `getUserInfo` boolean [​](#status-auth-connectors-oidc-getUserInfo "Direct link to status-auth-connectors-oidc-getUserInfo")GetUserInfo, if specified, tells the OIDCAuthenticator to try to populate the user's information from the UserInfo. ##### `groupsPrefix` string [​](#status-auth-connectors-oidc-groupsPrefix "Direct link to status-auth-connectors-oidc-groupsPrefix")GroupsPrefix, if specified, causes claims mapping to group names to be prefixed with the value. A value "oidc:" would result in groups like "oidc:engineering" and "oidc:marketing". ##### `type` string [​](#status-auth-connectors-oidc-type "Direct link to status-auth-connectors-oidc-type")Type of the OIDC to show in the UI. Only for displaying purposes ##### `resource` string [​](#status-auth-connectors-oidc-resource "Direct link to status-auth-connectors-oidc-resource")Resource, if specified, is the value that is set for the "resource" URL parameter when making a request to the /token endpoint of the OIDC provider. ##### `github` object [​](#status-auth-connectors-github "Direct link to status-auth-connectors-github")Github holds github authentication configuration ##### `clientId` string [​](#status-auth-connectors-github-clientId "Direct link to status-auth-connectors-github-clientId")ClientID holds the github client id ##### `clientSecret` required string [​](#status-auth-connectors-github-clientSecret "Direct link to status-auth-connectors-github-clientSecret")ClientID holds the github client secret ##### `redirectURI` required string [​](#status-auth-connectors-github-redirectURI "Direct link to status-auth-connectors-github-redirectURI")RedirectURI holds the redirect URI. Should be ##### `orgs` object\[] [​](#status-auth-connectors-github-orgs "Direct link to status-auth-connectors-github-orgs")Loft queries the following organizations for group information. Group claims are formatted as "(org):(team)". For example if a user is part of the "engineering" team of the "coreos" org, the group claim would include "coreos:engineering".If orgs are specified in the config then user MUST be a member of at least one of the specified orgs to authenticate with loft. ##### `name` string [​](#status-auth-connectors-github-orgs-name "Direct link to status-auth-connectors-github-orgs-name")Organization name in github (not slug, full name). Only users in this github organization can authenticate. ##### `teams` string\[] [​](#status-auth-connectors-github-orgs-teams "Direct link to status-auth-connectors-github-orgs-teams")Names of teams in a github organization. A user will be able to authenticate if they are members of at least one of these teams. Users in the organization can authenticate if this field is omitted from the config file. ##### `hostName` string [​](#status-auth-connectors-github-hostName "Direct link to status-auth-connectors-github-hostName")Required ONLY for GitHub Enterprise. This is the Hostname of the GitHub Enterprise account listed on the management console. Ensure this domain is routable on your network. ##### `rootCA` string [​](#status-auth-connectors-github-rootCA "Direct link to status-auth-connectors-github-rootCA")ONLY for GitHub Enterprise. Optional field. Used to support self-signed or untrusted CA root certificates. ##### `gitlab` object [​](#status-auth-connectors-gitlab "Direct link to status-auth-connectors-gitlab")Gitlab holds gitlab authentication configuration ##### `clientId` required string [​](#status-auth-connectors-gitlab-clientId "Direct link to status-auth-connectors-gitlab-clientId")Gitlab client id ##### `clientSecret` required string [​](#status-auth-connectors-gitlab-clientSecret "Direct link to status-auth-connectors-gitlab-clientSecret")Gitlab client secret ##### `redirectURI` required string [​](#status-auth-connectors-gitlab-redirectURI "Direct link to status-auth-connectors-gitlab-redirectURI")Redirect URI ##### `baseURL` string [​](#status-auth-connectors-gitlab-baseURL "Direct link to status-auth-connectors-gitlab-baseURL")BaseURL is optional, default = ##### `groups` string\[] [​](#status-auth-connectors-gitlab-groups "Direct link to status-auth-connectors-gitlab-groups")Optional groups whitelist, communicated through the "groups" scope. If `groups` is omitted, all of the user's GitLab groups are returned. If `groups` is provided, this acts as a whitelist - only the user's GitLab groups that are in the configured `groups` below will go into the groups claim. Conversely, if the user is not in any of the configured `groups`, the user will not be authenticated. ##### `google` object [​](#status-auth-connectors-google "Direct link to status-auth-connectors-google")Google holds google authentication configuration ##### `clientId` required string [​](#status-auth-connectors-google-clientId "Direct link to status-auth-connectors-google-clientId")Google client id ##### `clientSecret` required string [​](#status-auth-connectors-google-clientSecret "Direct link to status-auth-connectors-google-clientSecret")Google client secret ##### `redirectURI` required string [​](#status-auth-connectors-google-redirectURI "Direct link to status-auth-connectors-google-redirectURI")loft redirect uri. E.g. ##### `scopes` string\[] [​](#status-auth-connectors-google-scopes "Direct link to status-auth-connectors-google-scopes")defaults to "profile" and "email" ##### `hostedDomains` string\[] [​](#status-auth-connectors-google-hostedDomains "Direct link to status-auth-connectors-google-hostedDomains")Optional list of whitelisted domains If this field is nonempty, only users from a listed domain will be allowed to log in ##### `groups` string\[] [​](#status-auth-connectors-google-groups "Direct link to status-auth-connectors-google-groups")Optional list of whitelisted groups If this field is nonempty, only users from a listed group will be allowed to log in ##### `serviceAccountFilePath` string [​](#status-auth-connectors-google-serviceAccountFilePath "Direct link to status-auth-connectors-google-serviceAccountFilePath")Optional path to service account json If nonempty, and groups claim is made, will use authentication from file to check groups with the admin directory api ##### `adminEmail` string [​](#status-auth-connectors-google-adminEmail "Direct link to status-auth-connectors-google-adminEmail")Required if ServiceAccountFilePath The email of a GSuite super user which the service account will impersonate when listing groups ##### `microsoft` object [​](#status-auth-connectors-microsoft "Direct link to status-auth-connectors-microsoft")Microsoft holds microsoft authentication configuration ##### `clientId` required string [​](#status-auth-connectors-microsoft-clientId "Direct link to status-auth-connectors-microsoft-clientId")Microsoft client id ##### `clientSecret` required string [​](#status-auth-connectors-microsoft-clientSecret "Direct link to status-auth-connectors-microsoft-clientSecret")Microsoft client secret ##### `redirectURI` required string [​](#status-auth-connectors-microsoft-redirectURI "Direct link to status-auth-connectors-microsoft-redirectURI")loft redirect uri. Usually ##### `tenant` string [​](#status-auth-connectors-microsoft-tenant "Direct link to status-auth-connectors-microsoft-tenant")tenant configuration parameter controls what kinds of accounts may be authenticated in loft. By default, all types of Microsoft accounts (consumers and organizations) can authenticate in loft via Microsoft. To change this, set the tenant parameter to one of the following:common - both personal and business/school accounts can authenticate in loft via Microsoft (default) consumers - only personal accounts can authenticate in loft organizations - only business/school accounts can authenticate in loft tenant uuid or tenant name - only accounts belonging to specific tenant identified by either tenant uuid or tenant name can authenticate in loft ##### `groups` string\[] [​](#status-auth-connectors-microsoft-groups "Direct link to status-auth-connectors-microsoft-groups")It is possible to require a user to be a member of a particular group in order to be successfully authenticated in loft. ##### `onlySecurityGroups` boolean [​](#status-auth-connectors-microsoft-onlySecurityGroups "Direct link to status-auth-connectors-microsoft-onlySecurityGroups")configuration option restricts the list to include only security groups. By default all groups (security, Office 365, mailing lists) are included. ##### `useGroupsAsWhitelist` boolean [​](#status-auth-connectors-microsoft-useGroupsAsWhitelist "Direct link to status-auth-connectors-microsoft-useGroupsAsWhitelist")Restrict the groups claims to include only the user’s groups that are in the configured groups ##### `saml` object [​](#status-auth-connectors-saml "Direct link to status-auth-connectors-saml")SAML holds saml authentication configuration ##### `redirectURI` string [​](#status-auth-connectors-saml-redirectURI "Direct link to status-auth-connectors-saml-redirectURI")If the response assertion status value contains a Destination element, it must match this value exactly. Usually looks like ##### `ssoURL` string [​](#status-auth-connectors-saml-ssoURL "Direct link to status-auth-connectors-saml-ssoURL")SSO URL used for POST value. ##### `caData` string [​](#status-auth-connectors-saml-caData "Direct link to status-auth-connectors-saml-caData")CAData is a base64 encoded string that holds the ca certificate for validating the signature of the SAML response. Either CAData, CA or InsecureSkipSignatureValidation needs to be defined. ##### `usernameAttr` string [​](#status-auth-connectors-saml-usernameAttr "Direct link to status-auth-connectors-saml-usernameAttr")Name of attribute in the returned assertions to map to username ##### `emailAttr` string [​](#status-auth-connectors-saml-emailAttr "Direct link to status-auth-connectors-saml-emailAttr")Name of attribute in the returned assertions to map to email ##### `groupsAttr` string [​](#status-auth-connectors-saml-groupsAttr "Direct link to status-auth-connectors-saml-groupsAttr")Name of attribute in the returned assertions to map to groups ##### `ca` string [​](#status-auth-connectors-saml-ca "Direct link to status-auth-connectors-saml-ca")CA to use when validating the signature of the SAML response. ##### `insecureSkipSignatureValidation` boolean [​](#status-auth-connectors-saml-insecureSkipSignatureValidation "Direct link to status-auth-connectors-saml-insecureSkipSignatureValidation")Ignore the ca cert ##### `entityIssuer` string [​](#status-auth-connectors-saml-entityIssuer "Direct link to status-auth-connectors-saml-entityIssuer")When provided Loft will include this as the Issuer value during AuthnRequest. It will also override the redirectURI as the required audience when evaluating AudienceRestriction elements in the response. ##### `ssoIssuer` string [​](#status-auth-connectors-saml-ssoIssuer "Direct link to status-auth-connectors-saml-ssoIssuer")Issuer value expected in the SAML response. Optional. ##### `groupsDelim` string [​](#status-auth-connectors-saml-groupsDelim "Direct link to status-auth-connectors-saml-groupsDelim")If GroupsDelim is supplied the connector assumes groups are returned as a single string instead of multiple attribute values. This delimiter will be used split the groups string. ##### `allowedGroups` string\[] [​](#status-auth-connectors-saml-allowedGroups "Direct link to status-auth-connectors-saml-allowedGroups")List of groups to filter access based on membership ##### `filterGroups` boolean [​](#status-auth-connectors-saml-filterGroups "Direct link to status-auth-connectors-saml-filterGroups")If used with allowed groups, only forwards the allowed groups and not all groups specified. ##### `nameIDPolicyFormat` string [​](#status-auth-connectors-saml-nameIDPolicyFormat "Direct link to status-auth-connectors-saml-nameIDPolicyFormat")Requested format of the NameID. The NameID value is is mapped to the ID Token 'sub' claim.This can be an abbreviated form of the full URI with just the last component. For example, if this value is set to "emailAddress" the format will resolve to:urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddressIf no value is specified, this value defaults to:urn:oasis:names:tc:SAML:2.0:nameid-format:persistent #### `disableTeamCreation` boolean [​](#status-auth-disableTeamCreation "Direct link to status-auth-disableTeamCreation")Prevents from team creation for the new groups associated with the user at the time of logging in through sso, Default behaviour is false, this means that teams will be created for new groups. #### `disableUserCreation` boolean [​](#status-auth-disableUserCreation "Direct link to status-auth-disableUserCreation")DisableUserCreation prevents the SSO connectors from creating a new user on a users initial signin through sso. Default behaviour is false, this means that a new user object will be created once a user without a Kubernetes user object logs in. #### `accessKeyMaxTTLSeconds` integer [​](#status-auth-accessKeyMaxTTLSeconds "Direct link to status-auth-accessKeyMaxTTLSeconds")AccessKeyMaxTTLSeconds is the global maximum lifespan of an accesskey in seconds. Leaving it 0 or unspecified will disable it. Specifying 2592000 will mean all keys have a Time-To-Live of 30 days. #### `loginAccessKeyTTLSeconds` integer [​](#status-auth-loginAccessKeyTTLSeconds "Direct link to status-auth-loginAccessKeyTTLSeconds")LoginAccessKeyTTLSeconds is the time in seconds an access key is kept until it is deleted. Leaving it unspecified will default to 20 days. Setting it to zero will disable the ttl. Specifying 2592000 will mean all keys have a default Time-To-Live of 30 days. #### `customHttpHeaders` object [​](#status-auth-customHttpHeaders "Direct link to status-auth-customHttpHeaders")CustomHttpHeaders are additional headers that should be set for the authentication endpoints #### `groupsFilters` string\[] [​](#status-auth-groupsFilters "Direct link to status-auth-groupsFilters")GroupsFilters is a regex expression to only save matching sso groups into the user resource ### `oidc` object [​](#status-oidc "Direct link to status-oidc")DEPRECATED: Configure the OIDC clients using either the OIDC Client UI or a secret. By default, vCluster Platform as an OIDC Provider is enabled but does not function without OIDC clients. #### `enabled` boolean [​](#status-oidc-enabled "Direct link to status-oidc-enabled")If true indicates that loft will act as an OIDC server #### `wildcardRedirect` boolean [​](#status-oidc-wildcardRedirect "Direct link to status-oidc-wildcardRedirect")If true indicates that loft will allow wildcard '\*' in client redirectURIs #### `clients` object\[] [​](#status-oidc-clients "Direct link to status-oidc-clients")The clients that are allowed to request loft tokens ##### `name` string [​](#status-oidc-clients-name "Direct link to status-oidc-clients-name")The client name ##### `clientId` string [​](#status-oidc-clients-clientId "Direct link to status-oidc-clients-clientId")The client id of the client ##### `clientSecret` string [​](#status-oidc-clients-clientSecret "Direct link to status-oidc-clients-clientSecret")The client secret of the client ##### `redirectURIs` required string\[] [​](#status-oidc-clients-redirectURIs "Direct link to status-oidc-clients-redirectURIs")A registered set of redirect URIs. When redirecting from dex to the client, the URI requested to redirect to MUST match one of these values, unless the client is "public". ### `audit` object [​](#status-audit "Direct link to status-audit")Audit holds audit configuration #### `enabled` boolean [​](#status-audit-enabled "Direct link to status-audit-enabled")If audit is enabled and incoming api requests will be logged based on the supplied policy. #### `disableAgentSyncBack` boolean [​](#status-audit-disableAgentSyncBack "Direct link to status-audit-disableAgentSyncBack")If true, the agent will not send back any audit logs to Loft itself. #### `level` integer [​](#status-audit-level "Direct link to status-audit-level")Level is an optional log level for audit logs. Cannot be used together with policy #### `policy` object [​](#status-audit-policy "Direct link to status-audit-policy")The audit policy to use and log requests. By default loft will not log anything ##### `rules` object\[] [​](#status-audit-policy-rules "Direct link to status-audit-policy-rules")Rules specify the audit Level a request should be recorded at. A request may match multiple rules, in which case the FIRST matching rule is used. The default audit level is None, but can be overridden by a catch-all rule at the end of the list. PolicyRules are strictly ordered. ##### `level` required string [​](#status-audit-policy-rules-level "Direct link to status-audit-policy-rules-level")The Level that requests matching this rule are recorded at. ##### `users` string\[] [​](#status-audit-policy-rules-users "Direct link to status-audit-policy-rules-users")The users (by authenticated user name) this rule applies to. An empty list implies every user. ##### `userGroups` string\[] [​](#status-audit-policy-rules-userGroups "Direct link to status-audit-policy-rules-userGroups")The user groups this rule applies to. A user is considered matching if it is a member of any of the UserGroups. An empty list implies every user group. ##### `verbs` string\[] [​](#status-audit-policy-rules-verbs "Direct link to status-audit-policy-rules-verbs")The verbs that match this rule. An empty list implies every verb. ##### `resources` object\[] [​](#status-audit-policy-rules-resources "Direct link to status-audit-policy-rules-resources")Resources that this rule matches. An empty list implies all kinds in all API groups. ##### `group` string [​](#status-audit-policy-rules-resources-group "Direct link to status-audit-policy-rules-resources-group")Group is the name of the API group that contains the resources. The empty string represents the core API group. ##### `resources` string\[] [​](#status-audit-policy-rules-resources-resources "Direct link to status-audit-policy-rules-resources-resources")Resources is a list of resources this rule applies to.For example: 'pods' matches pods. 'pods/log' matches the log subresource of pods. '*' matches all resources and their subresources. 'pods/*' matches all subresources of pods. '\*/scale' matches all scale subresources.If wildcard is present, the validation rule will ensure resources do not overlap with each other.An empty list implies all resources and subresources in this API groups apply. ##### `resourceNames` string\[] [​](#status-audit-policy-rules-resources-resourceNames "Direct link to status-audit-policy-rules-resources-resourceNames")ResourceNames is a list of resource instance names that the policy matches. Using this field requires Resources to be specified. An empty list implies that every instance of the resource is matched. ##### `namespaces` string\[] [​](#status-audit-policy-rules-namespaces "Direct link to status-audit-policy-rules-namespaces")Namespaces that this rule matches. The empty string "" matches non-namespaced resources. An empty list implies every namespace. ##### `nonResourceURLs` string\[] [​](#status-audit-policy-rules-nonResourceURLs "Direct link to status-audit-policy-rules-nonResourceURLs")NonResourceURLs is a set of URL paths that should be audited. *s are allowed, but only as the full, final step in the path. Examples: "/metrics" - Log requests for apiserver metrics "/healthz*" - Log all health checks ##### `omitStages` string\[] [​](#status-audit-policy-rules-omitStages "Direct link to status-audit-policy-rules-omitStages")OmitStages is a list of stages for which no events are created. Note that this can also be specified policy wide in which case the union of both are omitted. An empty list means no restrictions will apply. ##### `requestTargets` string\[] [​](#status-audit-policy-rules-requestTargets "Direct link to status-audit-policy-rules-requestTargets")RequestTargets is a list of request targets for which events are created. An empty list implies every request. ##### `clusters` string\[] [​](#status-audit-policy-rules-clusters "Direct link to status-audit-policy-rules-clusters")Clusters that this rule matches. Only applies to cluster requests. If this is set, no events for non cluster requests will be created. An empty list means no restrictions will apply. ##### `omitStages` string\[] [​](#status-audit-policy-omitStages "Direct link to status-audit-policy-omitStages")OmitStages is a list of stages for which no events are created. Note that this can also be specified per rule in which case the union of both are omitted. #### `dataStoreEndpoint` string [​](#status-audit-dataStoreEndpoint "Direct link to status-audit-dataStoreEndpoint")DataStoreEndpoint is an endpoint to store events in. #### `dataStoreIdentityProvider` string [​](#status-audit-dataStoreIdentityProvider "Direct link to status-audit-dataStoreIdentityProvider")DataStoreIdentityProvider is the identity provider to use when generating temporary authentication tokens for the audit datastore, instead of a static password embedded in DataStoreEndpoint. Examples:- aws: RDS IAM Authentication #### `dataStoreCAFile` string [​](#status-audit-dataStoreCAFile "Direct link to status-audit-dataStoreCAFile")DataStoreCAFile is the path to a PEM-encoded certificate authority bundle used to verify the audit datastore's server certificate, read directly by the platform pod at startup. Mount it in the platform pod via the chart's top-level volumes/volumeMounts (e.g. from a Secret or ConfigMap), the same way as config.database's caFile. When set, the connection is fully verified (postgres: sslmode=verify-full; mysql: a registered TLS config with the given CA as RootCAs) instead of the default encrypted-but-unverified connection. Optional; has no effect on the sqlite backend.An explicit ?sslmode=/?tls= already present on DataStoreEndpoint always overrides this and is not upgraded, even when DataStoreCAFile is set - e.g. an endpoint carried over from before CA support existed with ?tls=skip-verify stays unverified. The platform logs this case at startup so it isn't silent. #### `dataStoreTTL` integer [​](#status-audit-dataStoreTTL "Direct link to status-audit-dataStoreTTL")DataStoreMaxAge is the maximum number of hours to retain old log events in the datastore #### `path` string [​](#status-audit-path "Direct link to status-audit-path")The path where to save the audit log files. This is required if audit is enabled. Backup log files will be retained in the same directory. #### `maxAge` integer [​](#status-audit-maxAge "Direct link to status-audit-maxAge")MaxAge is the maximum number of days to retain old log files based on the timestamp encoded in their filename. Note that a day is defined as 24 hours and may not exactly correspond to calendar days due to daylight savings, leap seconds, etc. The default is not to remove old log files based on age. #### `maxBackups` integer [​](#status-audit-maxBackups "Direct link to status-audit-maxBackups")MaxBackups is the maximum number of old log files to retain. The default is to retain all old log files (though MaxAge may still cause them to get deleted.) #### `maxSize` integer [​](#status-audit-maxSize "Direct link to status-audit-maxSize")MaxSize is the maximum size in megabytes of the log file before it gets rotated. It defaults to 100 megabytes. #### `compress` boolean [​](#status-audit-compress "Direct link to status-audit-compress")Compress determines if the rotated log files should be compressed using gzip. The default is not to perform compression. ### `loftHost` string [​](#status-loftHost "Direct link to status-loftHost")LoftHost holds the domain where the loft instance is hosted. This should not include https or http. E.g. loft.my-domain.com ### `projectNamespacePrefix` string [​](#status-projectNamespacePrefix "Direct link to status-projectNamespacePrefix")ProjectNamespacePrefix holds the prefix for loft project namespaces. Omitted defaults to "p-" ### `devPodSubDomain` string [​](#status-devPodSubDomain "Direct link to status-devPodSubDomain")DEPRECATED: DevPodSubDomain holds a subdomain in the following form \*.workspace.my-domain.com DevPodSubDomain holds a subdomain in the following form \*.workspace.my-domain.com ### `uiSettings` object [​](#status-uiSettings "Direct link to status-uiSettings")UISettings holds the settings for modifying the Loft user interface #### `loftVersion` string [​](#status-uiSettings-loftVersion "Direct link to status-uiSettings-loftVersion")LoftVersion holds the current loft version #### `logoURL` string [​](#status-uiSettings-logoURL "Direct link to status-uiSettings-logoURL")LogoURL is url pointing to the logo to use in the vCluster Platform UI. This path must be accessible for clients accessing the vCluster Platform UI! #### `faviconURL` string [​](#status-uiSettings-faviconURL "Direct link to status-uiSettings-faviconURL")FaviconURL is url pointing to the favicon to use in the vCluster Platform UI. This path must be accessible for clients accessing the vCluster Platform UI! #### `smallLogoURL` string [​](#status-uiSettings-smallLogoURL "Direct link to status-uiSettings-smallLogoURL")SmallLogoURL is url pointing to the small logo to use in the vCluster Platform UI. This path must be accessible for clients accessing the vCluster Platform UI! #### `logoBackgroundColor` string [​](#status-uiSettings-logoBackgroundColor "Direct link to status-uiSettings-logoBackgroundColor")LogoBackgroundColor is the color value (ex: "#12345") to use as the background color for the logo #### `legalTemplate` string [​](#status-uiSettings-legalTemplate "Direct link to status-uiSettings-legalTemplate")LegalTemplate is a text (html) string containing the legal template to prompt to users when authenticating to Loft #### `primaryColor` string [​](#status-uiSettings-primaryColor "Direct link to status-uiSettings-primaryColor")PrimaryColor is the color value (ex: "#12345") to use as the primary color #### `sidebarColor` string [​](#status-uiSettings-sidebarColor "Direct link to status-uiSettings-sidebarColor")SidebarColor is the color value (ex: "#12345") to use for the sidebar #### `accentColor` string [​](#status-uiSettings-accentColor "Direct link to status-uiSettings-accentColor")AccentColor is the color value (ex: "#12345") to use for the accent #### `customCss` string\[] [​](#status-uiSettings-customCss "Direct link to status-uiSettings-customCss")CustomCSS holds URLs with custom css files that should be included when loading the UI #### `customJavaScript` string\[] [​](#status-uiSettings-customJavaScript "Direct link to status-uiSettings-customJavaScript")CustomJavaScript holds URLs with custom js files that should be included when loading the UI #### `navBarButtons` object\[] [​](#status-uiSettings-navBarButtons "Direct link to status-uiSettings-navBarButtons")NavBarButtons holds extra nav bar buttons ##### `position` string [​](#status-uiSettings-navBarButtons-position "Direct link to status-uiSettings-navBarButtons-position")Position holds the position of the button, can be one of: TopStart, TopEnd, BottomStart, BottomEnd. Defaults to BottomEnd ##### `text` string [​](#status-uiSettings-navBarButtons-text "Direct link to status-uiSettings-navBarButtons-text")Text holds text for the button ##### `link` string [​](#status-uiSettings-navBarButtons-link "Direct link to status-uiSettings-navBarButtons-link")Link holds the link of the navbar button ##### `icon` string [​](#status-uiSettings-navBarButtons-icon "Direct link to status-uiSettings-navBarButtons-icon")Icon holds the url of the icon to display #### `externalURLs` object [​](#status-uiSettings-externalURLs "Direct link to status-uiSettings-externalURLs")External URLs that can be called from the UI ##### `block` boolean [​](#status-uiSettings-externalURLs-block "Direct link to status-uiSettings-externalURLs-block")Block determines if requests to external URLs from the UI should be blocked ##### `allow` string\[] [​](#status-uiSettings-externalURLs-allow "Direct link to status-uiSettings-externalURLs-allow")Allow specifies which external URLs can be called. In addition to the predefined modules,- "vcluster" (license page, feature descriptions, ...) - "gtm" (google tag manager) - "featurebase" (changelog) any URL can be added to this list. This will allow the UI to make any request to this URL. This is only active when Block is true. ### `vault` object [​](#status-vault "Direct link to status-vault")VaultIntegration holds the vault integration configuration #### `enabled` boolean [​](#status-vault-enabled "Direct link to status-vault-enabled")Enabled indicates if the Vault Integration is enabled for the project -- this knob only enables the syncing of secrets to or from Vault, but does not setup Kubernetes authentication methods or Kubernetes secrets engines for vclusters. #### `address` string [​](#status-vault-address "Direct link to status-vault-address")Address defines the address of the Vault instance to use for this project. Will default to the `VAULT_ADDR` environment variable if not provided. #### `skipTLSVerify` boolean [​](#status-vault-skipTLSVerify "Direct link to status-vault-skipTLSVerify")SkipTLSVerify defines if TLS verification should be skipped when connecting to Vault. #### `namespace` string [​](#status-vault-namespace "Direct link to status-vault-namespace")Namespace defines the namespace to use when storing secrets in Vault. #### `auth` object [​](#status-vault-auth "Direct link to status-vault-auth")Auth defines the authentication method to use for this project. ##### `token` string [​](#status-vault-auth-token "Direct link to status-vault-auth-token")Token defines the token to use for authentication. ##### `tokenSecretRef` object [​](#status-vault-auth-tokenSecretRef "Direct link to status-vault-auth-tokenSecretRef")TokenSecretRef defines the Kubernetes secret to use for token authentication. Will be used if `token` is not provided.Secret data should contain the `token` key. ##### `name` string [​](#status-vault-auth-tokenSecretRef-name "Direct link to status-vault-auth-tokenSecretRef-name")Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: TODO: Drop `kubebuilder:default` when controller-gen doesn't need it . ##### `key` required string [​](#status-vault-auth-tokenSecretRef-key "Direct link to status-vault-auth-tokenSecretRef-key")The key of the secret to select from. Must be a valid secret key. ##### `optional` boolean [​](#status-vault-auth-tokenSecretRef-optional "Direct link to status-vault-auth-tokenSecretRef-optional")Specify whether the Secret or its key must be defined #### `syncInterval` string [​](#status-vault-syncInterval "Direct link to status-vault-syncInterval")SyncInterval defines the interval at which to sync secrets from Vault. Defaults to `1m.` See for supported formats. ### `disableConfigEndpoint` boolean [​](#status-disableConfigEndpoint "Direct link to status-disableConfigEndpoint")DisableLoftConfigEndpoint will disable setting config via the UI and config.management.loft.sh endpoint ### `authenticateVersionEndpoint` boolean [​](#status-authenticateVersionEndpoint "Direct link to status-authenticateVersionEndpoint")AuthenticateVersionEndpoint will force authentication for the '/version' endpoint. Will only work with vCluster v0.27 & later ### `cloud` object [​](#status-cloud "Direct link to status-cloud")Cloud holds the settings to be used exclusively in vCluster Cloud based environments and deployments. #### `releaseChannel` string [​](#status-cloud-releaseChannel "Direct link to status-cloud-releaseChannel")ReleaseChannel specifies the release channel for the cloud configuration. This can be used to determine which updates or versions are applied. #### `maintenanceWindow` object [​](#status-cloud-maintenanceWindow "Direct link to status-cloud-maintenanceWindow")MaintenanceWindow specifies the maintenance window for the cloud configuration. This is a structured representation of the time window during which maintenance can occur. ##### `dayOfWeek` string [​](#status-cloud-maintenanceWindow-dayOfWeek "Direct link to status-cloud-maintenanceWindow-dayOfWeek")DayOfWeek specifies the day of the week for the maintenance window. It should be a string representing the day, e.g., "Monday", "Tuesday", etc. ##### `timeWindow` string [​](#status-cloud-maintenanceWindow-timeWindow "Direct link to status-cloud-maintenanceWindow-timeWindow")TimeWindow specifies the time window for the maintenance. It should be a string representing the time range in 24-hour format, in UTC, e.g., "02:00-03:00". ### `costControl` object [​](#status-costControl "Direct link to status-costControl")CostControl holds the settings related to the Cost Control ROI dashboard and its metrics gathering infrastructure #### `enabled` boolean [​](#status-costControl-enabled "Direct link to status-costControl-enabled")Enabled specifies whether the ROI dashboard should be available in the UI, and if the metrics infrastructure that provides dashboard data is deployed #### `global` object [​](#status-costControl-global "Direct link to status-costControl-global")Global are settings for globally managed components ##### `metrics` object [​](#status-costControl-global-metrics "Direct link to status-costControl-global-metrics")Metrics these settings apply to metric infrastructure used to aggregate metrics across all connected clusters ##### `replicas` integer [​](#status-costControl-global-metrics-replicas "Direct link to status-costControl-global-metrics-replicas")Replicas is the number of desired replicas. ##### `resources` object [​](#status-costControl-global-metrics-resources "Direct link to status-costControl-global-metrics-resources")Resources are compute resource required by the metrics backend ##### `limits` object [​](#status-costControl-global-metrics-resources-limits "Direct link to status-costControl-global-metrics-resources-limits")Limits describes the maximum amount of compute resources allowed. More info: ##### `requests` object [​](#status-costControl-global-metrics-resources-requests "Direct link to status-costControl-global-metrics-resources-requests")Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: ##### `claims` object\[] [​](#status-costControl-global-metrics-resources-claims "Direct link to status-costControl-global-metrics-resources-claims")Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.This field depends on the DynamicResourceAllocation feature gate.This field is immutable. It can only be set for containers. ##### `name` required string [​](#status-costControl-global-metrics-resources-claims-name "Direct link to status-costControl-global-metrics-resources-claims-name")Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. ##### `request` string [​](#status-costControl-global-metrics-resources-claims-request "Direct link to status-costControl-global-metrics-resources-claims-request")Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. ##### `retention` string [​](#status-costControl-global-metrics-retention "Direct link to status-costControl-global-metrics-retention")Retention is the metrics data retention period. Default is 1y ##### `storage` object [​](#status-costControl-global-metrics-storage "Direct link to status-costControl-global-metrics-storage") ##### `storageClass` string [​](#status-costControl-global-metrics-storage-storageClass "Direct link to status-costControl-global-metrics-storage-storageClass")StorageClass the storage class to use when provisioning the metrics backend's persistent volume If set to "-" or "" dynamic provisioning is disabled If set to undefined or null (the default), the cluster's default storage class is used for provisioning ##### `size` string [​](#status-costControl-global-metrics-storage-size "Direct link to status-costControl-global-metrics-storage-size")Size the size of the metrics backend's persistent volume #### `cluster` object [​](#status-costControl-cluster "Direct link to status-costControl-cluster")Cluster are settings for each cluster's managed components. These settings apply to all connected clusters unless overridden by modifying the Cluster's spec ##### `metrics` object [​](#status-costControl-cluster-metrics "Direct link to status-costControl-cluster-metrics")Metrics are settings applied to metric infrastructure in each connected cluster. These can be overridden in individual clusters by modifying the Cluster's spec ##### `replicas` integer [​](#status-costControl-cluster-metrics-replicas "Direct link to status-costControl-cluster-metrics-replicas")Replicas is the number of desired replicas. ##### `resources` object [​](#status-costControl-cluster-metrics-resources "Direct link to status-costControl-cluster-metrics-resources")Resources are compute resource required by the metrics backend ##### `limits` object [​](#status-costControl-cluster-metrics-resources-limits "Direct link to status-costControl-cluster-metrics-resources-limits")Limits describes the maximum amount of compute resources allowed. More info: ##### `requests` object [​](#status-costControl-cluster-metrics-resources-requests "Direct link to status-costControl-cluster-metrics-resources-requests")Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: ##### `claims` object\[] [​](#status-costControl-cluster-metrics-resources-claims "Direct link to status-costControl-cluster-metrics-resources-claims")Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.This field depends on the DynamicResourceAllocation feature gate.This field is immutable. It can only be set for containers. ##### `name` required string [​](#status-costControl-cluster-metrics-resources-claims-name "Direct link to status-costControl-cluster-metrics-resources-claims-name")Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. ##### `request` string [​](#status-costControl-cluster-metrics-resources-claims-request "Direct link to status-costControl-cluster-metrics-resources-claims-request")Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. ##### `retention` string [​](#status-costControl-cluster-metrics-retention "Direct link to status-costControl-cluster-metrics-retention")Retention is the metrics data retention period. Default is 1y ##### `storage` object [​](#status-costControl-cluster-metrics-storage "Direct link to status-costControl-cluster-metrics-storage") ##### `storageClass` string [​](#status-costControl-cluster-metrics-storage-storageClass "Direct link to status-costControl-cluster-metrics-storage-storageClass")StorageClass the storage class to use when provisioning the metrics backend's persistent volume If set to "-" or "" dynamic provisioning is disabled If set to undefined or null (the default), the cluster's default storage class is used for provisioning ##### `size` string [​](#status-costControl-cluster-metrics-storage-size "Direct link to status-costControl-cluster-metrics-storage-size")Size the size of the metrics backend's persistent volume ##### `opencost` object [​](#status-costControl-cluster-opencost "Direct link to status-costControl-cluster-opencost")OpenCost are settings applied to OpenCost deployments in each connected cluster. These can be overridden in individual clusters by modifying the Cluster's spec ##### `replicas` integer [​](#status-costControl-cluster-opencost-replicas "Direct link to status-costControl-cluster-opencost-replicas")Replicas is the number of desired replicas. ##### `resources` object [​](#status-costControl-cluster-opencost-resources "Direct link to status-costControl-cluster-opencost-resources")Resources are compute resource required by the OpenCost backend ##### `limits` object [​](#status-costControl-cluster-opencost-resources-limits "Direct link to status-costControl-cluster-opencost-resources-limits")Limits describes the maximum amount of compute resources allowed. More info: ##### `requests` object [​](#status-costControl-cluster-opencost-resources-requests "Direct link to status-costControl-cluster-opencost-resources-requests")Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: ##### `claims` object\[] [​](#status-costControl-cluster-opencost-resources-claims "Direct link to status-costControl-cluster-opencost-resources-claims")Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.This field depends on the DynamicResourceAllocation feature gate.This field is immutable. It can only be set for containers. ##### `name` required string [​](#status-costControl-cluster-opencost-resources-claims-name "Direct link to status-costControl-cluster-opencost-resources-claims-name")Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. ##### `request` string [​](#status-costControl-cluster-opencost-resources-claims-request "Direct link to status-costControl-cluster-opencost-resources-claims-request")Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. #### `settings` object [​](#status-costControl-settings "Direct link to status-costControl-settings")Settings specify price-related settings that are taken into account for the ROI dashboard calculations. ##### `priceCurrency` string [​](#status-costControl-settings-priceCurrency "Direct link to status-costControl-settings-priceCurrency")PriceCurrency specifies the currency. ##### `averageCPUPricePerNode` object [​](#status-costControl-settings-averageCPUPricePerNode "Direct link to status-costControl-settings-averageCPUPricePerNode")AvgCPUPricePerNode specifies the average CPU price per node. ##### `price` number [​](#status-costControl-settings-averageCPUPricePerNode-price "Direct link to status-costControl-settings-averageCPUPricePerNode-price")Price specifies the price. ##### `timePeriod` string [​](#status-costControl-settings-averageCPUPricePerNode-timePeriod "Direct link to status-costControl-settings-averageCPUPricePerNode-timePeriod")TimePeriod specifies the time period for the price. ##### `averageRAMPricePerNode` object [​](#status-costControl-settings-averageRAMPricePerNode "Direct link to status-costControl-settings-averageRAMPricePerNode")AvgRAMPricePerNode specifies the average RAM price per node. ##### `price` number [​](#status-costControl-settings-averageRAMPricePerNode-price "Direct link to status-costControl-settings-averageRAMPricePerNode-price")Price specifies the price. ##### `timePeriod` string [​](#status-costControl-settings-averageRAMPricePerNode-timePeriod "Direct link to status-costControl-settings-averageRAMPricePerNode-timePeriod")TimePeriod specifies the time period for the price. ##### `gpuSettings` object [​](#status-costControl-settings-gpuSettings "Direct link to status-costControl-settings-gpuSettings")GPUSettings specifies GPU related settings. ##### `enabled` boolean [​](#status-costControl-settings-gpuSettings-enabled "Direct link to status-costControl-settings-gpuSettings-enabled")Enabled specifies whether GPU settings should be available in the UI. ##### `averageGPUPrice` object [​](#status-costControl-settings-gpuSettings-averageGPUPrice "Direct link to status-costControl-settings-gpuSettings-averageGPUPrice")AvgGPUPrice specifies the average GPU price. ##### `price` number [​](#status-costControl-settings-gpuSettings-averageGPUPrice-price "Direct link to status-costControl-settings-gpuSettings-averageGPUPrice-price")Price specifies the price. ##### `timePeriod` string [​](#status-costControl-settings-gpuSettings-averageGPUPrice-timePeriod "Direct link to status-costControl-settings-gpuSettings-averageGPUPrice-timePeriod")TimePeriod specifies the time period for the price. ##### `controlPlanePricePerCluster` object [​](#status-costControl-settings-controlPlanePricePerCluster "Direct link to status-costControl-settings-controlPlanePricePerCluster")ControlPlanePricePerCluster specifies the price of one physical cluster. ##### `price` number [​](#status-costControl-settings-controlPlanePricePerCluster-price "Direct link to status-costControl-settings-controlPlanePricePerCluster-price")Price specifies the price. ##### `timePeriod` string [​](#status-costControl-settings-controlPlanePricePerCluster-timePeriod "Direct link to status-costControl-settings-controlPlanePricePerCluster-timePeriod")TimePeriod specifies the time period for the price. ### `platformDB` object [​](#status-platformDB "Direct link to status-platformDB")PlatformDB holds the settings related to the postgres database that platform uses to store data #### `storageClass` string [​](#status-platformDB-storageClass "Direct link to status-platformDB-storageClass")StorageClass sets the storage class for the PersistentVolumeClaim used by the platform database statefulSet. ### `imageBuilder` object [​](#status-imageBuilder "Direct link to status-imageBuilder")ImageBuilder holds the settings related to the image builder #### `enabled` boolean [​](#status-imageBuilder-enabled "Direct link to status-imageBuilder-enabled")Enabled specifies whether the remote image builder should be available. If it's not available building ad-hoc images from a devcontainer.json is not supported #### `replicas` integer [​](#status-imageBuilder-replicas "Direct link to status-imageBuilder-replicas")Replicas is the number of desired replicas. #### `resources` object [​](#status-imageBuilder-resources "Direct link to status-imageBuilder-resources")Resources are compute resource required by the buildkit containers ##### `limits` object [​](#status-imageBuilder-resources-limits "Direct link to status-imageBuilder-resources-limits")Limits describes the maximum amount of compute resources allowed. More info: ##### `requests` object [​](#status-imageBuilder-resources-requests "Direct link to status-imageBuilder-resources-requests")Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: ##### `claims` object\[] [​](#status-imageBuilder-resources-claims "Direct link to status-imageBuilder-resources-claims")Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.This field depends on the DynamicResourceAllocation feature gate.This field is immutable. It can only be set for containers. ##### `name` required string [​](#status-imageBuilder-resources-claims-name "Direct link to status-imageBuilder-resources-claims-name")Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. ##### `request` string [​](#status-imageBuilder-resources-claims-request "Direct link to status-imageBuilder-resources-claims-request")Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. ### `database` object [​](#status-database "Direct link to status-database")Database represents the database connection settings when deploying the platform with an embedded Kubernetes backed by kine #### `enabled` boolean [​](#status-database-enabled "Direct link to status-database-enabled")Enabled defines if the database should be used. #### `dataSource` string [​](#status-database-dataSource "Direct link to status-database-dataSource")DataSource is the kine dataSource to use for the database. This depends on the database format. This is optional for the external database. Examples:- mysql: mysql://username:password\@tcp(hostname:3306)/k3s - postgres: postgres\://username:password\@hostname:5432/k3s #### `identityProvider` string [​](#status-database-identityProvider "Direct link to status-database-identityProvider")IdentityProvider is the kine identity provider to use when generating temporary authentication tokens for enhanced security. This is optional for the external database. Examples:- aws: RDS IAM Authentication #### `keyFile` string [​](#status-database-keyFile "Direct link to status-database-keyFile")KeyFile is the key file to use for the database. This is optional. #### `certFile` string [​](#status-database-certFile "Direct link to status-database-certFile")CertFile is the cert file to use for the database. This is optional. #### `caFile` string [​](#status-database-caFile "Direct link to status-database-caFile")CaFile is the ca file to use for the database. This is optional. #### `extraArgs` string\[] [​](#status-database-extraArgs "Direct link to status-database-extraArgs")ExtraArgs are additional arguments to pass to Kine. ## Retrieve: Configs[​](#retrieve-configs "Direct link to Retrieve: Configs") You can either use curl or kubectl to retrieve Configs. * kubectl * curl ### Retrieve a list of Configs[​](#retrieve-a-list-of-configs "Direct link to Retrieve a list of Configs") Run the following command to list all Configs: ``` kubectl get config.management.loft.sh -o yaml ``` ### Retrieve a single Config by name[​](#retrieve-a-single-config-by-name "Direct link to Retrieve a single Config by name") Run the following kubectl command to get Config `my-object`: ``` kubectl get config.management.loft.sh my-object -o yaml ``` ### Retrieve a list of Configs[​](#retrieve-a-list-of-configs-1 "Direct link to Retrieve a list of Configs") Run the following curl command to list all Configs: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/config" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single Config by name[​](#get-a-single-config-by-name "Direct link to Get a single Config by name") Run the following curl command to get Config `my-object`: ``` # Exchange my-object in the url below with the name of the Config curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/config/my-object" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Direct Cluster Endpoint Token Direct Cluster Endpoints are a feature to avoid the central Loft instance and directly connect to a connected Loft cluster. ## Direct Cluster Endpoint Token example[​](#direct-cluster-endpoint-token-example "Direct link to Direct Cluster Endpoint Token example") An example Direct Cluster Endpoint Token: ``` apiVersion: management.loft.sh/v1 kind: DirectClusterEndpointToken metadata: {} spec: {} status: token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjV3RlZOcmU1Ty1ZazBMRmNoN3F1NzVlYlctdk5rdGJ5eTRlelJNQ3dKLVEifQ.eyJhdWQiOlsiaHR0cHM6V0ZXMuZGVmYXVsdCJdLCJleHAiOjE5ODMwNzXIubG... ``` ## Direct Cluster Endpoint Token reference[​](#direct-cluster-endpoint-token-reference "Direct link to Direct Cluster Endpoint Token reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `ttl` integer [​](#spec-ttl "Direct link to spec-ttl")The time to live for this access token in seconds ### `scope` object [​](#spec-scope "Direct link to spec-scope")Scope is the optional scope of the direct cluster endpoint #### `roles` object\[] [​](#spec-scope-roles "Direct link to spec-scope-roles")Roles is a set of managed permissions to apply to the access key. ##### `role` string [​](#spec-scope-roles-role "Direct link to spec-scope-roles-role")Role is the name of the role to apply to the access key scope. ##### `projects` string\[] [​](#spec-scope-roles-projects "Direct link to spec-scope-roles-projects")Projects specifies the projects the access key should have access to. ##### `virtualClusters` string\[] [​](#spec-scope-roles-virtualClusters "Direct link to spec-scope-roles-virtualClusters")VirtualClusters specifies the tenant clusters the access key is allowed to access. #### `projects` object\[] [​](#spec-scope-projects "Direct link to spec-scope-projects")Projects specifies the projects the access key should have access to. ##### `project` string [​](#spec-scope-projects-project "Direct link to spec-scope-projects-project")Project is the name of the project. You can specify \* to select all projects. #### `spaces` object\[] [​](#spec-scope-spaces "Direct link to spec-scope-spaces")Spaces specifies the spaces the access key is allowed to access. ##### `project` string [​](#spec-scope-spaces-project "Direct link to spec-scope-spaces-project")Project is the name of the project. ##### `space` string [​](#spec-scope-spaces-space "Direct link to spec-scope-spaces-space")Space is the name of the space. You can specify \* to select all spaces. #### `virtualClusters` object\[] [​](#spec-scope-virtualClusters "Direct link to spec-scope-virtualClusters")VirtualClusters specifies the tenant clusters the access key is allowed to access. ##### `project` string [​](#spec-scope-virtualClusters-project "Direct link to spec-scope-virtualClusters-project")Project is the name of the project. ##### `virtualCluster` string [​](#spec-scope-virtualClusters-virtualCluster "Direct link to spec-scope-virtualClusters-virtualCluster")VirtualCluster is the name of the tenant cluster to access. You can specify \* to select all tenant clusters. #### `clusters` object\[] [​](#spec-scope-clusters "Direct link to spec-scope-clusters")Clusters specifies the project cluster the access key is allowed to access. ##### `cluster` string [​](#spec-scope-clusters-cluster "Direct link to spec-scope-clusters-cluster")Cluster is the name of the cluster to access. You can specify \* to select all clusters. #### `rules` object\[] [​](#spec-scope-rules "Direct link to spec-scope-rules")DEPRECATED: Use Projects, Spaces and VirtualClusters instead Rules specifies the rules that should apply to the access key. ##### `verbs` string\[] [​](#spec-scope-rules-verbs "Direct link to spec-scope-rules-verbs")The verbs that match this rule. An empty list implies every verb. ##### `resources` object\[] [​](#spec-scope-rules-resources "Direct link to spec-scope-rules-resources")Resources that this rule matches. An empty list implies all kinds in all API groups. ##### `group` string [​](#spec-scope-rules-resources-group "Direct link to spec-scope-rules-resources-group")Group is the name of the API group that contains the resources. The empty string represents the core API group. ##### `resources` string\[] [​](#spec-scope-rules-resources-resources "Direct link to spec-scope-rules-resources-resources")Resources is a list of resources this rule applies to.For example: 'pods' matches pods. 'pods/log' matches the log subresource of pods. '*' matches all resources and their subresources. 'pods/*' matches all subresources of pods. '\*/scale' matches all scale subresources.If wildcard is present, the validation rule will ensure resources do not overlap with each other.An empty list implies all resources and subresources in this API groups apply. ##### `resourceNames` string\[] [​](#spec-scope-rules-resources-resourceNames "Direct link to spec-scope-rules-resources-resourceNames")ResourceNames is a list of resource instance names that the policy matches. Using this field requires Resources to be specified. An empty list implies that every instance of the resource is matched. ##### `namespaces` string\[] [​](#spec-scope-rules-namespaces "Direct link to spec-scope-rules-namespaces")Namespaces that this rule matches. The empty string "" matches non-namespaced resources. An empty list implies every namespace. ##### `nonResourceURLs` string\[] [​](#spec-scope-rules-nonResourceURLs "Direct link to spec-scope-rules-nonResourceURLs")NonResourceURLs is a set of URL paths that should be checked. *s are allowed, but only as the full, final step in the path. Examples: "/metrics" - Log requests for apiserver metrics "/healthz*" - Log all health checks ##### `requestTargets` string\[] [​](#spec-scope-rules-requestTargets "Direct link to spec-scope-rules-requestTargets")RequestTargets is a list of request targets that are allowed. An empty list implies every request. ##### `cluster` string [​](#spec-scope-rules-cluster "Direct link to spec-scope-rules-cluster")Cluster that this rule matches. Only applies to cluster requests. If this is set, no requests for non cluster requests are allowed. An empty cluster means no restrictions will apply. ##### `virtualClusters` object\[] [​](#spec-scope-rules-virtualClusters "Direct link to spec-scope-rules-virtualClusters")VirtualClusters that this rule matches. Only applies to tenant cluster requests. An empty list means no restrictions will apply. ##### `name` string [​](#spec-scope-rules-virtualClusters-name "Direct link to spec-scope-rules-virtualClusters-name")Name of the tenant cluster. Empty means all tenant clusters. ##### `namespace` string [​](#spec-scope-rules-virtualClusters-namespace "Direct link to spec-scope-rules-virtualClusters-namespace")Namespace of the tenant cluster. Empty means all namespaces. #### `allowLoftCli` boolean [​](#spec-scope-allowLoftCli "Direct link to spec-scope-allowLoftCli")AllowLoftCLI allows certain read-only management requests to make sure loft cli works correctly with this specific access key.Deprecated: Use the `roles` field instead``` # Example: roles: - role: loftCLI ``` ## `status` object [​](#status "Direct link to status") ### `token` string [​](#status-token "Direct link to status-token") ## Create: Direct Cluster Endpoint Token[​](#create-direct-cluster-endpoint-token "Direct link to Create: Direct Cluster Endpoint Token") You can either use curl or kubectl to create a new Direct Cluster Endpoint Token. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: DirectClusterEndpointToken metadata: {} spec: {} status: token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjV3RlZOcmU1Ty1ZazBMRmNoN3F1NzVlYlctdk5rdGJ5eTRlelJNQ3dKLVEifQ.eyJhdWQiOlsiaHR0cHM6V0ZXMuZGVmYXVsdCJdLCJleHAiOjE5ODMwNzXIubG... ``` Then create the Direct Cluster Endpoint Token `my-object` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: DirectClusterEndpointToken metadata: {} spec: {} status: token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjV3RlZOcmU1Ty1ZazBMRmNoN3F1NzVlYlctdk5rdGJ5eTRlelJNQ3dKLVEifQ.eyJhdWQiOlsiaHR0cHM6V0ZXMuZGVmYXVsdCJdLCJleHAiOjE5ODMwNzXIubG... ``` Run the following curl command to create a new Direct Cluster Endpoint Token `my-object`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/directclusterendpointtokens" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Machine Config Template MachineConfigTemplate holds a reusable Go template for a cloud-init (user data) or network-data document. A node provider renders the template when it provisions a Machine. ## MachineConfigTemplate example[​](#machineconfigtemplate-example "Direct link to MachineConfigTemplate example") An example MachineConfigTemplate: ``` apiVersion: management.loft.sh/v1 kind: MachineConfigTemplate metadata: name: ubuntu-cloud-init spec: cloudInitTemplate: |- #cloud-config hostname: {{ .Values.NodeClaim.Name }} users: - name: ubuntu sudo: ALL=(ALL) NOPASSWD:ALL displayName: Ubuntu cloud-init status: {} ``` ## MachineConfigTemplate reference[​](#machineconfigtemplate-reference "Direct link to MachineConfigTemplate reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name of the NodeClaim that is displayed in the UI. ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ### `cloudInitTemplate` string [​](#spec-cloudInitTemplate "Direct link to spec-cloudInitTemplate")CloudInitTemplate is the cloud init template to use for the machine config. ### `networkDataTemplate` string [​](#spec-networkDataTemplate "Direct link to spec-networkDataTemplate")NetworkDataTemplate is the network data template to use for the machine config. ## `status` object [​](#status "Direct link to status") ## Retrieve: MachineConfigTemplates[​](#retrieve-machineconfigtemplates "Direct link to Retrieve: MachineConfigTemplates") You can either use curl or kubectl to retrieve MachineConfigTemplates. * kubectl * curl ### Retrieve a list of MachineConfigTemplates[​](#retrieve-a-list-of-machineconfigtemplates "Direct link to Retrieve a list of MachineConfigTemplates") Run the following command to list all MachineConfigTemplates: ``` kubectl get machineconfigtemplates.management.loft.sh -o yaml ``` ### Retrieve a single MachineConfigTemplate by name[​](#retrieve-a-single-machineconfigtemplate-by-name "Direct link to Retrieve a single MachineConfigTemplate by name") Run the following kubectl command to get MachineConfigTemplate `ubuntu-cloud-init`: ``` kubectl get machineconfigtemplates.management.loft.sh ubuntu-cloud-init -o yaml ``` ### Retrieve a list of MachineConfigTemplates[​](#retrieve-a-list-of-machineconfigtemplates-1 "Direct link to Retrieve a list of MachineConfigTemplates") Run the following curl command to list all MachineConfigTemplates: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/machineconfigtemplates" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single MachineConfigTemplate by name[​](#get-a-single-machineconfigtemplate-by-name "Direct link to Get a single MachineConfigTemplate by name") Run the following curl command to get MachineConfigTemplate `ubuntu-cloud-init`: ``` # Exchange ubuntu-cloud-init in the url below with the name of the MachineConfigTemplate curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/machineconfigtemplates/ubuntu-cloud-init" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: MachineConfigTemplate[​](#create-machineconfigtemplate "Direct link to Create: MachineConfigTemplate") You can either use curl or kubectl to create a new MachineConfigTemplate. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: MachineConfigTemplate metadata: name: ubuntu-cloud-init spec: cloudInitTemplate: |- #cloud-config hostname: {{ .Values.NodeClaim.Name }} users: - name: ubuntu sudo: ALL=(ALL) NOPASSWD:ALL displayName: Ubuntu cloud-init status: {} ``` Then create the MachineConfigTemplate `ubuntu-cloud-init` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: MachineConfigTemplate metadata: name: ubuntu-cloud-init spec: cloudInitTemplate: |- #cloud-config hostname: {{ .Values.NodeClaim.Name }} users: - name: ubuntu sudo: ALL=(ALL) NOPASSWD:ALL displayName: Ubuntu cloud-init status: {} ``` Run the following curl command to create a new MachineConfigTemplate `ubuntu-cloud-init`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/machineconfigtemplates" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: MachineConfigTemplate[​](#update-machineconfigtemplate "Direct link to Update: MachineConfigTemplate") You can either use curl or kubectl to update MachineConfigTemplates. * kubectl * curl ### Update MachineConfigTemplate[​](#update-machineconfigtemplate "Direct link to Update MachineConfigTemplate") Run the following command to update MachineConfigTemplate `ubuntu-cloud-init`: ``` kubectl edit machineconfigtemplates.management.loft.sh ubuntu-cloud-init ``` Then edit the object and upon save, kubectl will update the resource. ### Patch MachineConfigTemplate[​](#patch-machineconfigtemplate "Direct link to Patch MachineConfigTemplate") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the MachineConfigTemplate `ubuntu-cloud-init` via a patch: ``` kubectl patch machineconfigtemplates.management.loft.sh ubuntu-cloud-init \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update MachineConfigTemplate[​](#update-machineconfigtemplate-1 "Direct link to Update MachineConfigTemplate") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: MachineConfigTemplate metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: ubuntu-cloud-init resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: cloudInitTemplate: |- #cloud-config hostname: {{ .Values.NodeClaim.Name }} users: - name: ubuntu sudo: ALL=(ALL) NOPASSWD:ALL displayName: Ubuntu cloud-init status: {} ``` Run the following curl command to update a single MachineConfigTemplate `ubuntu-cloud-init`: ``` # Replace the ubuntu-cloud-init in the url below with the name of the MachineConfigTemplate you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/machineconfigtemplates/ubuntu-cloud-init" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch MachineConfigTemplate[​](#patch-machineconfigtemplate-1 "Direct link to Patch MachineConfigTemplate") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the MachineConfigTemplate `ubuntu-cloud-init` via a patch: ``` # Replace the ubuntu-cloud-init in the url below with the name of the MachineConfigTemplate you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/machineconfigtemplates/ubuntu-cloud-init" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: MachineConfigTemplate[​](#delete-machineconfigtemplate "Direct link to Delete: MachineConfigTemplate") You can either use curl or kubectl to delete MachineConfigTemplates. * kubectl * curl Run the following command to delete MachineConfigTemplate `ubuntu-cloud-init`: ``` kubectl delete machineconfigtemplates.management.loft.sh ubuntu-cloud-init ``` Run the following curl command to delete MachineConfigTemplate `ubuntu-cloud-init`: ``` # Replace the ubuntu-cloud-init in the url below with the name of the MachineConfigTemplate you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/machineconfigtemplates/ubuntu-cloud-init" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Node Claim NodeClaim tracks a node that the platform has provisioned or is provisioning for a tenant cluster. Auto-nodes creates these automatically; they are not typically authored by hand. ## NodeClaim example[​](#nodeclaim-example "Direct link to NodeClaim example") An example NodeClaim: ``` apiVersion: management.loft.sh/v1 kind: NodeClaim metadata: name: vcluster-62ftb namespace: p-default spec: controlPlane: false desiredCapacity: cpu: 120m memory: 114Mi providerRef: my-node-provider vClusterRef: my-vcluster status: {} ``` ## NodeClaim reference[​](#nodeclaim-reference "Direct link to NodeClaim reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name of the NodeClaim that is displayed in the UI. ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ### `providerRef` required string [​](#spec-providerRef "Direct link to spec-providerRef")ProviderRef is the name of the NodeProvider that this NodeClaim is based on. ### `typeRef` string [​](#spec-typeRef "Direct link to spec-typeRef")TypeRef is the full name of the NodeType that this NodeClaim is based on. ### `environmentRef` string [​](#spec-environmentRef "Direct link to spec-environmentRef")EnvironmentRef is the name of the NodeEnvironment that this NodeClaim is based on. ### `properties` object [​](#spec-properties "Direct link to spec-properties")Properties are extra properties for the NodeClaim. ### `vClusterRef` string [​](#spec-vClusterRef "Direct link to spec-vClusterRef")VClusterRef references source vCluster. ### `controlPlane` boolean [​](#spec-controlPlane "Direct link to spec-controlPlane")ControlPlane indicates if the node claim is for a control plane node. This is intentionally not omitempty as we want to ensure that the control plane is always set for easier checking in for example terraform templates. ### `taints` object\[] [​](#spec-taints "Direct link to spec-taints")Taints will be applied to the NodeClaim's node. #### `key` required string [​](#spec-taints-key "Direct link to spec-taints-key")Required. The taint key to be applied to a node. #### `value` string [​](#spec-taints-value "Direct link to spec-taints-value")The taint value corresponding to the taint key. #### `effect` required string [​](#spec-taints-effect "Direct link to spec-taints-effect")Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. #### `timeAdded` object [​](#spec-taints-timeAdded "Direct link to spec-taints-timeAdded")TimeAdded represents the time at which the taint was added. ### `startupTaints` object\[] [​](#spec-startupTaints "Direct link to spec-startupTaints")StartupTaints are taints that are applied to nodes upon startup which are expected to be removed automatically within a short period of time, typically by a DaemonSet that tolerates the taint. These are commonly used by daemonsets to allow initialization and enforce startup ordering. StartupTaints are ignored for provisioning purposes in that pods are not required to tolerate a StartupTaint in order to have nodes provisioned for them. #### `key` required string [​](#spec-startupTaints-key "Direct link to spec-startupTaints-key")Required. The taint key to be applied to a node. #### `value` string [​](#spec-startupTaints-value "Direct link to spec-startupTaints-value")The taint value corresponding to the taint key. #### `effect` required string [​](#spec-startupTaints-effect "Direct link to spec-startupTaints-effect")Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. #### `timeAdded` object [​](#spec-startupTaints-timeAdded "Direct link to spec-startupTaints-timeAdded")TimeAdded represents the time at which the taint was added. ### `kubeletArgs` object [​](#spec-kubeletArgs "Direct link to spec-kubeletArgs")KubeletArgs are additional arguments to pass to the kubelet. ### `profileRef` string [​](#spec-profileRef "Direct link to spec-profileRef")ProfileRef is the name of a NodeProfile in the catalog to apply to the resulting node. The referenced profile must exist and must be permitted by the owning project's allowedNodeProfiles. Optional. ### `desiredCapacity` object [​](#spec-desiredCapacity "Direct link to spec-desiredCapacity")DesiredCapacity specifies the resources requested by the NodeClaim. ### `requirements` object\[] [​](#spec-requirements "Direct link to spec-requirements")Requirements are the requirements for the NodeClaim. #### `key` required string [​](#spec-requirements-key "Direct link to spec-requirements-key")The label key that the selector applies to. #### `operator` required string [​](#spec-requirements-operator "Direct link to spec-requirements-operator")Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. #### `values` string\[] [​](#spec-requirements-values "Direct link to spec-requirements-values")An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. ### `power` object [​](#spec-power "Direct link to spec-power")Power describes the desired power state of the machine. #### `state` string [​](#spec-power-state "Direct link to spec-power-state")State is the power state.In spec, only "On" and "Off" are meaningful; an empty value is treated as "On".In status, "On" and "Off" indicate the observed state is stable. Providers may also surface their native intermediate state (e.g. "Starting", "Stopping", "Migrating", ...) as a passthrough. An empty value means the state could not be determined. ## `status` object [​](#status "Direct link to status") ### `phase` string [​](#status-phase "Direct link to status-phase")Phase is the current lifecycle phase of the NodeClaim. ### `reason` string [​](#status-reason "Direct link to status-reason")Reason describes the reason in machine-readable form ### `message` string [​](#status-message "Direct link to status-message")Message describes the reason in human-readable form ### `conditions` object [​](#status-conditions "Direct link to status-conditions")Conditions describe the current state of the platform NodeClaim. ### `power` object [​](#status-power "Direct link to status-power")Power describes the observed power state of the machine. #### `state` string [​](#status-power-state "Direct link to status-power-state")State is the power state.In spec, only "On" and "Off" are meaningful; an empty value is treated as "On".In status, "On" and "Off" indicate the observed state is stable. Providers may also surface their native intermediate state (e.g. "Starting", "Stopping", "Migrating", ...) as a passthrough. An empty value means the state could not be determined. ## Retrieve: NodeClaims[​](#retrieve-nodeclaims "Direct link to Retrieve: NodeClaims") You can either use curl or kubectl to retrieve NodeClaims. * kubectl * curl ### Retrieve a list of NodeClaims[​](#retrieve-a-list-of-nodeclaims "Direct link to Retrieve a list of NodeClaims") Run the following command to list all NodeClaims in project `my-project`: ``` kubectl get nodeclaims.management.loft.sh -n p-my-project -o yaml ``` ### Retrieve a single NodeClaim by name[​](#retrieve-a-single-nodeclaim-by-name "Direct link to Retrieve a single NodeClaim by name") Run the following kubectl command to get NodeClaim `vcluster-62ftb` in project `my-project`: ``` kubectl get nodeclaims.management.loft.sh vcluster-62ftb -n p-my-project -o yaml ``` ### Retrieve a list of NodeClaims[​](#retrieve-a-list-of-nodeclaims-1 "Direct link to Retrieve a list of NodeClaims") Run the following curl command to list all NodeClaims in project `my-project`: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/nodeclaims" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single NodeClaim by name[​](#get-a-single-nodeclaim-by-name "Direct link to Get a single NodeClaim by name") Run the following curl command to get NodeClaim `vcluster-62ftb` in project `my-project`: ``` # Exchange vcluster-62ftb in the url below with the name of the NodeClaim curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/nodeclaims/vcluster-62ftb" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Delete: NodeClaim[​](#delete-nodeclaim "Direct link to Delete: NodeClaim") You can either use curl or kubectl to delete NodeClaims. * kubectl * curl Run the following command to delete NodeClaim `vcluster-62ftb` in project `my-project`: ``` kubectl delete nodeclaims.management.loft.sh vcluster-62ftb -n p-my-project ``` Run the following curl command to delete NodeClaim `vcluster-62ftb` in project `my-project`: ``` # Replace the vcluster-62ftb in the url below with the name of the NodeClaim you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/nodeclaims/vcluster-62ftb" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Node Profile NodeProfile holds reusable node runtime configuration (labels, annotations, and taints) that can be referenced by manual joins, auto nodes, and platform NodeClaims. ## NodeProfile example[​](#nodeprofile-example "Direct link to NodeProfile example") An example NodeProfile: ``` apiVersion: management.loft.sh/v1 kind: NodeProfile metadata: name: gpu-training spec: description: GPU worker defaults for training workloads. displayName: GPU Training nodeAnnotations: ops.vcluster.com/owner: ml-platform nodeLabels: workload.vcluster.com/class: gpu taints: - effect: NoSchedule key: nvidia.com/gpu value: "true" status: {} ``` ## NodeProfile reference[​](#nodeprofile-reference "Direct link to NodeProfile reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI. ### `description` string [​](#spec-description "Direct link to spec-description")Description describes what the profile is intended for. ### `taints` object\[] [​](#spec-taints "Direct link to spec-taints")Taints are taints that should be applied to the joined node. #### `key` required string [​](#spec-taints-key "Direct link to spec-taints-key")Required. The taint key to be applied to a node. #### `value` string [​](#spec-taints-value "Direct link to spec-taints-value")The taint value corresponding to the taint key. #### `effect` required string [​](#spec-taints-effect "Direct link to spec-taints-effect")Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. #### `timeAdded` object [​](#spec-taints-timeAdded "Direct link to spec-taints-timeAdded")TimeAdded represents the time at which the taint was added. ### `startupTaints` object\[] [​](#spec-startupTaints "Direct link to spec-startupTaints")StartupTaints are temporary taints applied while the node initializes. #### `key` required string [​](#spec-startupTaints-key "Direct link to spec-startupTaints-key")Required. The taint key to be applied to a node. #### `value` string [​](#spec-startupTaints-value "Direct link to spec-startupTaints-value")The taint value corresponding to the taint key. #### `effect` required string [​](#spec-startupTaints-effect "Direct link to spec-startupTaints-effect")Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. #### `timeAdded` object [​](#spec-startupTaints-timeAdded "Direct link to spec-startupTaints-timeAdded")TimeAdded represents the time at which the taint was added. ### `nodeLabels` object [​](#spec-nodeLabels "Direct link to spec-nodeLabels")NodeLabels are labels that should be applied to the joined node. ### `nodeAnnotations` object [​](#spec-nodeAnnotations "Direct link to spec-nodeAnnotations")NodeAnnotations are annotations that should be applied to the joined node. ## `status` object [​](#status "Direct link to status") ## Retrieve: NodeProfiles[​](#retrieve-nodeprofiles "Direct link to Retrieve: NodeProfiles") You can either use curl or kubectl to retrieve NodeProfiles. * kubectl * curl ### Retrieve a list of NodeProfiles[​](#retrieve-a-list-of-nodeprofiles "Direct link to Retrieve a list of NodeProfiles") Run the following command to list all NodeProfiles: ``` kubectl get nodeprofiles.management.loft.sh -o yaml ``` ### Retrieve a single NodeProfile by name[​](#retrieve-a-single-nodeprofile-by-name "Direct link to Retrieve a single NodeProfile by name") Run the following kubectl command to get NodeProfile `gpu-training`: ``` kubectl get nodeprofiles.management.loft.sh gpu-training -o yaml ``` ### Retrieve a list of NodeProfiles[​](#retrieve-a-list-of-nodeprofiles-1 "Direct link to Retrieve a list of NodeProfiles") Run the following curl command to list all NodeProfiles: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/nodeprofiles" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single NodeProfile by name[​](#get-a-single-nodeprofile-by-name "Direct link to Get a single NodeProfile by name") Run the following curl command to get NodeProfile `gpu-training`: ``` # Exchange gpu-training in the url below with the name of the NodeProfile curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/nodeprofiles/gpu-training" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: NodeProfile[​](#create-nodeprofile "Direct link to Create: NodeProfile") You can either use curl or kubectl to create a new NodeProfile. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: NodeProfile metadata: name: gpu-training spec: description: GPU worker defaults for training workloads. displayName: GPU Training nodeAnnotations: ops.vcluster.com/owner: ml-platform nodeLabels: workload.vcluster.com/class: gpu taints: - effect: NoSchedule key: nvidia.com/gpu value: "true" status: {} ``` Then create the NodeProfile `gpu-training` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: NodeProfile metadata: name: gpu-training spec: description: GPU worker defaults for training workloads. displayName: GPU Training nodeAnnotations: ops.vcluster.com/owner: ml-platform nodeLabels: workload.vcluster.com/class: gpu taints: - effect: NoSchedule key: nvidia.com/gpu value: "true" status: {} ``` Run the following curl command to create a new NodeProfile `gpu-training`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/nodeprofiles" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: NodeProfile[​](#update-nodeprofile "Direct link to Update: NodeProfile") You can either use curl or kubectl to update NodeProfiles. * kubectl * curl ### Update NodeProfile[​](#update-nodeprofile "Direct link to Update NodeProfile") Run the following command to update NodeProfile `gpu-training`: ``` kubectl edit nodeprofiles.management.loft.sh gpu-training ``` Then edit the object and upon save, kubectl will update the resource. ### Patch NodeProfile[​](#patch-nodeprofile "Direct link to Patch NodeProfile") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the NodeProfile `gpu-training` via a patch: ``` kubectl patch nodeprofiles.management.loft.sh gpu-training \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update NodeProfile[​](#update-nodeprofile-1 "Direct link to Update NodeProfile") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: NodeProfile metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: gpu-training resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: description: GPU worker defaults for training workloads. displayName: GPU Training nodeAnnotations: ops.vcluster.com/owner: ml-platform nodeLabels: workload.vcluster.com/class: gpu taints: - effect: NoSchedule key: nvidia.com/gpu value: "true" status: {} ``` Run the following curl command to update a single NodeProfile `gpu-training`: ``` # Replace the gpu-training in the url below with the name of the NodeProfile you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/nodeprofiles/gpu-training" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch NodeProfile[​](#patch-nodeprofile-1 "Direct link to Patch NodeProfile") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the NodeProfile `gpu-training` via a patch: ``` # Replace the gpu-training in the url below with the name of the NodeProfile you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/nodeprofiles/gpu-training" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: NodeProfile[​](#delete-nodeprofile "Direct link to Delete: NodeProfile") You can either use curl or kubectl to delete NodeProfiles. * kubectl * curl Run the following command to delete NodeProfile `gpu-training`: ``` kubectl delete nodeprofiles.management.loft.sh gpu-training ``` Run the following curl command to delete NodeProfile `gpu-training`: ``` # Replace the gpu-training in the url below with the name of the NodeProfile you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/nodeprofiles/gpu-training" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Node Provider | | | Enterprise | | | | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [ClusterAPI Node Provider](https://vcluster.com/docs/platform/api/resources/nodeprovider.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). NodeProvider holds the node provider information ## NodeProvider example[​](#nodeprovider-example "Direct link to NodeProvider example") An example NodeProvider: ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: {} spec: displayName: Terraform Example terraform: nodeTemplate: git: repository: https://github.com/my-org/my-repo.git nodeTypes: - metadata: {} name: medium properties: instance-type: t3.medium resources: cpu: "2" memory: 4Gi - metadata: {} name: large properties: instance-type: t3.large resources: cpu: "2" memory: 8Gi - metadata: {} name: xlarge properties: instance-type: c5.xlarge resources: cpu: "4" memory: 8Gi status: {} ``` ## NodeProvider reference[​](#nodeprovider-reference "Direct link to NodeProvider reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `properties` object [​](#spec-properties "Direct link to spec-properties")Properties are global properties that are applied to all node claims and environments managed by this provider. ### `bcm` object [​](#spec-bcm "Direct link to spec-bcm")BCM configures a node provider for BCM Bare Metal Cloud environments. #### `secretRef` required object [​](#spec-bcm-secretRef "Direct link to spec-bcm-secretRef")SecretRef is a reference to secret with keys for BCM auth. ##### `name` required string [​](#spec-bcm-secretRef-name "Direct link to spec-bcm-secretRef-name")Name is the name of this resource ##### `namespace` required string [​](#spec-bcm-secretRef-namespace "Direct link to spec-bcm-secretRef-namespace")Namespace is the namespace of this resource #### `endpoint` required string [​](#spec-bcm-endpoint "Direct link to spec-bcm-endpoint")Endpoint is an address for head node. #### `nodeTypes` object\[] [​](#spec-bcm-nodeTypes "Direct link to spec-bcm-nodeTypes")NodeTypes define NodeTypes that should be automatically created for this provider. ##### `providerRef` string [​](#spec-bcm-nodeTypes-providerRef "Direct link to spec-bcm-nodeTypes-providerRef")ProviderRef is the node provider to use for this node type. ##### `properties` object [​](#spec-bcm-nodeTypes-properties "Direct link to spec-bcm-nodeTypes-properties")Properties returns a flexible set of properties that may be selected for scheduling. ##### `resources` object [​](#spec-bcm-nodeTypes-resources "Direct link to spec-bcm-nodeTypes-resources")Resources lists the full resources for a single node. ##### `overhead` object [​](#spec-bcm-nodeTypes-overhead "Direct link to spec-bcm-nodeTypes-overhead")Overhead defines the resource overhead for this node type. ##### `kubeReserved` object [​](#spec-bcm-nodeTypes-overhead-kubeReserved "Direct link to spec-bcm-nodeTypes-overhead-kubeReserved")KubeReserved is the resource overhead for kubelet and other Kubernetes system daemons. ##### `cost` integer [​](#spec-bcm-nodeTypes-cost "Direct link to spec-bcm-nodeTypes-cost")Cost is the instance cost. The higher the cost, the less likely it is to be selected. If empty, cost is automatically calculated from the resources specified. ##### `displayName` string [​](#spec-bcm-nodeTypes-displayName "Direct link to spec-bcm-nodeTypes-displayName")DisplayName is the name that should be displayed in the UI ##### `name` required string [​](#spec-bcm-nodeTypes-name "Direct link to spec-bcm-nodeTypes-name")Name is the name of this node type. ##### `metadata` object [​](#spec-bcm-nodeTypes-metadata "Direct link to spec-bcm-nodeTypes-metadata")Metadata holds metadata to add to this managed NodeType. ##### `labels` object [​](#spec-bcm-nodeTypes-metadata-labels "Direct link to spec-bcm-nodeTypes-metadata-labels")Labels holds labels to add to this managed NodeType. ##### `annotations` object [​](#spec-bcm-nodeTypes-metadata-annotations "Direct link to spec-bcm-nodeTypes-metadata-annotations")Annotations holds annotations to add to this managed NodeType. ##### `nodes` string\[] [​](#spec-bcm-nodeTypes-nodes "Direct link to spec-bcm-nodeTypes-nodes")Nodes specifies nodes. ##### `nodeGroups` string\[] [​](#spec-bcm-nodeTypes-nodeGroups "Direct link to spec-bcm-nodeTypes-nodeGroups")NodeGroups is the name of the node groups to use for this provider. ### `kubeVirt` object [​](#spec-kubeVirt "Direct link to spec-kubeVirt")Kubevirt configures a node provider using KubeVirt, enabling virtual machines to be provisioned as nodes within a vCluster. #### `clusterRef` object [​](#spec-kubeVirt-clusterRef "Direct link to spec-kubeVirt-clusterRef")ClusterRef is a reference to connected control plane cluster in which KubeVirt operator is running ##### `cluster` required string [​](#spec-kubeVirt-clusterRef-cluster "Direct link to spec-kubeVirt-clusterRef-cluster")Cluster is the connected cluster the VMs will be created in ##### `namespace` string [​](#spec-kubeVirt-clusterRef-namespace "Direct link to spec-kubeVirt-clusterRef-namespace")Namespace is the namespace inside the connected cluster holding VMs #### `namespaceStrategy` string [​](#spec-kubeVirt-namespaceStrategy "Direct link to spec-kubeVirt-namespaceStrategy")NamespaceStrategy determines in which namespace of the connected cluster the VirtualMachines are created. "Provider" (default) creates all VirtualMachines in clusterRef.namespace. "VirtualCluster" creates the VirtualMachines in the namespace of the tenant cluster the NodeClaim belongs to. If the NodeClaim cannot be traced back to a tenant cluster namespace within clusterRef.cluster, clusterRef.namespace is used instead. #### `deploy` object [​](#spec-kubeVirt-deploy "Direct link to spec-kubeVirt-deploy")Deploy configures components deployed into the connected control plane cluster. ##### `kubevirt` object [​](#spec-kubeVirt-deploy-kubevirt "Direct link to spec-kubeVirt-deploy-kubevirt")KubeVirt configures the KubeVirt operator deployment. ##### `enabled` required boolean [​](#spec-kubeVirt-deploy-kubevirt-enabled "Direct link to spec-kubeVirt-deploy-kubevirt-enabled")Enabled controls whether the KubeVirt operator is deployed into the cluster. ##### `chartRepo` string [​](#spec-kubeVirt-deploy-kubevirt-chartRepo "Direct link to spec-kubeVirt-deploy-kubevirt-chartRepo")ChartRepo overrides the Helm chart repository used to install the KubeVirt operator. ##### `chart` string [​](#spec-kubeVirt-deploy-kubevirt-chart "Direct link to spec-kubeVirt-deploy-kubevirt-chart")Chart overrides the Helm chart name used to install the KubeVirt operator. ##### `version` string [​](#spec-kubeVirt-deploy-kubevirt-version "Direct link to spec-kubeVirt-deploy-kubevirt-version")Version overrides the Helm chart version used to install the KubeVirt operator. ##### `helmValues` string [​](#spec-kubeVirt-deploy-kubevirt-helmValues "Direct link to spec-kubeVirt-deploy-kubevirt-helmValues")HelmValues is raw YAML that will be passed as values to the KubeVirt Helm chart. ##### `vClusterDeviceOperator` object [​](#spec-kubeVirt-deploy-vClusterDeviceOperator "Direct link to spec-kubeVirt-deploy-vClusterDeviceOperator")VClusterDeviceOperator configures the vCluster device operator deployment. ##### `enabled` required boolean [​](#spec-kubeVirt-deploy-vClusterDeviceOperator-enabled "Direct link to spec-kubeVirt-deploy-vClusterDeviceOperator-enabled")Enabled controls whether the vCluster device operator is deployed into the cluster. ##### `chartRepo` string [​](#spec-kubeVirt-deploy-vClusterDeviceOperator-chartRepo "Direct link to spec-kubeVirt-deploy-vClusterDeviceOperator-chartRepo")ChartRepo overrides the Helm chart repository used to install the operator. ##### `chart` string [​](#spec-kubeVirt-deploy-vClusterDeviceOperator-chart "Direct link to spec-kubeVirt-deploy-vClusterDeviceOperator-chart")Chart overrides the Helm chart name used to install the operator. ##### `version` string [​](#spec-kubeVirt-deploy-vClusterDeviceOperator-version "Direct link to spec-kubeVirt-deploy-vClusterDeviceOperator-version")Version overrides the Helm chart version used to install the operator. ##### `helmValues` string [​](#spec-kubeVirt-deploy-vClusterDeviceOperator-helmValues "Direct link to spec-kubeVirt-deploy-vClusterDeviceOperator-helmValues")HelmValues is raw YAML that will be passed as values to the Helm chart. #### `virtualMachineTemplate` object [​](#spec-kubeVirt-virtualMachineTemplate "Direct link to spec-kubeVirt-virtualMachineTemplate")VirtualMachineTemplate is a KubeVirt VirtualMachine template to use by NodeTypes managed by this NodeProvider #### `nodeTypes` required object\[] [​](#spec-kubeVirt-nodeTypes "Direct link to spec-kubeVirt-nodeTypes")NodeTypes define NodeTypes that should be automatically created for this provider. ##### `providerRef` string [​](#spec-kubeVirt-nodeTypes-providerRef "Direct link to spec-kubeVirt-nodeTypes-providerRef")ProviderRef is the node provider to use for this node type. ##### `properties` object [​](#spec-kubeVirt-nodeTypes-properties "Direct link to spec-kubeVirt-nodeTypes-properties")Properties returns a flexible set of properties that may be selected for scheduling. ##### `resources` object [​](#spec-kubeVirt-nodeTypes-resources "Direct link to spec-kubeVirt-nodeTypes-resources")Resources lists the full resources for a single node. ##### `overhead` object [​](#spec-kubeVirt-nodeTypes-overhead "Direct link to spec-kubeVirt-nodeTypes-overhead")Overhead defines the resource overhead for this node type. ##### `kubeReserved` object [​](#spec-kubeVirt-nodeTypes-overhead-kubeReserved "Direct link to spec-kubeVirt-nodeTypes-overhead-kubeReserved")KubeReserved is the resource overhead for kubelet and other Kubernetes system daemons. ##### `cost` integer [​](#spec-kubeVirt-nodeTypes-cost "Direct link to spec-kubeVirt-nodeTypes-cost")Cost is the instance cost. The higher the cost, the less likely it is to be selected. If empty, cost is automatically calculated from the resources specified. ##### `displayName` string [​](#spec-kubeVirt-nodeTypes-displayName "Direct link to spec-kubeVirt-nodeTypes-displayName")DisplayName is the name that should be displayed in the UI ##### `name` required string [​](#spec-kubeVirt-nodeTypes-name "Direct link to spec-kubeVirt-nodeTypes-name")Name is the name of this node type. ##### `metadata` object [​](#spec-kubeVirt-nodeTypes-metadata "Direct link to spec-kubeVirt-nodeTypes-metadata")Metadata holds metadata to add to this managed NodeType. ##### `labels` object [​](#spec-kubeVirt-nodeTypes-metadata-labels "Direct link to spec-kubeVirt-nodeTypes-metadata-labels")Labels holds labels to add to this managed NodeType. ##### `annotations` object [​](#spec-kubeVirt-nodeTypes-metadata-annotations "Direct link to spec-kubeVirt-nodeTypes-metadata-annotations")Annotations holds annotations to add to this managed NodeType. ##### `virtualMachineTemplate` object [​](#spec-kubeVirt-nodeTypes-virtualMachineTemplate "Direct link to spec-kubeVirt-nodeTypes-virtualMachineTemplate")VirtualMachineTemplate is a full KubeVirt VirtualMachine template to use for this NodeType. This is mutually exclusive with MergeVirtualMachineTemplate ##### `mergeVirtualMachineTemplate` object [​](#spec-kubeVirt-nodeTypes-mergeVirtualMachineTemplate "Direct link to spec-kubeVirt-nodeTypes-mergeVirtualMachineTemplate")MergeVirtualMachineTemplate will be merged into base VirtualMachine template for this NodeProvider. This allows overwriting of specific fields from top level template by individual NodeTypes This is mutually exclusive with VirtualMachineTemplate ##### `maxCapacity` integer [​](#spec-kubeVirt-nodeTypes-maxCapacity "Direct link to spec-kubeVirt-nodeTypes-maxCapacity")MaxCapacity is the maximum number of nodes that can be created for this NodeType. ### `terraform` object [​](#spec-terraform "Direct link to spec-terraform")Terraform configures a node provider using Terraform, enabling nodes to be provisioned using Terraform. #### `nodeTemplate` object [​](#spec-terraform-nodeTemplate "Direct link to spec-terraform-nodeTemplate")NodeTemplate is the template to use for this node provider. ##### `inline` string [​](#spec-terraform-nodeTemplate-inline "Direct link to spec-terraform-nodeTemplate-inline")Inline is the inline template to use for this node type. ##### `git` object [​](#spec-terraform-nodeTemplate-git "Direct link to spec-terraform-nodeTemplate-git")Git is the git repository to use for this node type. ##### `repository` string [​](#spec-terraform-nodeTemplate-git-repository "Direct link to spec-terraform-nodeTemplate-git-repository")Repository is the repository to clone ##### `branch` string [​](#spec-terraform-nodeTemplate-git-branch "Direct link to spec-terraform-nodeTemplate-git-branch")Branch is the branch to use ##### `commit` string [​](#spec-terraform-nodeTemplate-git-commit "Direct link to spec-terraform-nodeTemplate-git-commit")Commit is the commit SHA to checkout ##### `tag` string [​](#spec-terraform-nodeTemplate-git-tag "Direct link to spec-terraform-nodeTemplate-git-tag")Tag is the tag reference to checkout ##### `subPath` string [​](#spec-terraform-nodeTemplate-git-subPath "Direct link to spec-terraform-nodeTemplate-git-subPath")SubPath is the subpath in the repo to use ##### `credentials` object [​](#spec-terraform-nodeTemplate-git-credentials "Direct link to spec-terraform-nodeTemplate-git-credentials")Credentials is the reference to a secret containing the username and password for the git repository. ##### `secretName` string [​](#spec-terraform-nodeTemplate-git-credentials-secretName "Direct link to spec-terraform-nodeTemplate-git-credentials-secretName") ##### `secretNamespace` string [​](#spec-terraform-nodeTemplate-git-credentials-secretNamespace "Direct link to spec-terraform-nodeTemplate-git-credentials-secretNamespace") ##### `key` string [​](#spec-terraform-nodeTemplate-git-credentials-key "Direct link to spec-terraform-nodeTemplate-git-credentials-key") ##### `fetchInterval` string [​](#spec-terraform-nodeTemplate-git-fetchInterval "Direct link to spec-terraform-nodeTemplate-git-fetchInterval")FetchInterval is the interval to use for refetching the git repository. Defaults to 5m. Refetching only checks for remote changes but does not do a complete repull. ##### `extraEnv` string\[] [​](#spec-terraform-nodeTemplate-git-extraEnv "Direct link to spec-terraform-nodeTemplate-git-extraEnv")ExtraEnv is the extra environment variables to use for the clone ##### `timeout` string [​](#spec-terraform-nodeTemplate-timeout "Direct link to spec-terraform-nodeTemplate-timeout")Timeout is the timeout to use for the terraform operations. Defaults to 60m. #### `nodeEnvironmentTemplate` object [​](#spec-terraform-nodeEnvironmentTemplate "Direct link to spec-terraform-nodeEnvironmentTemplate")NodeEnvironmentTemplate is the template to use for this node environment. ##### `inline` string [​](#spec-terraform-nodeEnvironmentTemplate-inline "Direct link to spec-terraform-nodeEnvironmentTemplate-inline")Inline is the inline template to use for this node type. ##### `git` object [​](#spec-terraform-nodeEnvironmentTemplate-git "Direct link to spec-terraform-nodeEnvironmentTemplate-git")Git is the git repository to use for this node type. ##### `repository` string [​](#spec-terraform-nodeEnvironmentTemplate-git-repository "Direct link to spec-terraform-nodeEnvironmentTemplate-git-repository")Repository is the repository to clone ##### `branch` string [​](#spec-terraform-nodeEnvironmentTemplate-git-branch "Direct link to spec-terraform-nodeEnvironmentTemplate-git-branch")Branch is the branch to use ##### `commit` string [​](#spec-terraform-nodeEnvironmentTemplate-git-commit "Direct link to spec-terraform-nodeEnvironmentTemplate-git-commit")Commit is the commit SHA to checkout ##### `tag` string [​](#spec-terraform-nodeEnvironmentTemplate-git-tag "Direct link to spec-terraform-nodeEnvironmentTemplate-git-tag")Tag is the tag reference to checkout ##### `subPath` string [​](#spec-terraform-nodeEnvironmentTemplate-git-subPath "Direct link to spec-terraform-nodeEnvironmentTemplate-git-subPath")SubPath is the subpath in the repo to use ##### `credentials` object [​](#spec-terraform-nodeEnvironmentTemplate-git-credentials "Direct link to spec-terraform-nodeEnvironmentTemplate-git-credentials")Credentials is the reference to a secret containing the username and password for the git repository. ##### `secretName` string [​](#spec-terraform-nodeEnvironmentTemplate-git-credentials-secretName "Direct link to spec-terraform-nodeEnvironmentTemplate-git-credentials-secretName") ##### `secretNamespace` string [​](#spec-terraform-nodeEnvironmentTemplate-git-credentials-secretNamespace "Direct link to spec-terraform-nodeEnvironmentTemplate-git-credentials-secretNamespace") ##### `key` string [​](#spec-terraform-nodeEnvironmentTemplate-git-credentials-key "Direct link to spec-terraform-nodeEnvironmentTemplate-git-credentials-key") ##### `fetchInterval` string [​](#spec-terraform-nodeEnvironmentTemplate-git-fetchInterval "Direct link to spec-terraform-nodeEnvironmentTemplate-git-fetchInterval")FetchInterval is the interval to use for refetching the git repository. Defaults to 5m. Refetching only checks for remote changes but does not do a complete repull. ##### `extraEnv` string\[] [​](#spec-terraform-nodeEnvironmentTemplate-git-extraEnv "Direct link to spec-terraform-nodeEnvironmentTemplate-git-extraEnv")ExtraEnv is the extra environment variables to use for the clone ##### `timeout` string [​](#spec-terraform-nodeEnvironmentTemplate-timeout "Direct link to spec-terraform-nodeEnvironmentTemplate-timeout")Timeout is the timeout to use for the terraform operations. Defaults to 60m. ##### `infrastructure` object [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure")Infrastructure is the infrastructure template to use for this node environment. ##### `inline` string [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure-inline "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure-inline")Inline is the inline template to use for this node type. ##### `git` object [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure-git "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure-git")Git is the git repository to use for this node type. ##### `repository` string [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure-git-repository "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure-git-repository")Repository is the repository to clone ##### `branch` string [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure-git-branch "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure-git-branch")Branch is the branch to use ##### `commit` string [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure-git-commit "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure-git-commit")Commit is the commit SHA to checkout ##### `tag` string [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure-git-tag "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure-git-tag")Tag is the tag reference to checkout ##### `subPath` string [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure-git-subPath "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure-git-subPath")SubPath is the subpath in the repo to use ##### `credentials` object [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure-git-credentials "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure-git-credentials")Credentials is the reference to a secret containing the username and password for the git repository. ##### `secretName` string [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure-git-credentials-secretName "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure-git-credentials-secretName") ##### `secretNamespace` string [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure-git-credentials-secretNamespace "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure-git-credentials-secretNamespace") ##### `key` string [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure-git-credentials-key "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure-git-credentials-key") ##### `fetchInterval` string [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure-git-fetchInterval "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure-git-fetchInterval")FetchInterval is the interval to use for refetching the git repository. Defaults to 5m. Refetching only checks for remote changes but does not do a complete repull. ##### `extraEnv` string\[] [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure-git-extraEnv "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure-git-extraEnv")ExtraEnv is the extra environment variables to use for the clone ##### `timeout` string [​](#spec-terraform-nodeEnvironmentTemplate-infrastructure-timeout "Direct link to spec-terraform-nodeEnvironmentTemplate-infrastructure-timeout")Timeout is the timeout to use for the terraform operations. Defaults to 60m. ##### `kubernetes` object [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes")Kubernetes is the kubernetes template to use for this node environment. ##### `inline` string [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes-inline "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes-inline")Inline is the inline template to use for this node type. ##### `git` object [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes-git "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes-git")Git is the git repository to use for this node type. ##### `repository` string [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes-git-repository "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes-git-repository")Repository is the repository to clone ##### `branch` string [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes-git-branch "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes-git-branch")Branch is the branch to use ##### `commit` string [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes-git-commit "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes-git-commit")Commit is the commit SHA to checkout ##### `tag` string [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes-git-tag "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes-git-tag")Tag is the tag reference to checkout ##### `subPath` string [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes-git-subPath "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes-git-subPath")SubPath is the subpath in the repo to use ##### `credentials` object [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes-git-credentials "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes-git-credentials")Credentials is the reference to a secret containing the username and password for the git repository. ##### `secretName` string [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes-git-credentials-secretName "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes-git-credentials-secretName") ##### `secretNamespace` string [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes-git-credentials-secretNamespace "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes-git-credentials-secretNamespace") ##### `key` string [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes-git-credentials-key "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes-git-credentials-key") ##### `fetchInterval` string [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes-git-fetchInterval "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes-git-fetchInterval")FetchInterval is the interval to use for refetching the git repository. Defaults to 5m. Refetching only checks for remote changes but does not do a complete repull. ##### `extraEnv` string\[] [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes-git-extraEnv "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes-git-extraEnv")ExtraEnv is the extra environment variables to use for the clone ##### `timeout` string [​](#spec-terraform-nodeEnvironmentTemplate-kubernetes-timeout "Direct link to spec-terraform-nodeEnvironmentTemplate-kubernetes-timeout")Timeout is the timeout to use for the terraform operations. Defaults to 60m. #### `nodeTypes` object\[] [​](#spec-terraform-nodeTypes "Direct link to spec-terraform-nodeTypes")NodeTypes define NodeTypes that should be automatically created for this provider. ##### `providerRef` string [​](#spec-terraform-nodeTypes-providerRef "Direct link to spec-terraform-nodeTypes-providerRef")ProviderRef is the node provider to use for this node type. ##### `properties` object [​](#spec-terraform-nodeTypes-properties "Direct link to spec-terraform-nodeTypes-properties")Properties returns a flexible set of properties that may be selected for scheduling. ##### `resources` object [​](#spec-terraform-nodeTypes-resources "Direct link to spec-terraform-nodeTypes-resources")Resources lists the full resources for a single node. ##### `overhead` object [​](#spec-terraform-nodeTypes-overhead "Direct link to spec-terraform-nodeTypes-overhead")Overhead defines the resource overhead for this node type. ##### `kubeReserved` object [​](#spec-terraform-nodeTypes-overhead-kubeReserved "Direct link to spec-terraform-nodeTypes-overhead-kubeReserved")KubeReserved is the resource overhead for kubelet and other Kubernetes system daemons. ##### `cost` integer [​](#spec-terraform-nodeTypes-cost "Direct link to spec-terraform-nodeTypes-cost")Cost is the instance cost. The higher the cost, the less likely it is to be selected. If empty, cost is automatically calculated from the resources specified. ##### `displayName` string [​](#spec-terraform-nodeTypes-displayName "Direct link to spec-terraform-nodeTypes-displayName")DisplayName is the name that should be displayed in the UI ##### `name` required string [​](#spec-terraform-nodeTypes-name "Direct link to spec-terraform-nodeTypes-name")Name is the name of this node type. ##### `metadata` object [​](#spec-terraform-nodeTypes-metadata "Direct link to spec-terraform-nodeTypes-metadata")Metadata holds metadata to add to this managed NodeType. ##### `labels` object [​](#spec-terraform-nodeTypes-metadata-labels "Direct link to spec-terraform-nodeTypes-metadata-labels")Labels holds labels to add to this managed NodeType. ##### `annotations` object [​](#spec-terraform-nodeTypes-metadata-annotations "Direct link to spec-terraform-nodeTypes-metadata-annotations")Annotations holds annotations to add to this managed NodeType. ##### `nodeTemplate` object [​](#spec-terraform-nodeTypes-nodeTemplate "Direct link to spec-terraform-nodeTypes-nodeTemplate")NodeTemplate is the template to use for this node type. ##### `inline` string [​](#spec-terraform-nodeTypes-nodeTemplate-inline "Direct link to spec-terraform-nodeTypes-nodeTemplate-inline")Inline is the inline template to use for this node type. ##### `git` object [​](#spec-terraform-nodeTypes-nodeTemplate-git "Direct link to spec-terraform-nodeTypes-nodeTemplate-git")Git is the git repository to use for this node type. ##### `repository` string [​](#spec-terraform-nodeTypes-nodeTemplate-git-repository "Direct link to spec-terraform-nodeTypes-nodeTemplate-git-repository")Repository is the repository to clone ##### `branch` string [​](#spec-terraform-nodeTypes-nodeTemplate-git-branch "Direct link to spec-terraform-nodeTypes-nodeTemplate-git-branch")Branch is the branch to use ##### `commit` string [​](#spec-terraform-nodeTypes-nodeTemplate-git-commit "Direct link to spec-terraform-nodeTypes-nodeTemplate-git-commit")Commit is the commit SHA to checkout ##### `tag` string [​](#spec-terraform-nodeTypes-nodeTemplate-git-tag "Direct link to spec-terraform-nodeTypes-nodeTemplate-git-tag")Tag is the tag reference to checkout ##### `subPath` string [​](#spec-terraform-nodeTypes-nodeTemplate-git-subPath "Direct link to spec-terraform-nodeTypes-nodeTemplate-git-subPath")SubPath is the subpath in the repo to use ##### `credentials` object [​](#spec-terraform-nodeTypes-nodeTemplate-git-credentials "Direct link to spec-terraform-nodeTypes-nodeTemplate-git-credentials")Credentials is the reference to a secret containing the username and password for the git repository. ##### `secretName` string [​](#spec-terraform-nodeTypes-nodeTemplate-git-credentials-secretName "Direct link to spec-terraform-nodeTypes-nodeTemplate-git-credentials-secretName") ##### `secretNamespace` string [​](#spec-terraform-nodeTypes-nodeTemplate-git-credentials-secretNamespace "Direct link to spec-terraform-nodeTypes-nodeTemplate-git-credentials-secretNamespace") ##### `key` string [​](#spec-terraform-nodeTypes-nodeTemplate-git-credentials-key "Direct link to spec-terraform-nodeTypes-nodeTemplate-git-credentials-key") ##### `fetchInterval` string [​](#spec-terraform-nodeTypes-nodeTemplate-git-fetchInterval "Direct link to spec-terraform-nodeTypes-nodeTemplate-git-fetchInterval")FetchInterval is the interval to use for refetching the git repository. Defaults to 5m. Refetching only checks for remote changes but does not do a complete repull. ##### `extraEnv` string\[] [​](#spec-terraform-nodeTypes-nodeTemplate-git-extraEnv "Direct link to spec-terraform-nodeTypes-nodeTemplate-git-extraEnv")ExtraEnv is the extra environment variables to use for the clone ##### `timeout` string [​](#spec-terraform-nodeTypes-nodeTemplate-timeout "Direct link to spec-terraform-nodeTypes-nodeTemplate-timeout")Timeout is the timeout to use for the terraform operations. Defaults to 60m. ##### `maxCapacity` integer [​](#spec-terraform-nodeTypes-maxCapacity "Direct link to spec-terraform-nodeTypes-maxCapacity")MaxCapacity is the maximum number of nodes that can be created for this NodeType. ### `clusterAPI` object [​](#spec-clusterAPI "Direct link to spec-clusterAPI")ClusterAPI configures a node provider using Cluster API, enabling nodes to be provisioned using Cluster API. This requires the vCluster to be deployed with Cluster API as well. #### `infrastructureMachineTemplate` object [​](#spec-clusterAPI-infrastructureMachineTemplate "Direct link to spec-clusterAPI-infrastructureMachineTemplate")InfrastructureMachineTemplate is a template for the infrastructure machine, e.g. AWSMachine #### `bootstrapConfigTemplate` object [​](#spec-clusterAPI-bootstrapConfigTemplate "Direct link to spec-clusterAPI-bootstrapConfigTemplate")BootstrapConfigTemplate is a template for the bootstrap config. Currently only KubeadmConfig is supported. #### `clusterRef` object [​](#spec-clusterAPI-clusterRef "Direct link to spec-clusterAPI-clusterRef")ClusterRef is a reference to connected control plane cluster in which KubeVirt operator is running ##### `cluster` required string [​](#spec-clusterAPI-clusterRef-cluster "Direct link to spec-clusterAPI-clusterRef-cluster")Cluster is the connected cluster the VMs will be created in ##### `namespace` string [​](#spec-clusterAPI-clusterRef-namespace "Direct link to spec-clusterAPI-clusterRef-namespace")Namespace is the namespace inside the connected cluster holding VMs #### `nodeTypes` object\[] [​](#spec-clusterAPI-nodeTypes "Direct link to spec-clusterAPI-nodeTypes")NodeTypes define NodeTypes that should be automatically created for this provider. ##### `providerRef` string [​](#spec-clusterAPI-nodeTypes-providerRef "Direct link to spec-clusterAPI-nodeTypes-providerRef")ProviderRef is the node provider to use for this node type. ##### `properties` object [​](#spec-clusterAPI-nodeTypes-properties "Direct link to spec-clusterAPI-nodeTypes-properties")Properties returns a flexible set of properties that may be selected for scheduling. ##### `resources` object [​](#spec-clusterAPI-nodeTypes-resources "Direct link to spec-clusterAPI-nodeTypes-resources")Resources lists the full resources for a single node. ##### `overhead` object [​](#spec-clusterAPI-nodeTypes-overhead "Direct link to spec-clusterAPI-nodeTypes-overhead")Overhead defines the resource overhead for this node type. ##### `kubeReserved` object [​](#spec-clusterAPI-nodeTypes-overhead-kubeReserved "Direct link to spec-clusterAPI-nodeTypes-overhead-kubeReserved")KubeReserved is the resource overhead for kubelet and other Kubernetes system daemons. ##### `cost` integer [​](#spec-clusterAPI-nodeTypes-cost "Direct link to spec-clusterAPI-nodeTypes-cost")Cost is the instance cost. The higher the cost, the less likely it is to be selected. If empty, cost is automatically calculated from the resources specified. ##### `displayName` string [​](#spec-clusterAPI-nodeTypes-displayName "Direct link to spec-clusterAPI-nodeTypes-displayName")DisplayName is the name that should be displayed in the UI ##### `name` required string [​](#spec-clusterAPI-nodeTypes-name "Direct link to spec-clusterAPI-nodeTypes-name")Name is the name of this node type. ##### `metadata` object [​](#spec-clusterAPI-nodeTypes-metadata "Direct link to spec-clusterAPI-nodeTypes-metadata")Metadata holds metadata to add to this managed NodeType. ##### `labels` object [​](#spec-clusterAPI-nodeTypes-metadata-labels "Direct link to spec-clusterAPI-nodeTypes-metadata-labels")Labels holds labels to add to this managed NodeType. ##### `annotations` object [​](#spec-clusterAPI-nodeTypes-metadata-annotations "Direct link to spec-clusterAPI-nodeTypes-metadata-annotations")Annotations holds annotations to add to this managed NodeType. ##### `infrastructureMachineTemplate` object [​](#spec-clusterAPI-nodeTypes-infrastructureMachineTemplate "Direct link to spec-clusterAPI-nodeTypes-infrastructureMachineTemplate")InfrastructureMachineTemplate is a template for the infrastructure machine, e.g. AWSMachine ##### `bootstrapConfigTemplate` object [​](#spec-clusterAPI-nodeTypes-bootstrapConfigTemplate "Direct link to spec-clusterAPI-nodeTypes-bootstrapConfigTemplate")BootstrapConfigTemplate is a template for the bootstrap config. Currently only KubeadmConfig is supported. ##### `mergeInfrastructureMachineTemplate` object [​](#spec-clusterAPI-nodeTypes-mergeInfrastructureMachineTemplate "Direct link to spec-clusterAPI-nodeTypes-mergeInfrastructureMachineTemplate")MergeInfrastructureMachineTemplate will be merged into base InfrastructureMachine template for this NodeProvider. This allows overwriting of specific fields from top level template by individual NodeTypes This is mutually exclusive with InfrastructureMachineTemplate ##### `mergeBootstrapConfigTemplate` object [​](#spec-clusterAPI-nodeTypes-mergeBootstrapConfigTemplate "Direct link to spec-clusterAPI-nodeTypes-mergeBootstrapConfigTemplate")MergeBootstrapConfigTemplate will be merged into base BootstrapConfig template for this NodeProvider. This allows overwriting of specific fields from top level template by individual NodeTypes This is mutually exclusive with BootstrapConfigTemplate ##### `maxCapacity` integer [​](#spec-clusterAPI-nodeTypes-maxCapacity "Direct link to spec-clusterAPI-nodeTypes-maxCapacity")MaxCapacity is the maximum number of nodes that can be created for this NodeType. ### `metal3` object [​](#spec-metal3 "Direct link to spec-metal3")Metal3 configures a node provider using metal3.io BareMetalHost resources. #### `clusterRef` object [​](#spec-metal3-clusterRef "Direct link to spec-metal3-clusterRef")ClusterRef is a reference to connected control plane cluster in which KubeVirt operator is running ##### `cluster` required string [​](#spec-metal3-clusterRef-cluster "Direct link to spec-metal3-clusterRef-cluster")Cluster is the connected cluster the VMs will be created in ##### `namespace` string [​](#spec-metal3-clusterRef-namespace "Direct link to spec-metal3-clusterRef-namespace")Namespace is the namespace inside the connected cluster holding VMs #### `deploy` object [​](#spec-metal3-deploy "Direct link to spec-metal3-deploy") ##### `multus` object [​](#spec-metal3-deploy-multus "Direct link to spec-metal3-deploy-multus")Multus configures the Multus CNI deployment. ##### `enabled` required boolean [​](#spec-metal3-deploy-multus-enabled "Direct link to spec-metal3-deploy-multus-enabled")Enabled controls whether Multus CNI is deployed into the cluster. ##### `helmValues` string [​](#spec-metal3-deploy-multus-helmValues "Direct link to spec-metal3-deploy-multus-helmValues")HelmValues is raw YAML that will be passed as values to the Multus Helm chart. ##### `dhcp` object [​](#spec-metal3-deploy-dhcp "Direct link to spec-metal3-deploy-dhcp")DHCP configures the DHCP server deployment. ##### `enabled` required boolean [​](#spec-metal3-deploy-dhcp-enabled "Direct link to spec-metal3-deploy-dhcp-enabled")Enabled controls whether the DHCP server is deployed into the cluster. ##### `chartRepo` string [​](#spec-metal3-deploy-dhcp-chartRepo "Direct link to spec-metal3-deploy-dhcp-chartRepo")ChartRepo overrides the Helm chart repository used to install the DHCP server. ##### `chart` string [​](#spec-metal3-deploy-dhcp-chart "Direct link to spec-metal3-deploy-dhcp-chart")Chart overrides the Helm chart name used to install the DHCP server. ##### `version` string [​](#spec-metal3-deploy-dhcp-version "Direct link to spec-metal3-deploy-dhcp-version")Version overrides the Helm chart version used to install the DHCP server. ##### `helmValues` string [​](#spec-metal3-deploy-dhcp-helmValues "Direct link to spec-metal3-deploy-dhcp-helmValues")HelmValues is raw YAML that will be passed as values to the DHCP Helm chart. ##### `metal3` object [​](#spec-metal3-deploy-metal3 "Direct link to spec-metal3-deploy-metal3")Metal3 configures the Metal3/Ironic deployment. ##### `enabled` required boolean [​](#spec-metal3-deploy-metal3-enabled "Direct link to spec-metal3-deploy-metal3-enabled")Enabled controls whether Metal3 and Ironic are deployed into the cluster. ##### `chartRepo` string [​](#spec-metal3-deploy-metal3-chartRepo "Direct link to spec-metal3-deploy-metal3-chartRepo")ChartRepo overrides the Helm chart repository used to install Metal3. ##### `chart` string [​](#spec-metal3-deploy-metal3-chart "Direct link to spec-metal3-deploy-metal3-chart")Chart overrides the Helm chart name used to install Metal3. ##### `version` string [​](#spec-metal3-deploy-metal3-version "Direct link to spec-metal3-deploy-metal3-version")Version overrides the Helm chart version used to install Metal3. ##### `helmValues` string [​](#spec-metal3-deploy-metal3-helmValues "Direct link to spec-metal3-deploy-metal3-helmValues")HelmValues is raw YAML that will be passed as values to the Metal3 Helm chart. #### `nodeTypes` object\[] [​](#spec-metal3-nodeTypes "Direct link to spec-metal3-nodeTypes")NodeTypes define NodeTypes that should be automatically created for this provider. ##### `providerRef` string [​](#spec-metal3-nodeTypes-providerRef "Direct link to spec-metal3-nodeTypes-providerRef")ProviderRef is the node provider to use for this node type. ##### `properties` object [​](#spec-metal3-nodeTypes-properties "Direct link to spec-metal3-nodeTypes-properties")Properties returns a flexible set of properties that may be selected for scheduling. ##### `resources` object [​](#spec-metal3-nodeTypes-resources "Direct link to spec-metal3-nodeTypes-resources")Resources lists the full resources for a single node. ##### `overhead` object [​](#spec-metal3-nodeTypes-overhead "Direct link to spec-metal3-nodeTypes-overhead")Overhead defines the resource overhead for this node type. ##### `kubeReserved` object [​](#spec-metal3-nodeTypes-overhead-kubeReserved "Direct link to spec-metal3-nodeTypes-overhead-kubeReserved")KubeReserved is the resource overhead for kubelet and other Kubernetes system daemons. ##### `cost` integer [​](#spec-metal3-nodeTypes-cost "Direct link to spec-metal3-nodeTypes-cost")Cost is the instance cost. The higher the cost, the less likely it is to be selected. If empty, cost is automatically calculated from the resources specified. ##### `displayName` string [​](#spec-metal3-nodeTypes-displayName "Direct link to spec-metal3-nodeTypes-displayName")DisplayName is the name that should be displayed in the UI ##### `name` required string [​](#spec-metal3-nodeTypes-name "Direct link to spec-metal3-nodeTypes-name")Name is the name of this node type. ##### `metadata` object [​](#spec-metal3-nodeTypes-metadata "Direct link to spec-metal3-nodeTypes-metadata")Metadata holds metadata to add to this managed NodeType. ##### `labels` object [​](#spec-metal3-nodeTypes-metadata-labels "Direct link to spec-metal3-nodeTypes-metadata-labels")Labels holds labels to add to this managed NodeType. ##### `annotations` object [​](#spec-metal3-nodeTypes-metadata-annotations "Direct link to spec-metal3-nodeTypes-metadata-annotations")Annotations holds annotations to add to this managed NodeType. ##### `bareMetalHosts` object [​](#spec-metal3-nodeTypes-bareMetalHosts "Direct link to spec-metal3-nodeTypes-bareMetalHosts")BareMetalHosts is a list of BareMetalHosts to use for this NodeType. ##### `selector` object [​](#spec-metal3-nodeTypes-bareMetalHosts-selector "Direct link to spec-metal3-nodeTypes-bareMetalHosts-selector")Selector is a label selector to select the BareMetalHosts to use for this NodeType. ##### `matchLabels` object [​](#spec-metal3-nodeTypes-bareMetalHosts-selector-matchLabels "Direct link to spec-metal3-nodeTypes-bareMetalHosts-selector-matchLabels")matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. ##### `matchExpressions` object\[] [​](#spec-metal3-nodeTypes-bareMetalHosts-selector-matchExpressions "Direct link to spec-metal3-nodeTypes-bareMetalHosts-selector-matchExpressions")matchExpressions is a list of label selector requirements. The requirements are ANDed. ##### `key` required string [​](#spec-metal3-nodeTypes-bareMetalHosts-selector-matchExpressions-key "Direct link to spec-metal3-nodeTypes-bareMetalHosts-selector-matchExpressions-key")key is the label key that the selector applies to. ##### `operator` required string [​](#spec-metal3-nodeTypes-bareMetalHosts-selector-matchExpressions-operator "Direct link to spec-metal3-nodeTypes-bareMetalHosts-selector-matchExpressions-operator")operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. ##### `values` string\[] [​](#spec-metal3-nodeTypes-bareMetalHosts-selector-matchExpressions-values "Direct link to spec-metal3-nodeTypes-bareMetalHosts-selector-matchExpressions-values")values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. #### `neutronEnabled` boolean [​](#spec-metal3-neutronEnabled "Direct link to spec-metal3-neutronEnabled")NeutronEnabled turns on the neutron network shim for this provider: BareMetalHost network attachments are allocated by the platform and reconciled through ConfigMaps instead of being written directly as DHCP annotations. #### `netris` object [​](#spec-metal3-netris "Direct link to spec-metal3-netris")Netris attaches BareMetalHosts to a Netris server cluster on provisioning. ##### `secretRef` required object [​](#spec-metal3-netris-secretRef "Direct link to spec-metal3-netris-secretRef")SecretRef references a Secret with keys url, username and password for the Netris API. ##### `name` required string [​](#spec-metal3-netris-secretRef-name "Direct link to spec-metal3-netris-secretRef-name")Name is the name of this resource ##### `namespace` required string [​](#spec-metal3-netris-secretRef-namespace "Direct link to spec-metal3-netris-secretRef-namespace")Namespace is the namespace of this resource ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ## `status` object [​](#status "Direct link to status") ### `conditions` object [​](#status-conditions "Direct link to status-conditions")Conditions describe the current state of the platform NodeProvider. ### `reason` string [​](#status-reason "Direct link to status-reason")Reason describes the reason in machine-readable form ### `phase` string [​](#status-phase "Direct link to status-phase")Phase is the current lifecycle phase of the NodeProvider. ### `message` string [​](#status-message "Direct link to status-message")Message is a human-readable message indicating details about why the NodeProvider is in its current state. ## Create: NodeProvider[​](#create-nodeprovider "Direct link to Create: NodeProvider") You can either use curl or kubectl to create a new NodeProvider. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: {} spec: displayName: Terraform Example terraform: nodeTemplate: git: repository: https://github.com/my-org/my-repo.git nodeTypes: - metadata: {} name: medium properties: instance-type: t3.medium resources: cpu: "2" memory: 4Gi - metadata: {} name: large properties: instance-type: t3.large resources: cpu: "2" memory: 8Gi - metadata: {} name: xlarge properties: instance-type: c5.xlarge resources: cpu: "4" memory: 8Gi status: {} ``` Then create the NodeProvider `my-object` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: {} spec: displayName: Terraform Example terraform: nodeTemplate: git: repository: https://github.com/my-org/my-repo.git nodeTypes: - metadata: {} name: medium properties: instance-type: t3.medium resources: cpu: "2" memory: 4Gi - metadata: {} name: large properties: instance-type: t3.large resources: cpu: "2" memory: 8Gi - metadata: {} name: xlarge properties: instance-type: c5.xlarge resources: cpu: "4" memory: 8Gi status: {} ``` Run the following curl command to create a new NodeProvider `my-object`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/nodeproviders" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # OS Image OSImage describes an operating system image available for bare metal provisioning. ## OSImage example[​](#osimage-example "Direct link to OSImage example") An example OSImage: ``` apiVersion: management.loft.sh/v1 kind: OSImage metadata: name: ubuntu-noble spec: description: Ubuntu Noble (24.04) server cloud image for bare metal provisioning displayName: Ubuntu Noble (24.04 LTS) properties: metal3.vcluster.com/image-checksum: https://cloud-images.ubuntu.com/noble/current/SHA256SUMS metal3.vcluster.com/image-checksum-type: sha256 metal3.vcluster.com/image-url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img status: {} ``` ## OSImage reference[​](#osimage-reference "Direct link to OSImage reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes an OS image ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ### `properties` object [​](#spec-properties "Direct link to spec-properties")Properties is the configuration for the OS image ## `status` object [​](#status "Direct link to status") ## Retrieve: OSImages[​](#retrieve-osimages "Direct link to Retrieve: OSImages") You can either use curl or kubectl to retrieve OSImages. * kubectl * curl ### Retrieve a list of OSImages[​](#retrieve-a-list-of-osimages "Direct link to Retrieve a list of OSImages") Run the following command to list all OSImages: ``` kubectl get osimages.management.loft.sh -o yaml ``` ### Retrieve a single OSImage by name[​](#retrieve-a-single-osimage-by-name "Direct link to Retrieve a single OSImage by name") Run the following kubectl command to get OSImage `ubuntu-noble`: ``` kubectl get osimages.management.loft.sh ubuntu-noble -o yaml ``` ### Retrieve a list of OSImages[​](#retrieve-a-list-of-osimages-1 "Direct link to Retrieve a list of OSImages") Run the following curl command to list all OSImages: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/osimages" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single OSImage by name[​](#get-a-single-osimage-by-name "Direct link to Get a single OSImage by name") Run the following curl command to get OSImage `ubuntu-noble`: ``` # Exchange ubuntu-noble in the url below with the name of the OSImage curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/osimages/ubuntu-noble" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: OSImage[​](#create-osimage "Direct link to Create: OSImage") You can either use curl or kubectl to create a new OSImage. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: OSImage metadata: name: ubuntu-noble spec: description: Ubuntu Noble (24.04) server cloud image for bare metal provisioning displayName: Ubuntu Noble (24.04 LTS) properties: metal3.vcluster.com/image-checksum: https://cloud-images.ubuntu.com/noble/current/SHA256SUMS metal3.vcluster.com/image-checksum-type: sha256 metal3.vcluster.com/image-url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img status: {} ``` Then create the OSImage `ubuntu-noble` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: OSImage metadata: name: ubuntu-noble spec: description: Ubuntu Noble (24.04) server cloud image for bare metal provisioning displayName: Ubuntu Noble (24.04 LTS) properties: metal3.vcluster.com/image-checksum: https://cloud-images.ubuntu.com/noble/current/SHA256SUMS metal3.vcluster.com/image-checksum-type: sha256 metal3.vcluster.com/image-url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img status: {} ``` Run the following curl command to create a new OSImage `ubuntu-noble`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/osimages" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: OSImage[​](#update-osimage "Direct link to Update: OSImage") You can either use curl or kubectl to update OSImages. * kubectl * curl ### Update OSImage[​](#update-osimage "Direct link to Update OSImage") Run the following command to update OSImage `ubuntu-noble`: ``` kubectl edit osimages.management.loft.sh ubuntu-noble ``` Then edit the object and upon save, kubectl will update the resource. ### Patch OSImage[​](#patch-osimage "Direct link to Patch OSImage") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the OSImage `ubuntu-noble` via a patch: ``` kubectl patch osimages.management.loft.sh ubuntu-noble \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update OSImage[​](#update-osimage-1 "Direct link to Update OSImage") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: OSImage metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: ubuntu-noble resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: description: Ubuntu Noble (24.04) server cloud image for bare metal provisioning displayName: Ubuntu Noble (24.04 LTS) properties: metal3.vcluster.com/image-checksum: https://cloud-images.ubuntu.com/noble/current/SHA256SUMS metal3.vcluster.com/image-checksum-type: sha256 metal3.vcluster.com/image-url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img status: {} ``` Run the following curl command to update a single OSImage `ubuntu-noble`: ``` # Replace the ubuntu-noble in the url below with the name of the OSImage you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/osimages/ubuntu-noble" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch OSImage[​](#patch-osimage-1 "Direct link to Patch OSImage") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the OSImage `ubuntu-noble` via a patch: ``` # Replace the ubuntu-noble in the url below with the name of the OSImage you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/osimages/ubuntu-noble" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: OSImage[​](#delete-osimage "Direct link to Delete: OSImage") You can either use curl or kubectl to delete OSImages. * kubectl * curl Run the following command to delete OSImage `ubuntu-noble`: ``` kubectl delete osimages.management.loft.sh ubuntu-noble ``` Run the following curl command to delete OSImage `ubuntu-noble`: ``` # Replace the ubuntu-noble in the url below with the name of the OSImage you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/osimages/ubuntu-noble" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Owned Access Key Access keys let you authenticate with Loft API endpoints and Loft CLI in non-interactive environments such as from within CI/CD pipelines. ## Owned Access Key example[​](#owned-access-key-example "Direct link to Owned Access Key example") An example Owned Access Key: ``` apiVersion: management.loft.sh/v1 kind: OwnedAccessKey metadata: name: my-access-key spec: displayName: My Access Key ttl: 1728000 type: User user: my-user status: {} ``` ## Owned Access Key reference[​](#owned-access-key-reference "Direct link to Owned Access Key reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")The display name shown in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes an app ### `user` string [​](#spec-user "Direct link to spec-user")The user this access key refers to ### `team` string [​](#spec-team "Direct link to spec-team")The team this access key refers to ### `subject` string [​](#spec-subject "Direct link to spec-subject")Subject is a generic subject that can be used instead of user or team ### `groups` string\[] [​](#spec-groups "Direct link to spec-groups")Groups specifies extra groups to apply when using this access key ### `key` string [​](#spec-key "Direct link to spec-key")The actual access key that will be used as a bearer token ### `disabled` boolean [​](#spec-disabled "Direct link to spec-disabled")If this field is true, the access key is still allowed to exist, however will not work to access the api ### `ttl` integer [​](#spec-ttl "Direct link to spec-ttl")The time to live for this access key ### `ttlAfterLastActivity` boolean [​](#spec-ttlAfterLastActivity "Direct link to spec-ttlAfterLastActivity")If this is specified, the time to live for this access key will start after the lastActivity instead of creation timestamp ### `scope` object [​](#spec-scope "Direct link to spec-scope")Scope defines the scope of the access key. #### `roles` object\[] [​](#spec-scope-roles "Direct link to spec-scope-roles")Roles is a set of managed permissions to apply to the access key. ##### `role` string [​](#spec-scope-roles-role "Direct link to spec-scope-roles-role")Role is the name of the role to apply to the access key scope. ##### `projects` string\[] [​](#spec-scope-roles-projects "Direct link to spec-scope-roles-projects")Projects specifies the projects the access key should have access to. ##### `virtualClusters` string\[] [​](#spec-scope-roles-virtualClusters "Direct link to spec-scope-roles-virtualClusters")VirtualClusters specifies the tenant clusters the access key is allowed to access. #### `projects` object\[] [​](#spec-scope-projects "Direct link to spec-scope-projects")Projects specifies the projects the access key should have access to. ##### `project` string [​](#spec-scope-projects-project "Direct link to spec-scope-projects-project")Project is the name of the project. You can specify \* to select all projects. #### `spaces` object\[] [​](#spec-scope-spaces "Direct link to spec-scope-spaces")Spaces specifies the spaces the access key is allowed to access. ##### `project` string [​](#spec-scope-spaces-project "Direct link to spec-scope-spaces-project")Project is the name of the project. ##### `space` string [​](#spec-scope-spaces-space "Direct link to spec-scope-spaces-space")Space is the name of the space. You can specify \* to select all spaces. #### `virtualClusters` object\[] [​](#spec-scope-virtualClusters "Direct link to spec-scope-virtualClusters")VirtualClusters specifies the tenant clusters the access key is allowed to access. ##### `project` string [​](#spec-scope-virtualClusters-project "Direct link to spec-scope-virtualClusters-project")Project is the name of the project. ##### `virtualCluster` string [​](#spec-scope-virtualClusters-virtualCluster "Direct link to spec-scope-virtualClusters-virtualCluster")VirtualCluster is the name of the tenant cluster to access. You can specify \* to select all tenant clusters. #### `clusters` object\[] [​](#spec-scope-clusters "Direct link to spec-scope-clusters")Clusters specifies the project cluster the access key is allowed to access. ##### `cluster` string [​](#spec-scope-clusters-cluster "Direct link to spec-scope-clusters-cluster")Cluster is the name of the cluster to access. You can specify \* to select all clusters. #### `rules` object\[] [​](#spec-scope-rules "Direct link to spec-scope-rules")DEPRECATED: Use Projects, Spaces and VirtualClusters instead Rules specifies the rules that should apply to the access key. ##### `verbs` string\[] [​](#spec-scope-rules-verbs "Direct link to spec-scope-rules-verbs")The verbs that match this rule. An empty list implies every verb. ##### `resources` object\[] [​](#spec-scope-rules-resources "Direct link to spec-scope-rules-resources")Resources that this rule matches. An empty list implies all kinds in all API groups. ##### `group` string [​](#spec-scope-rules-resources-group "Direct link to spec-scope-rules-resources-group")Group is the name of the API group that contains the resources. The empty string represents the core API group. ##### `resources` string\[] [​](#spec-scope-rules-resources-resources "Direct link to spec-scope-rules-resources-resources")Resources is a list of resources this rule applies to.For example: 'pods' matches pods. 'pods/log' matches the log subresource of pods. '*' matches all resources and their subresources. 'pods/*' matches all subresources of pods. '\*/scale' matches all scale subresources.If wildcard is present, the validation rule will ensure resources do not overlap with each other.An empty list implies all resources and subresources in this API groups apply. ##### `resourceNames` string\[] [​](#spec-scope-rules-resources-resourceNames "Direct link to spec-scope-rules-resources-resourceNames")ResourceNames is a list of resource instance names that the policy matches. Using this field requires Resources to be specified. An empty list implies that every instance of the resource is matched. ##### `namespaces` string\[] [​](#spec-scope-rules-namespaces "Direct link to spec-scope-rules-namespaces")Namespaces that this rule matches. The empty string "" matches non-namespaced resources. An empty list implies every namespace. ##### `nonResourceURLs` string\[] [​](#spec-scope-rules-nonResourceURLs "Direct link to spec-scope-rules-nonResourceURLs")NonResourceURLs is a set of URL paths that should be checked. *s are allowed, but only as the full, final step in the path. Examples: "/metrics" - Log requests for apiserver metrics "/healthz*" - Log all health checks ##### `requestTargets` string\[] [​](#spec-scope-rules-requestTargets "Direct link to spec-scope-rules-requestTargets")RequestTargets is a list of request targets that are allowed. An empty list implies every request. ##### `cluster` string [​](#spec-scope-rules-cluster "Direct link to spec-scope-rules-cluster")Cluster that this rule matches. Only applies to cluster requests. If this is set, no requests for non cluster requests are allowed. An empty cluster means no restrictions will apply. ##### `virtualClusters` object\[] [​](#spec-scope-rules-virtualClusters "Direct link to spec-scope-rules-virtualClusters")VirtualClusters that this rule matches. Only applies to tenant cluster requests. An empty list means no restrictions will apply. ##### `name` string [​](#spec-scope-rules-virtualClusters-name "Direct link to spec-scope-rules-virtualClusters-name")Name of the tenant cluster. Empty means all tenant clusters. ##### `namespace` string [​](#spec-scope-rules-virtualClusters-namespace "Direct link to spec-scope-rules-virtualClusters-namespace")Namespace of the tenant cluster. Empty means all namespaces. #### `allowLoftCli` boolean [​](#spec-scope-allowLoftCli "Direct link to spec-scope-allowLoftCli")AllowLoftCLI allows certain read-only management requests to make sure loft cli works correctly with this specific access key.Deprecated: Use the `roles` field instead``` # Example: roles: - role: loftCLI ``` ### `type` string [​](#spec-type "Direct link to spec-type")The type of an access key, which basically describes if the access key is user managed or managed by loft itself. ### `identity` object [​](#spec-identity "Direct link to spec-identity")If available, contains information about the sso login data for this access key #### `userId` string [​](#spec-identity-userId "Direct link to spec-identity-userId")The subject of the user #### `username` string [​](#spec-identity-username "Direct link to spec-identity-username")The username #### `preferredUsername` string [​](#spec-identity-preferredUsername "Direct link to spec-identity-preferredUsername")The preferred username / display name #### `email` string [​](#spec-identity-email "Direct link to spec-identity-email")The user email #### `emailVerified` boolean [​](#spec-identity-emailVerified "Direct link to spec-identity-emailVerified")If the user email was verified #### `extraClaims` object [​](#spec-identity-extraClaims "Direct link to spec-identity-extraClaims")ExtraClaims are claims that are not otherwise contained in this struct but may be provided by the OIDC provider. Only extra claims that are allowed by the auth config are included. #### `groups` string\[] [​](#spec-identity-groups "Direct link to spec-identity-groups")The groups from the identity provider #### `connector` string [​](#spec-identity-connector "Direct link to spec-identity-connector")Connector is the name of the connector this access key was created from #### `connectorData` string [​](#spec-identity-connectorData "Direct link to spec-identity-connectorData")ConnectorData holds data used by the connector for subsequent requests after initial authentication, such as access tokens for upstream providers.This data is never shared with end users, OAuth clients, or through the API. ### `identityRefresh` object [​](#spec-identityRefresh "Direct link to spec-identityRefresh")The last time the identity was refreshed ### `oidcProvider` object [​](#spec-oidcProvider "Direct link to spec-oidcProvider")If the token is a refresh token, contains information about it #### `clientId` string [​](#spec-oidcProvider-clientId "Direct link to spec-oidcProvider-clientId")ClientId the token was generated for #### `nonce` string [​](#spec-oidcProvider-nonce "Direct link to spec-oidcProvider-nonce")Nonce to use #### `redirectUri` string [​](#spec-oidcProvider-redirectUri "Direct link to spec-oidcProvider-redirectUri")RedirectUri to use #### `scopes` string [​](#spec-oidcProvider-scopes "Direct link to spec-oidcProvider-scopes")Scopes to use ### `parent` string [​](#spec-parent "Direct link to spec-parent")DEPRECATED: do not use anymore Parent is used to share OIDC and external token information with multiple access keys. Since copying an OIDC refresh token would result in the other access keys becoming invalid after a refresh parent allows access keys to share that information.The use case for this is primarily user generated access keys, which will have the users current access key as parent if it contains an OIDC token. ### `oidcLogin` object [​](#spec-oidcLogin "Direct link to spec-oidcLogin")DEPRECATED: Use identity instead If available, contains information about the oidc login data for this access key #### `idToken` string [​](#spec-oidcLogin-idToken "Direct link to spec-oidcLogin-idToken")The current id token that was created during login #### `accessToken` string [​](#spec-oidcLogin-accessToken "Direct link to spec-oidcLogin-accessToken")The current access token that was created during login #### `refreshToken` string [​](#spec-oidcLogin-refreshToken "Direct link to spec-oidcLogin-refreshToken")The current refresh token that was created during login #### `lastRefresh` object [​](#spec-oidcLogin-lastRefresh "Direct link to spec-oidcLogin-lastRefresh")The last time the id token was refreshed ## `status` object [​](#status "Direct link to status") ### `lastActivity` object [​](#status-lastActivity "Direct link to status-lastActivity")The last time this access key was used to access the api ## Create: Owned Access Key[​](#create-owned-access-key "Direct link to Create: Owned Access Key") You can either use curl or kubectl to create a new Owned Access Key. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: OwnedAccessKey metadata: name: my-access-key spec: displayName: My Access Key ttl: 1728000 type: User user: my-user status: {} ``` Then create the Owned Access Key `my-access-key` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: OwnedAccessKey metadata: name: my-access-key spec: displayName: My Access Key ttl: 1728000 type: User user: my-user status: {} ``` Run the following curl command to create a new Owned Access Key `my-access-key`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/ownedaccesskeys" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: Owned Access Key[​](#update-owned-access-key "Direct link to Update: Owned Access Key") You can either use curl or kubectl to update Owned Access Keys. * kubectl * curl ### Update Owned Access Key[​](#update-owned-access-key "Direct link to Update Owned Access Key") Run the following command to update Owned Access Key `my-access-key`: ``` kubectl edit ownedaccesskeys.management.loft.sh my-access-key ``` Then edit the object and upon save, kubectl will update the resource. ### Patch Owned Access Key[​](#patch-owned-access-key "Direct link to Patch Owned Access Key") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the Owned Access Key `my-access-key` via a patch: ``` kubectl patch ownedaccesskeys.management.loft.sh my-access-key \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update Owned Access Key[​](#update-owned-access-key-1 "Direct link to Update Owned Access Key") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: OwnedAccessKey metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-access-key resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: displayName: My Access Key ttl: 1728000 type: User user: my-user status: {} ``` Run the following curl command to update a single Owned Access Key `my-access-key`: ``` # Replace the my-access-key in the url below with the name of the Owned Access Key you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/ownedaccesskeys/my-access-key" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch Owned Access Key[​](#patch-owned-access-key-1 "Direct link to Patch Owned Access Key") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the Owned Access Key `my-access-key` via a patch: ``` # Replace the my-access-key in the url below with the name of the Owned Access Key you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/ownedaccesskeys/my-access-key" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: Owned Access Key[​](#delete-owned-access-key "Direct link to Delete: Owned Access Key") You can either use curl or kubectl to delete Owned Access Keys. * kubectl * curl Run the following command to delete Owned Access Key `my-access-key`: ``` kubectl delete ownedaccesskeys.management.loft.sh my-access-key ``` Run the following curl command to delete Owned Access Key `my-access-key`: ``` # Replace the my-access-key in the url below with the name of the Owned Access Key you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/ownedaccesskeys/my-access-key" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Project Projects are used to group virtual clusters and spaces together. ## Project example[​](#project-example "Direct link to Project example") An example Project: ``` apiVersion: management.loft.sh/v1 kind: Project metadata: name: my-project spec: allowedClusters: - name: my-allowed-cluster allowedNodeProfiles: null allowedNodeTypes: null allowedTemplates: - kind: VirtualClusterTemplate name: '*' - kind: SpaceTemplate name: '*' members: - clusterRole: project-admin group: storage.loft.sh kind: User name: admin - clusterRole: project-user group: storage.loft.sh kind: Team name: my-team quotas: {} requirePreset: {} requireTemplate: {} vault: {} status: {} ``` ## Project reference[​](#project-reference "Direct link to Project reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes an app ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `quotas` object [​](#spec-quotas "Direct link to spec-quotas")Quotas define the quotas inside the project #### `project` object [​](#spec-quotas-project "Direct link to spec-quotas-project")Project holds the quotas for the whole project #### `user` object [​](#spec-quotas-user "Direct link to spec-quotas-user")User holds the quotas per user / team ### `allowedClusters` object\[] [​](#spec-allowedClusters "Direct link to spec-allowedClusters")AllowedClusters are target clusters that are allowed to target with environments. #### `name` string [​](#spec-allowedClusters-name "Direct link to spec-allowedClusters-name")Name is the name of the cluster that is allowed to create an environment in. ### `allowedRunners` object\[] [​](#spec-allowedRunners "Direct link to spec-allowedRunners")AllowedRunners are target runners that are allowed to target with #### `name` string [​](#spec-allowedRunners-name "Direct link to spec-allowedRunners-name")Name is the name of the runner that is allowed to create an environment in. ### `allowedTemplates` object\[] [​](#spec-allowedTemplates "Direct link to spec-allowedTemplates")AllowedTemplates are the templates that are allowed to use in this project. #### `kind` string [​](#spec-allowedTemplates-kind "Direct link to spec-allowedTemplates-kind")Kind of the template that is allowed. Supports DevPodWorkspaceTemplate, VirtualClusterTemplate, SpaceTemplate and StackTemplate. #### `group` string [​](#spec-allowedTemplates-group "Direct link to spec-allowedTemplates-group")Group of the template that is allowed. Currently only supports storage.loft.sh #### `name` string [​](#spec-allowedTemplates-name "Direct link to spec-allowedTemplates-name")Name of the template #### `isDefault` boolean [​](#spec-allowedTemplates-isDefault "Direct link to spec-allowedTemplates-isDefault")IsDefault specifies if the template should be used as a default ### `allowedNodeTypes` array|null [​](#spec-allowedNodeTypes "Direct link to spec-allowedNodeTypes")AllowedNodeTypes restricts which NodeTypes can be referenced by NodeClaims in this project. An entry can be an exact name ("aws.large") or a provider wildcard ("aws.\*"). If unset (nil), all NodeTypes are allowed; an empty list disallows all NodeTypes. ### `allowedNodeProfiles` array|null [​](#spec-allowedNodeProfiles "Direct link to spec-allowedNodeProfiles")AllowedNodeProfiles restricts which NodeProfiles can be referenced by NodeClaims and joinscript requests in this project. An entry can be an exact name ("platform.gpu-training") or an owner wildcard ("platform.\*"). If unset (nil), all NodeProfiles are allowed; an empty list disallows all NodeProfiles. ### `defaultNodeProfile` string [​](#spec-defaultNodeProfile "Direct link to spec-defaultNodeProfile")DefaultNodeProfile is the default NodeProfile applied to nodes in this project when none is selected explicitly. It is overridden by the vCluster-level privateNodes.defaultProfile and by per-node/per-pool selection, and is validated against allowedNodeProfiles. ### `requireTemplate` object [​](#spec-requireTemplate "Direct link to spec-requireTemplate")RequireTemplate configures if a template is required for instance creation. #### `disabled` boolean [​](#spec-requireTemplate-disabled "Direct link to spec-requireTemplate-disabled")If true, all users within the project will be allowed to create a new instance without a template. By default, only admins are allowed to create a new instance without a template. ### `requirePreset` object [​](#spec-requirePreset "Direct link to spec-requirePreset")RequirePreset configures if a preset is required for instance creation. #### `disabled` boolean [​](#spec-requirePreset-disabled "Direct link to spec-requirePreset-disabled")If true, all users within the project will not be allowed to create a new instance without a preset. By default, all users are allowed to create a new instance without a preset. ### `members` object\[] [​](#spec-members "Direct link to spec-members")Members are the users and teams that are part of this project #### `kind` string [​](#spec-members-kind "Direct link to spec-members-kind")Kind is the kind of the member. Currently either User or Team #### `group` string [​](#spec-members-group "Direct link to spec-members-group")Group of the member. Currently only supports storage.loft.sh #### `name` string [​](#spec-members-name "Direct link to spec-members-name")Name of the member #### `clusterRole` required string [​](#spec-members-clusterRole "Direct link to spec-members-clusterRole")ClusterRole is the assigned role for the above member ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ### `namespaceTemplate` object [​](#spec-namespaceTemplate "Direct link to spec-namespaceTemplate")NamespaceTemplate defines metadata that should be applied to the project's namespace on creation. This is useful for environments where admission controllers (e.g., Kyverno) require specific labels or annotations on namespaces. #### `metadata` object [​](#spec-namespaceTemplate-metadata "Direct link to spec-namespaceTemplate-metadata") ##### `labels` object [​](#spec-namespaceTemplate-metadata-labels "Direct link to spec-namespaceTemplate-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-namespaceTemplate-metadata-annotations "Direct link to spec-namespaceTemplate-metadata-annotations")Annotations are annotations on the object ### `namespacePattern` object [​](#spec-namespacePattern "Direct link to spec-namespacePattern")NamespacePattern specifies template patterns to use for creating each space or tenant cluster's namespace #### `space` string [​](#spec-namespacePattern-space "Direct link to spec-namespacePattern-space")Space holds the namespace pattern to use for space instances #### `virtualCluster` string [​](#spec-namespacePattern-virtualCluster "Direct link to spec-namespacePattern-virtualCluster")VirtualCluster holds the namespace pattern to use for tenant cluster instances ### `argoCD` object [​](#spec-argoCD "Direct link to spec-argoCD")ArgoIntegration holds information about ArgoCD Integration #### `enabled` boolean [​](#spec-argoCD-enabled "Direct link to spec-argoCD-enabled")Enabled indicates if the ArgoCD Integration is enabled for the project -- this knob only enables the syncing of virtualclusters, but does not enable SSO integration or project creation (see subsequent spec sections!). #### `cluster` string [​](#spec-argoCD-cluster "Direct link to spec-argoCD-cluster")Cluster defines the name of the cluster that ArgoCD is deployed into -- if not provided this will default to 'loft-cluster'. #### `virtualClusterInstance` string [​](#spec-argoCD-virtualClusterInstance "Direct link to spec-argoCD-virtualClusterInstance")VirtualClusterInstance defines the name of *tenant cluster* (instance) that ArgoCD is deployed into. If provided, Cluster will be ignored and Loft will assume that ArgoCD is running in the specified tenant cluster. #### `namespace` string [​](#spec-argoCD-namespace "Direct link to spec-argoCD-namespace")Namespace defines the namespace in which ArgoCD is running in the cluster. #### `sso` object [​](#spec-argoCD-sso "Direct link to spec-argoCD-sso")SSO defines single-sign-on related values for the ArgoCD Integration. Enabling SSO will allow users to authenticate to ArgoCD via Loft. ##### `enabled` boolean [​](#spec-argoCD-sso-enabled "Direct link to spec-argoCD-sso-enabled")Enabled indicates if the ArgoCD SSO Integration is enabled for this project. Enabling this will cause Loft to configure SSO authentication via Loft in ArgoCD. If Projects are *not* enabled, all users associated with this Project will be assigned either the 'read-only' (default) role, *or* the roles set under the AssignedRoles field. ##### `host` string [​](#spec-argoCD-sso-host "Direct link to spec-argoCD-sso-host")Host defines the ArgoCD host address that will be used for OIDC authentication between loft and ArgoCD. If not specified OIDC integration will be skipped, but vclusters/spaces will still be synced to ArgoCD. ##### `assignedRoles` string\[] [​](#spec-argoCD-sso-assignedRoles "Direct link to spec-argoCD-sso-assignedRoles")AssignedRoles is a list of roles to assign for users who authenticate via Loft -- by default this will be the `read-only` role. If any roles are provided this will override the default setting. #### `project` object [​](#spec-argoCD-project "Direct link to spec-argoCD-project")Project defines project related values for the ArgoCD Integration. Enabling Project integration will cause Loft to generate and manage an ArgoCD appProject that corresponds to the Loft Project. ##### `enabled` boolean [​](#spec-argoCD-project-enabled "Direct link to spec-argoCD-project-enabled")Enabled indicates if the ArgoCD Project Integration is enabled for this project. Enabling this will cause Loft to create an appProject in ArgoCD that is associated with the Loft Project. When Project integration is enabled Loft will override the default assigned role set in the SSO integration spec. ##### `metadata` object [​](#spec-argoCD-project-metadata "Direct link to spec-argoCD-project-metadata")Metadata defines additional metadata to attach to the loft created project in ArgoCD. ##### `extraAnnotations` object [​](#spec-argoCD-project-metadata-extraAnnotations "Direct link to spec-argoCD-project-metadata-extraAnnotations")ExtraAnnotations are optional annotations that can be attached to the project in ArgoCD. ##### `extraLabels` object [​](#spec-argoCD-project-metadata-extraLabels "Direct link to spec-argoCD-project-metadata-extraLabels")ExtraLabels are optional labels that can be attached to the project in ArgoCD. ##### `description` string [​](#spec-argoCD-project-metadata-description "Direct link to spec-argoCD-project-metadata-description")Description to add to the ArgoCD project. ##### `sourceRepos` string\[] [​](#spec-argoCD-project-sourceRepos "Direct link to spec-argoCD-project-sourceRepos")SourceRepos is a list of source repositories to attach/allow on the project, if not specified will be "\*" indicating all source repositories. ##### `destinations` object\[] [​](#spec-argoCD-project-destinations "Direct link to spec-argoCD-project-destinations")Destinations contains list of destinations available for deployment ##### `server` string [​](#spec-argoCD-project-destinations-server "Direct link to spec-argoCD-project-destinations-server") ##### `namespace` string [​](#spec-argoCD-project-destinations-namespace "Direct link to spec-argoCD-project-destinations-namespace") ##### `name` string [​](#spec-argoCD-project-destinations-name "Direct link to spec-argoCD-project-destinations-name") ##### `roles` object\[] [​](#spec-argoCD-project-roles "Direct link to spec-argoCD-project-roles")Roles is a list of roles that should be attached to the ArgoCD project. If roles are provided no loft default roles will be set. If no roles are provided *and* SSO is enabled, loft will configure sane default values. ##### `name` string [​](#spec-argoCD-project-roles-name "Direct link to spec-argoCD-project-roles-name")Name of the ArgoCD role to attach to the project. ##### `description` string [​](#spec-argoCD-project-roles-description "Direct link to spec-argoCD-project-roles-description")Description to add to the ArgoCD project. ##### `rules` object\[] [​](#spec-argoCD-project-roles-rules "Direct link to spec-argoCD-project-roles-rules")Rules ist a list of policy rules to attach to the role. ##### `action` string [​](#spec-argoCD-project-roles-rules-action "Direct link to spec-argoCD-project-roles-rules-action")Action is one of "\*", "get", "create", "update", "delete", "sync", or "override". ##### `application` string [​](#spec-argoCD-project-roles-rules-application "Direct link to spec-argoCD-project-roles-rules-application")Application is the ArgoCD project/repository to apply the rule to. DEPRECATED: Wasn't used. Kind provides a more flexible way to specify the resource type. ##### `kind` string [​](#spec-argoCD-project-roles-rules-kind "Direct link to spec-argoCD-project-roles-rules-kind")Kind is the kind to apply the rule to ##### `permission` boolean [​](#spec-argoCD-project-roles-rules-permission "Direct link to spec-argoCD-project-roles-rules-permission")Allow applies the "allow" permission to the rule, if allow is not set, the permission will always be set to "deny". ##### `groups` string\[] [​](#spec-argoCD-project-roles-groups "Direct link to spec-argoCD-project-roles-groups")Groups is a list of OIDC group names to bind to the role. ##### `clusterResourceWhitelist` object\[] [​](#spec-argoCD-project-clusterResourceWhitelist "Direct link to spec-argoCD-project-clusterResourceWhitelist")ClusterResourceWhitelist contains a list of whitelisted cluster level resources If not specified or empty, deny all cluster-scope resources. ##### `group` required string [​](#spec-argoCD-project-clusterResourceWhitelist-group "Direct link to spec-argoCD-project-clusterResourceWhitelist-group") ##### `kind` required string [​](#spec-argoCD-project-clusterResourceWhitelist-kind "Direct link to spec-argoCD-project-clusterResourceWhitelist-kind") ##### `name` string [​](#spec-argoCD-project-clusterResourceWhitelist-name "Direct link to spec-argoCD-project-clusterResourceWhitelist-name") ##### `namespaceResourceWhitelist` object\[] [​](#spec-argoCD-project-namespaceResourceWhitelist "Direct link to spec-argoCD-project-namespaceResourceWhitelist")NamespaceResourceWhitelist contains a list of whitelisted namespace level resources If not specified or empty, allow all namespace-scope resources. ##### `group` required string [​](#spec-argoCD-project-namespaceResourceWhitelist-group "Direct link to spec-argoCD-project-namespaceResourceWhitelist-group") ##### `kind` required string [​](#spec-argoCD-project-namespaceResourceWhitelist-kind "Direct link to spec-argoCD-project-namespaceResourceWhitelist-kind") ##### `clusterResourceBlacklist` object\[] [​](#spec-argoCD-project-clusterResourceBlacklist "Direct link to spec-argoCD-project-clusterResourceBlacklist")ClusterResourceBlacklist contains a list of blacklisted cluster level resources ##### `group` required string [​](#spec-argoCD-project-clusterResourceBlacklist-group "Direct link to spec-argoCD-project-clusterResourceBlacklist-group") ##### `kind` required string [​](#spec-argoCD-project-clusterResourceBlacklist-kind "Direct link to spec-argoCD-project-clusterResourceBlacklist-kind") ##### `name` string [​](#spec-argoCD-project-clusterResourceBlacklist-name "Direct link to spec-argoCD-project-clusterResourceBlacklist-name") ##### `namespaceResourceBlacklist` object\[] [​](#spec-argoCD-project-namespaceResourceBlacklist "Direct link to spec-argoCD-project-namespaceResourceBlacklist")NamespaceResourceBlacklist contains a list of blacklisted namespace level resources ##### `group` required string [​](#spec-argoCD-project-namespaceResourceBlacklist-group "Direct link to spec-argoCD-project-namespaceResourceBlacklist-group") ##### `kind` required string [​](#spec-argoCD-project-namespaceResourceBlacklist-kind "Direct link to spec-argoCD-project-namespaceResourceBlacklist-kind") ##### `sourceNamespaces` string\[] [​](#spec-argoCD-project-sourceNamespaces "Direct link to spec-argoCD-project-sourceNamespaces")SourceNamespaces defines the namespaces application resources are allowed to be created in ### `vault` object [​](#spec-vault "Direct link to spec-vault")VaultIntegration holds information about Vault Integration #### `enabled` boolean [​](#spec-vault-enabled "Direct link to spec-vault-enabled")Enabled indicates if the Vault Integration is enabled for the project -- this knob only enables the syncing of secrets to or from Vault, but does not setup Kubernetes authentication methods or Kubernetes secrets engines for vclusters. #### `address` string [​](#spec-vault-address "Direct link to spec-vault-address")Address defines the address of the Vault instance to use for this project. Will default to the `VAULT_ADDR` environment variable if not provided. #### `skipTLSVerify` boolean [​](#spec-vault-skipTLSVerify "Direct link to spec-vault-skipTLSVerify")SkipTLSVerify defines if TLS verification should be skipped when connecting to Vault. #### `namespace` string [​](#spec-vault-namespace "Direct link to spec-vault-namespace")Namespace defines the namespace to use when storing secrets in Vault. #### `auth` object [​](#spec-vault-auth "Direct link to spec-vault-auth")Auth defines the authentication method to use for this project. ##### `token` string [​](#spec-vault-auth-token "Direct link to spec-vault-auth-token")Token defines the token to use for authentication. ##### `tokenSecretRef` object [​](#spec-vault-auth-tokenSecretRef "Direct link to spec-vault-auth-tokenSecretRef")TokenSecretRef defines the Kubernetes secret to use for token authentication. Will be used if `token` is not provided.Secret data should contain the `token` key. ##### `name` string [​](#spec-vault-auth-tokenSecretRef-name "Direct link to spec-vault-auth-tokenSecretRef-name")Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: TODO: Drop `kubebuilder:default` when controller-gen doesn't need it . ##### `key` required string [​](#spec-vault-auth-tokenSecretRef-key "Direct link to spec-vault-auth-tokenSecretRef-key")The key of the secret to select from. Must be a valid secret key. ##### `optional` boolean [​](#spec-vault-auth-tokenSecretRef-optional "Direct link to spec-vault-auth-tokenSecretRef-optional")Specify whether the Secret or its key must be defined #### `syncInterval` string [​](#spec-vault-syncInterval "Direct link to spec-vault-syncInterval")SyncInterval defines the interval at which to sync secrets from Vault. Defaults to `1m.` See for supported formats. ## `status` object [​](#status "Direct link to status") ### `quotas` object [​](#status-quotas "Direct link to status-quotas")Quotas holds the quota status #### `project` object [​](#status-quotas-project "Direct link to status-quotas-project")Project is the quota status for the whole project ##### `limit` object [​](#status-quotas-project-limit "Direct link to status-quotas-project-limit")Limit is the amount limited, copied from spec.quotas.project ##### `used` object [​](#status-quotas-project-used "Direct link to status-quotas-project-used")Used is the amount currently used across all clusters ##### `clusters` {key: object} [​](#status-quotas-project-clusters "Direct link to status-quotas-project-clusters")Clusters holds the used amount per cluster. Maps cluster name to used resources ##### `used` object [​](#status-quotas-project-clusters-used "Direct link to status-quotas-project-clusters-used")Used is the amount currently used. Maps resource name, such as pods, to their used amount. #### `user` object [​](#status-quotas-user "Direct link to status-quotas-user")User is the quota status for each user / team. An example status could look like this: status: quotas: user: limit: pods: "10" spaces: "5" users: admin: used: spaces: "3" # (- calculated in our apiserver pods: "8" # (- the sum calculated from clusters clusters: cluster-1: # (- populated by agent from cluster-1 users: admin: pods: "3" cluster-2: users: admin: pods: "5" ##### `limit` object [​](#status-quotas-user-limit "Direct link to status-quotas-user-limit")Limit is the amount limited per user / team ##### `used` object [​](#status-quotas-user-used "Direct link to status-quotas-user-used")Used is the used amount per user / team ##### `users` object [​](#status-quotas-user-used-users "Direct link to status-quotas-user-used-users")Users is a mapping of users to used resources ##### `teams` object [​](#status-quotas-user-used-teams "Direct link to status-quotas-user-used-teams")Teams is a mapping of teams to used resources ##### `clusters` {key: object} [​](#status-quotas-user-clusters "Direct link to status-quotas-user-clusters")Clusters holds the used amount per cluster. Maps cluster name to used resources ##### `users` object [​](#status-quotas-user-clusters-users "Direct link to status-quotas-user-clusters-users")Users is a mapping of users to used resources ##### `teams` object [​](#status-quotas-user-clusters-teams "Direct link to status-quotas-user-clusters-teams")Teams is a mapping of teams to used resources ### `conditions` object [​](#status-conditions "Direct link to status-conditions")Conditions holds several conditions the project might be in ## Retrieve: Projects[​](#retrieve-projects "Direct link to Retrieve: Projects") You can either use curl or kubectl to retrieve Projects. * kubectl * curl ### Retrieve a list of Projects[​](#retrieve-a-list-of-projects "Direct link to Retrieve a list of Projects") Run the following command to list all Projects: ``` kubectl get projects.management.loft.sh -o yaml ``` ### Retrieve a single Project by name[​](#retrieve-a-single-project-by-name "Direct link to Retrieve a single Project by name") Run the following kubectl command to get Project `my-project`: ``` kubectl get projects.management.loft.sh my-project -o yaml ``` ### Retrieve a list of Projects[​](#retrieve-a-list-of-projects-1 "Direct link to Retrieve a list of Projects") Run the following curl command to list all Projects: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/projects" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single Project by name[​](#get-a-single-project-by-name "Direct link to Get a single Project by name") Run the following curl command to get Project `my-project`: ``` # Exchange my-project in the url below with the name of the Project curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/projects/my-project" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: Project[​](#create-project "Direct link to Create: Project") You can either use curl or kubectl to create a new Project. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: Project metadata: name: my-project spec: allowedClusters: - name: my-allowed-cluster allowedNodeProfiles: null allowedNodeTypes: null allowedTemplates: - kind: VirtualClusterTemplate name: '*' - kind: SpaceTemplate name: '*' members: - clusterRole: project-admin group: storage.loft.sh kind: User name: admin - clusterRole: project-user group: storage.loft.sh kind: Team name: my-team quotas: {} requirePreset: {} requireTemplate: {} vault: {} status: {} ``` Then create the Project `my-project` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: Project metadata: name: my-project spec: allowedClusters: - name: my-allowed-cluster allowedNodeProfiles: null allowedNodeTypes: null allowedTemplates: - kind: VirtualClusterTemplate name: '*' - kind: SpaceTemplate name: '*' members: - clusterRole: project-admin group: storage.loft.sh kind: User name: admin - clusterRole: project-user group: storage.loft.sh kind: Team name: my-team quotas: {} requirePreset: {} requireTemplate: {} vault: {} status: {} ``` Run the following curl command to create a new Project `my-project`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/projects" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: Project[​](#update-project "Direct link to Update: Project") You can either use curl or kubectl to update Projects. * kubectl * curl ### Update Project[​](#update-project "Direct link to Update Project") Run the following command to update Project `my-project`: ``` kubectl edit projects.management.loft.sh my-project ``` Then edit the object and upon save, kubectl will update the resource. ### Patch Project[​](#patch-project "Direct link to Patch Project") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the Project `my-project` via a patch: ``` kubectl patch projects.management.loft.sh my-project \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update Project[​](#update-project-1 "Direct link to Update Project") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: Project metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-project resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: allowedClusters: - name: my-allowed-cluster allowedNodeProfiles: null allowedNodeTypes: null allowedTemplates: - kind: VirtualClusterTemplate name: '*' - kind: SpaceTemplate name: '*' members: - clusterRole: project-admin group: storage.loft.sh kind: User name: admin - clusterRole: project-user group: storage.loft.sh kind: Team name: my-team quotas: {} requirePreset: {} requireTemplate: {} vault: {} status: {} ``` Run the following curl command to update a single Project `my-project`: ``` # Replace the my-project in the url below with the name of the Project you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/projects/my-project" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch Project[​](#patch-project-1 "Direct link to Patch Project") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the Project `my-project` via a patch: ``` # Replace the my-project in the url below with the name of the Project you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/projects/my-project" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: Project[​](#delete-project "Direct link to Delete: Project") You can either use curl or kubectl to delete Projects. * kubectl * curl Run the following command to delete Project `my-project`: ``` kubectl delete projects.management.loft.sh my-project ``` Run the following curl command to delete Project `my-project`: ``` # Replace the my-project in the url below with the name of the Project you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/projects/my-project" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Retrieve Project Clusters This API can be used to retrieve all allowed clusters of this project. ## Project Clusters example[​](#project-clusters-example "Direct link to Project Clusters example") An example Project Clusters: ``` apiVersion: management.loft.sh/v1 clusters: - apiVersion: management.loft.sh/v1 kind: Cluster metadata: name: my-cluster spec: config: {} status: {} kind: ProjectClusters metadata: {} ``` ## Project Clusters reference[​](#project-clusters-reference "Direct link to Project Clusters reference") ## `clusters` object\[] [​](#clusters "Direct link to clusters")Clusters holds all the allowed clusters ### `kind` string [​](#clusters-kind "Direct link to clusters-kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ### `apiVersion` string [​](#clusters-apiVersion "Direct link to clusters-apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ### `metadata` object [​](#clusters-metadata "Direct link to clusters-metadata") #### `name` string [​](#clusters-metadata-name "Direct link to clusters-metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: #### `generateName` string [​](#clusters-metadata-generateName "Direct link to clusters-metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: #### `namespace` string [​](#clusters-metadata-namespace "Direct link to clusters-metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: #### `selfLink` string [​](#clusters-metadata-selfLink "Direct link to clusters-metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. #### `uid` string [​](#clusters-metadata-uid "Direct link to clusters-metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: #### `resourceVersion` string [​](#clusters-metadata-resourceVersion "Direct link to clusters-metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: #### `generation` integer [​](#clusters-metadata-generation "Direct link to clusters-metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. #### `creationTimestamp` object [​](#clusters-metadata-creationTimestamp "Direct link to clusters-metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: #### `deletionTimestamp` object [​](#clusters-metadata-deletionTimestamp "Direct link to clusters-metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: #### `deletionGracePeriodSeconds` integer [​](#clusters-metadata-deletionGracePeriodSeconds "Direct link to clusters-metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. #### `labels` object [​](#clusters-metadata-labels "Direct link to clusters-metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: #### `annotations` object [​](#clusters-metadata-annotations "Direct link to clusters-metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: #### `ownerReferences` object\[] [​](#clusters-metadata-ownerReferences "Direct link to clusters-metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. ##### `apiVersion` required string [​](#clusters-metadata-ownerReferences-apiVersion "Direct link to clusters-metadata-ownerReferences-apiVersion")API version of the referent. ##### `kind` required string [​](#clusters-metadata-ownerReferences-kind "Direct link to clusters-metadata-ownerReferences-kind")Kind of the referent. More info: ##### `name` required string [​](#clusters-metadata-ownerReferences-name "Direct link to clusters-metadata-ownerReferences-name")Name of the referent. More info: ##### `uid` required string [​](#clusters-metadata-ownerReferences-uid "Direct link to clusters-metadata-ownerReferences-uid")UID of the referent. More info: ##### `controller` boolean [​](#clusters-metadata-ownerReferences-controller "Direct link to clusters-metadata-ownerReferences-controller")If true, this reference points to the managing controller. ##### `blockOwnerDeletion` boolean [​](#clusters-metadata-ownerReferences-blockOwnerDeletion "Direct link to clusters-metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. #### `finalizers` string\[] [​](#clusters-metadata-finalizers "Direct link to clusters-metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. #### `managedFields` object\[] [​](#clusters-metadata-managedFields "Direct link to clusters-metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. ##### `manager` string [​](#clusters-metadata-managedFields-manager "Direct link to clusters-metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. ##### `operation` string [​](#clusters-metadata-managedFields-operation "Direct link to clusters-metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. ##### `apiVersion` string [​](#clusters-metadata-managedFields-apiVersion "Direct link to clusters-metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. ##### `time` object [​](#clusters-metadata-managedFields-time "Direct link to clusters-metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. ##### `fieldsType` string [​](#clusters-metadata-managedFields-fieldsType "Direct link to clusters-metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" ##### `fieldsV1` object [​](#clusters-metadata-managedFields-fieldsV1 "Direct link to clusters-metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. ##### `subresource` string [​](#clusters-metadata-managedFields-subresource "Direct link to clusters-metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ### `spec` object [​](#clusters-spec "Direct link to clusters-spec") #### `displayName` string [​](#clusters-spec-displayName "Direct link to clusters-spec-displayName")If specified this name is displayed in the UI instead of the metadata name #### `description` string [​](#clusters-spec-description "Direct link to clusters-spec-description")Description describes a cluster access object #### `owner` object [​](#clusters-spec-owner "Direct link to clusters-spec-owner")Owner holds the owner of this object ##### `user` string [​](#clusters-spec-owner-user "Direct link to clusters-spec-owner-user")User specifies a Loft user. ##### `team` string [​](#clusters-spec-owner-team "Direct link to clusters-spec-owner-team")Team specifies a Loft team. #### `config` object [​](#clusters-spec-config "Direct link to clusters-spec-config")Holds a reference to a secret that holds the kube config to access this cluster ##### `secretName` string [​](#clusters-spec-config-secretName "Direct link to clusters-spec-config-secretName") ##### `secretNamespace` string [​](#clusters-spec-config-secretNamespace "Direct link to clusters-spec-config-secretNamespace") ##### `key` string [​](#clusters-spec-config-key "Direct link to clusters-spec-config-key") #### `local` boolean [​](#clusters-spec-local "Direct link to clusters-spec-local")Local specifies if it is the local cluster that should be connected, when this is specified, config is optional #### `networkPeer` boolean [​](#clusters-spec-networkPeer "Direct link to clusters-spec-networkPeer")NetworkPeer specifies if the cluster is connected via tailscale, when this is specified, config is optional #### `managementNamespace` string [​](#clusters-spec-managementNamespace "Direct link to clusters-spec-managementNamespace")The namespace where the cluster components will be installed in #### `unusable` boolean [​](#clusters-spec-unusable "Direct link to clusters-spec-unusable")If unusable is true, no spaces or tenant clusters can be scheduled on this cluster. #### `access` object\[] [​](#clusters-spec-access "Direct link to clusters-spec-access")Access holds the access rights for users and teams ##### `name` string [​](#clusters-spec-access-name "Direct link to clusters-spec-access-name")Name is an optional name that is used for this access rule ##### `verbs` required string\[] [​](#clusters-spec-access-verbs "Direct link to clusters-spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. ##### `subresources` string\[] [​](#clusters-spec-access-subresources "Direct link to clusters-spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule ##### `users` string\[] [​](#clusters-spec-access-users "Direct link to clusters-spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs ##### `teams` string\[] [​](#clusters-spec-access-teams "Direct link to clusters-spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs #### `metrics` object [​](#clusters-spec-metrics "Direct link to clusters-spec-metrics")Metrics holds the cluster's metrics backend configuration ##### `replicas` integer [​](#clusters-spec-metrics-replicas "Direct link to clusters-spec-metrics-replicas")Replicas is the number of desired replicas. ##### `resources` object [​](#clusters-spec-metrics-resources "Direct link to clusters-spec-metrics-resources")Resources are compute resource required by the metrics backend ##### `limits` object [​](#clusters-spec-metrics-resources-limits "Direct link to clusters-spec-metrics-resources-limits")Limits describes the maximum amount of compute resources allowed. More info: ##### `requests` object [​](#clusters-spec-metrics-resources-requests "Direct link to clusters-spec-metrics-resources-requests")Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: ##### `claims` object\[] [​](#clusters-spec-metrics-resources-claims "Direct link to clusters-spec-metrics-resources-claims")Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.This field depends on the DynamicResourceAllocation feature gate.This field is immutable. It can only be set for containers. ##### `name` required string [​](#clusters-spec-metrics-resources-claims-name "Direct link to clusters-spec-metrics-resources-claims-name")Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. ##### `request` string [​](#clusters-spec-metrics-resources-claims-request "Direct link to clusters-spec-metrics-resources-claims-request")Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. ##### `retention` string [​](#clusters-spec-metrics-retention "Direct link to clusters-spec-metrics-retention")Retention is the metrics data retention period. Default is 1y ##### `storage` object [​](#clusters-spec-metrics-storage "Direct link to clusters-spec-metrics-storage") ##### `storageClass` string [​](#clusters-spec-metrics-storage-storageClass "Direct link to clusters-spec-metrics-storage-storageClass")StorageClass the storage class to use when provisioning the metrics backend's persistent volume If set to "-" or "" dynamic provisioning is disabled If set to undefined or null (the default), the cluster's default storage class is used for provisioning ##### `size` string [​](#clusters-spec-metrics-storage-size "Direct link to clusters-spec-metrics-storage-size")Size the size of the metrics backend's persistent volume #### `opencost` object [​](#clusters-spec-opencost "Direct link to clusters-spec-opencost")OpenCost holds the cluster's OpenCost backend configuration ##### `replicas` integer [​](#clusters-spec-opencost-replicas "Direct link to clusters-spec-opencost-replicas")Replicas is the number of desired replicas. ##### `resources` object [​](#clusters-spec-opencost-resources "Direct link to clusters-spec-opencost-resources")Resources are compute resource required by the OpenCost backend ##### `limits` object [​](#clusters-spec-opencost-resources-limits "Direct link to clusters-spec-opencost-resources-limits")Limits describes the maximum amount of compute resources allowed. More info: ##### `requests` object [​](#clusters-spec-opencost-resources-requests "Direct link to clusters-spec-opencost-resources-requests")Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: ##### `claims` object\[] [​](#clusters-spec-opencost-resources-claims "Direct link to clusters-spec-opencost-resources-claims")Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.This field depends on the DynamicResourceAllocation feature gate.This field is immutable. It can only be set for containers. ##### `name` required string [​](#clusters-spec-opencost-resources-claims-name "Direct link to clusters-spec-opencost-resources-claims-name")Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. ##### `request` string [​](#clusters-spec-opencost-resources-claims-request "Direct link to clusters-spec-opencost-resources-claims-request")Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. #### `argoCD` object [​](#clusters-spec-argoCD "Direct link to clusters-spec-argoCD")ArgoCD holds the cluster's argo cd configuration ##### `enabled` boolean [​](#clusters-spec-argoCD-enabled "Direct link to clusters-spec-argoCD-enabled")Enabled defines if argo cd is enabled ##### `connector` string [​](#clusters-spec-argoCD-connector "Direct link to clusters-spec-argoCD-connector")Connector specifies the argo cd connector name ##### `cluster` object [​](#clusters-spec-argoCD-cluster "Direct link to clusters-spec-argoCD-cluster")Cluster holds settings for the cluster entry that is registered in argo cd ##### `metadata` object [​](#clusters-spec-argoCD-cluster-metadata "Direct link to clusters-spec-argoCD-cluster-metadata")Metadata is set on the argo cd cluster, so an ApplicationSet can select it. The platform's own loft.sh/cluster label wins over it. On Akuity, a key removed here stays on the cluster until it is removed in Akuity. ##### `labels` object [​](#clusters-spec-argoCD-cluster-metadata-labels "Direct link to clusters-spec-argoCD-cluster-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#clusters-spec-argoCD-cluster-metadata-annotations "Direct link to clusters-spec-argoCD-cluster-metadata-annotations")Annotations are annotations on the object ### `status` object [​](#clusters-status "Direct link to clusters-status") #### `phase` string [​](#clusters-status-phase "Direct link to clusters-status-phase") #### `reason` string [​](#clusters-status-reason "Direct link to clusters-status-reason") #### `message` string [​](#clusters-status-message "Direct link to clusters-status-message") #### `conditions` object [​](#clusters-status-conditions "Direct link to clusters-status-conditions")Conditions holds several conditions the cluster might be in #### `online` boolean [​](#clusters-status-online "Direct link to clusters-status-online")Online is whether the cluster is currently connected to the coordination server. ## Project Clusters (Get)[​](#project-clusters-get "Direct link to Project Clusters (Get)") You can retrieve all allowed clusters through this API. * kubectl * curl Run the following command: ``` # Exchange my-object in the url below with the name of the Project Clusters kubectl get --raw "/kubernetes/management/apis/management.loft.sh/v1/projects/my-object/clusters" ``` Run the following curl command: ``` # Exchange my-object in the url below with the name of the Project Clusters curl -s -X GET --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/projects/my-object/clusters" \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Import Namespace This API can be used to import an existing namespace from a connected cluster into the project. ## Import Namespace example[​](#import-namespace-example "Direct link to Import Namespace example") An example Import Namespace: ``` apiVersion: management.loft.sh/v1 kind: ProjectImportSpace metadata: {} sourceSpace: cluster: my-connected-cluster importName: my-name-in-project name: my-namespace ``` ## Import Namespace reference[​](#import-namespace-reference "Direct link to Import Namespace reference") ## `sourceSpace` required object [​](#sourceSpace "Direct link to sourceSpace")SourceSpace is the space to import into this project ### `name` string [​](#sourceSpace-name "Direct link to sourceSpace-name")Name of the space to import ### `cluster` string [​](#sourceSpace-cluster "Direct link to sourceSpace-cluster")Cluster name of the cluster the space is running on ### `importName` string [​](#sourceSpace-importName "Direct link to sourceSpace-importName")ImportName is an optional name to use as the spaceinstance name, if not provided the space name will be used ## Import Namespace (Create)[​](#import-namespace-create "Direct link to Import Namespace (Create)") Import a namespace through this API. * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: ProjectImportSpace metadata: {} sourceSpace: cluster: my-connected-cluster importName: my-name-in-project name: my-namespace ``` Then run the following curl command: ``` # Exchange my-object in the url below with the name of the Import Namespace curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/projects/my-object/importspace" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Retrieve Project Members This API can be used to retrieve all users and teams that are members of this project. Does not include loft wide admins. ## Project Members example[​](#project-members-example "Direct link to Project Members example") An example Project Members: ``` apiVersion: management.loft.sh/v1 kind: ProjectMembers metadata: {} teams: - info: displayName: My Team name: my-team users: - info: displayName: My User email: my-email name: my-user ``` ## Project Members reference[​](#project-members-reference "Direct link to Project Members reference") ## `teams` object\[] [​](#teams "Direct link to teams")Teams holds all the teams that have access to the cluster ### `info` object [​](#teams-info "Direct link to teams-info")Info about the user or team #### `name` string [​](#teams-info-name "Direct link to teams-info-name")Name is the kubernetes name of the object #### `displayName` string [​](#teams-info-displayName "Direct link to teams-info-displayName")The display name shown in the UI #### `icon` string [​](#teams-info-icon "Direct link to teams-info-icon")Icon is the icon of the user / team #### `username` string [​](#teams-info-username "Direct link to teams-info-username")The username that is used to login #### `email` string [​](#teams-info-email "Direct link to teams-info-email")The users email address #### `subject` string [​](#teams-info-subject "Direct link to teams-info-subject")The user subject ## `users` object\[] [​](#users "Direct link to users")Users holds all the users that have access to the cluster ### `info` object [​](#users-info "Direct link to users-info")Info about the user or team #### `name` string [​](#users-info-name "Direct link to users-info-name")Name is the kubernetes name of the object #### `displayName` string [​](#users-info-displayName "Direct link to users-info-displayName")The display name shown in the UI #### `icon` string [​](#users-info-icon "Direct link to users-info-icon")Icon is the icon of the user / team #### `username` string [​](#users-info-username "Direct link to users-info-username")The username that is used to login #### `email` string [​](#users-info-email "Direct link to users-info-email")The users email address #### `subject` string [​](#users-info-subject "Direct link to users-info-subject")The user subject ## Project Members (Get)[​](#project-members-get "Direct link to Project Members (Get)") You can retrieve all project users and teams through this API. * kubectl * curl Run the following command: ``` # Exchange my-object in the url below with the name of the Project Members kubectl get --raw "/kubernetes/management/apis/management.loft.sh/v1/projects/my-object/members" ``` Run the following curl command: ``` # Exchange my-object in the url below with the name of the Project Members curl -s -X GET --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/projects/my-object/members" \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Move Space To Other Project This API can be used to move a space from one project to another. ## Move Space To Other Project example[​](#move-space-to-other-project-example "Direct link to Move Space To Other Project example") An example Move Space To Other Project: ``` apiVersion: management.loft.sh/v1 kind: ProjectMigrateSpaceInstance metadata: {} sourceSpaceInstance: name: my-space namespace: loft-p-my-other-project ``` ## Move Space To Other Project reference[​](#move-space-to-other-project-reference "Direct link to Move Space To Other Project reference") ## `sourceSpaceInstance` required object [​](#sourceSpaceInstance "Direct link to sourceSpaceInstance")SourceSpaceInstance is the spaceinstance to migrate into this project ### `name` string [​](#sourceSpaceInstance-name "Direct link to sourceSpaceInstance-name")Name of the spaceinstance to migrate ### `namespace` string [​](#sourceSpaceInstance-namespace "Direct link to sourceSpaceInstance-namespace")Namespace of the spaceinstance to migrate ## Move Space To Other Project (Create)[​](#move-space-to-other-project-create "Direct link to Move Space To Other Project (Create)") Move a space into another project using this API. * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: ProjectMigrateSpaceInstance metadata: {} sourceSpaceInstance: name: my-space namespace: loft-p-my-other-project ``` Then run the following curl command: ``` # Exchange my-object in the url below with the name of the Move Space To Other Project curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/projects/my-object/migratespaceinstance" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Move vCluster To Other Project Deprecated Migrating virtual cluster instances between projects is deprecated and will be removed in a future release. This API can be used to move a virtual cluster from one project to another. ## Move Virtual Cluster To Other Project example[​](#move-virtual-cluster-to-other-project-example "Direct link to Move Virtual Cluster To Other Project example") An example Move Virtual Cluster To Other Project: ``` apiVersion: management.loft.sh/v1 kind: ProjectMigrateVirtualClusterInstance metadata: {} sourceVirtualClusterInstance: name: my-virtual-cluster namespace: loft-p-my-other-project ``` ## Move Virtual Cluster To Other Project reference[​](#move-virtual-cluster-to-other-project-reference "Direct link to Move Virtual Cluster To Other Project reference") ## `sourceVirtualClusterInstance` required object [​](#sourceVirtualClusterInstance "Direct link to sourceVirtualClusterInstance")SourceVirtualClusterInstance is the tenant cluster instance to migrate into this project ### `name` string [​](#sourceVirtualClusterInstance-name "Direct link to sourceVirtualClusterInstance-name")Name of the tenant cluster instance to migrate ### `namespace` string [​](#sourceVirtualClusterInstance-namespace "Direct link to sourceVirtualClusterInstance-namespace")Namespace of the tenant cluster instance to migrate ## Move Virtual Cluster To Other Project (Create)[​](#move-virtual-cluster-to-other-project-create "Direct link to Move Virtual Cluster To Other Project (Create)") Move a virtual cluster into another project using this API. * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: ProjectMigrateVirtualClusterInstance metadata: {} sourceVirtualClusterInstance: name: my-virtual-cluster namespace: loft-p-my-other-project ``` Then run the following curl command: ``` # Exchange my-object in the url below with the name of the Move Virtual Cluster To Other Project curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/projects/my-object/migratevirtualclusterinstance" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Project Secret ProjectSecret is a Kubernetes Secret managed by the platform inside a project, often synced from an external secret store such as HashiCorp Vault. ## ProjectSecret example[​](#projectsecret-example "Direct link to ProjectSecret example") An example ProjectSecret: ``` apiVersion: management.loft.sh/v1 kind: ProjectSecret metadata: name: my-secret namespace: loft-p-my-project spec: data: password: cGFzc3dvcmQ= displayName: My Secret status: {} ``` ## ProjectSecret reference[​](#projectsecret-reference "Direct link to ProjectSecret reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes a Project secret ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `data` object [​](#spec-data "Direct link to spec-data")Data contains the secret data. Each key must consist of alphanumeric characters, '-', '\_' or '.'. The serialized form of the secret data is a base64 encoded string, representing the arbitrary (possibly non-string) data value here. Described in ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ## `status` object [​](#status "Direct link to status") ### `conditions` object [​](#status-conditions "Direct link to status-conditions")Conditions holds several conditions the project might be in ## Retrieve: ProjectSecrets[​](#retrieve-projectsecrets "Direct link to Retrieve: ProjectSecrets") You can either use curl or kubectl to retrieve ProjectSecrets. * kubectl * curl ### Retrieve a list of ProjectSecrets[​](#retrieve-a-list-of-projectsecrets "Direct link to Retrieve a list of ProjectSecrets") Run the following command to list all ProjectSecrets in project `my-project`: ``` kubectl get projectsecrets.management.loft.sh -n p-my-project -o yaml ``` ### Retrieve a single ProjectSecret by name[​](#retrieve-a-single-projectsecret-by-name "Direct link to Retrieve a single ProjectSecret by name") Run the following kubectl command to get ProjectSecret `my-secret` in project `my-project`: ``` kubectl get projectsecrets.management.loft.sh my-secret -n p-my-project -o yaml ``` ### Retrieve a list of ProjectSecrets[​](#retrieve-a-list-of-projectsecrets-1 "Direct link to Retrieve a list of ProjectSecrets") Run the following curl command to list all ProjectSecrets in project `my-project`: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/projectsecrets" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single ProjectSecret by name[​](#get-a-single-projectsecret-by-name "Direct link to Get a single ProjectSecret by name") Run the following curl command to get ProjectSecret `my-secret` in project `my-project`: ``` # Exchange my-secret in the url below with the name of the ProjectSecret curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/projectsecrets/my-secret" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: ProjectSecret[​](#create-projectsecret "Direct link to Create: ProjectSecret") You can either use curl or kubectl to create a new ProjectSecret. Make sure to set the project in the `metadata.namespace` field you want to create the ProjectSecret in. If your project has the id `my-project`, the corresponding namespace would be `p-my-project`. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: ProjectSecret metadata: name: my-secret namespace: loft-p-my-project spec: data: password: cGFzc3dvcmQ= displayName: My Secret status: {} ``` Then create the ProjectSecret `my-secret` in project `my-project` with: ``` kubectl create -f object.yaml -n p-my-project ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: ProjectSecret metadata: name: my-secret namespace: loft-p-my-project spec: data: password: cGFzc3dvcmQ= displayName: My Secret status: {} ``` Run the following curl command to create a new ProjectSecret `my-secret` in project `my-project`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/projectsecrets" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: ProjectSecret[​](#update-projectsecret "Direct link to Update: ProjectSecret") You can either use curl or kubectl to update ProjectSecrets. * kubectl * curl ### Update ProjectSecret[​](#update-projectsecret "Direct link to Update ProjectSecret") Run the following command to update ProjectSecret `my-secret` in project `my-project`: ``` kubectl edit projectsecrets.management.loft.sh my-secret -n p-my-project ``` Then edit the object and upon save, kubectl will update the resource. ### Patch ProjectSecret[​](#patch-projectsecret "Direct link to Patch ProjectSecret") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the ProjectSecret `my-secret` in project `my-project` via a patch: ``` kubectl patch projectsecrets.management.loft.sh my-secret -n p-my-project \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update ProjectSecret[​](#update-projectsecret-1 "Direct link to Update ProjectSecret") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: ProjectSecret metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-secret namespace: loft-p-my-project resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: data: password: cGFzc3dvcmQ= displayName: My Secret status: {} ``` Run the following curl command to update a single ProjectSecret `my-secret` in project `my-project`: ``` # Replace the my-secret in the url below with the name of the ProjectSecret you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/projectsecrets/my-secret" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch ProjectSecret[​](#patch-projectsecret-1 "Direct link to Patch ProjectSecret") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the ProjectSecret `my-secret` in project `my-project` via a patch: ``` # Replace the my-secret in the url below with the name of the ProjectSecret you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/projectsecrets/my-secret" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: ProjectSecret[​](#delete-projectsecret "Direct link to Delete: ProjectSecret") You can either use curl or kubectl to delete ProjectSecrets. * kubectl * curl Run the following command to delete ProjectSecret `my-secret` in project `my-project`: ``` kubectl delete projectsecrets.management.loft.sh my-secret -n p-my-project ``` Run the following curl command to delete ProjectSecret `my-secret` in project `my-project`: ``` # Replace the my-secret in the url below with the name of the ProjectSecret you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/projectsecrets/my-secret" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Self Shows information about the current user. Alternatively you can also provide an alternative access key to retrieve information from. ## Self example[​](#self-example "Direct link to Self example") An example Self: ``` apiVersion: management.loft.sh/v1 kind: Self metadata: {} spec: {} status: accessKey: my-access-key accessKeyType: Login groups: - loft:admins - system:authenticated - loft:authenticated - loft:user:* - loft:user:admin subject: admin ``` ## Self reference[​](#self-reference "Direct link to Self reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `accessKey` string [​](#spec-accessKey "Direct link to spec-accessKey")AccessKey is an optional access key to use instead of the provided one ## `status` object [​](#status "Direct link to status") ### `user` object [​](#status-user "Direct link to status-user")The name of the currently logged in user #### `name` string [​](#status-user-name "Direct link to status-user-name")Name is the kubernetes name of the object #### `displayName` string [​](#status-user-displayName "Direct link to status-user-displayName")The display name shown in the UI #### `icon` string [​](#status-user-icon "Direct link to status-user-icon")Icon is the icon of the user / team #### `username` string [​](#status-user-username "Direct link to status-user-username")The username that is used to login #### `email` string [​](#status-user-email "Direct link to status-user-email")The users email address #### `subject` string [​](#status-user-subject "Direct link to status-user-subject")The user subject #### `teams` object\[] [​](#status-user-teams "Direct link to status-user-teams")Teams are the teams the user is part of ##### `name` string [​](#status-user-teams-name "Direct link to status-user-teams-name")Name is the kubernetes name of the object ##### `displayName` string [​](#status-user-teams-displayName "Direct link to status-user-teams-displayName")The display name shown in the UI ##### `icon` string [​](#status-user-teams-icon "Direct link to status-user-teams-icon")Icon is the icon of the user / team ##### `username` string [​](#status-user-teams-username "Direct link to status-user-teams-username")The username that is used to login ##### `email` string [​](#status-user-teams-email "Direct link to status-user-teams-email")The users email address ##### `subject` string [​](#status-user-teams-subject "Direct link to status-user-teams-subject")The user subject ### `team` object [​](#status-team "Direct link to status-team")The name of the currently logged in team #### `name` string [​](#status-team-name "Direct link to status-team-name")Name is the kubernetes name of the object #### `displayName` string [​](#status-team-displayName "Direct link to status-team-displayName")The display name shown in the UI #### `icon` string [​](#status-team-icon "Direct link to status-team-icon")Icon is the icon of the user / team #### `username` string [​](#status-team-username "Direct link to status-team-username")The username that is used to login #### `email` string [​](#status-team-email "Direct link to status-team-email")The users email address #### `subject` string [​](#status-team-subject "Direct link to status-team-subject")The user subject ### `accessKey` string [​](#status-accessKey "Direct link to status-accessKey")The name of the currently used access key ### `accessKeyScope` object [​](#status-accessKeyScope "Direct link to status-accessKeyScope")The scope of the currently used access key #### `roles` object\[] [​](#status-accessKeyScope-roles "Direct link to status-accessKeyScope-roles")Roles is a set of managed permissions to apply to the access key. ##### `role` string [​](#status-accessKeyScope-roles-role "Direct link to status-accessKeyScope-roles-role")Role is the name of the role to apply to the access key scope. ##### `projects` string\[] [​](#status-accessKeyScope-roles-projects "Direct link to status-accessKeyScope-roles-projects")Projects specifies the projects the access key should have access to. ##### `virtualClusters` string\[] [​](#status-accessKeyScope-roles-virtualClusters "Direct link to status-accessKeyScope-roles-virtualClusters")VirtualClusters specifies the tenant clusters the access key is allowed to access. #### `projects` object\[] [​](#status-accessKeyScope-projects "Direct link to status-accessKeyScope-projects")Projects specifies the projects the access key should have access to. ##### `project` string [​](#status-accessKeyScope-projects-project "Direct link to status-accessKeyScope-projects-project")Project is the name of the project. You can specify \* to select all projects. #### `spaces` object\[] [​](#status-accessKeyScope-spaces "Direct link to status-accessKeyScope-spaces")Spaces specifies the spaces the access key is allowed to access. ##### `project` string [​](#status-accessKeyScope-spaces-project "Direct link to status-accessKeyScope-spaces-project")Project is the name of the project. ##### `space` string [​](#status-accessKeyScope-spaces-space "Direct link to status-accessKeyScope-spaces-space")Space is the name of the space. You can specify \* to select all spaces. #### `virtualClusters` object\[] [​](#status-accessKeyScope-virtualClusters "Direct link to status-accessKeyScope-virtualClusters")VirtualClusters specifies the tenant clusters the access key is allowed to access. ##### `project` string [​](#status-accessKeyScope-virtualClusters-project "Direct link to status-accessKeyScope-virtualClusters-project")Project is the name of the project. ##### `virtualCluster` string [​](#status-accessKeyScope-virtualClusters-virtualCluster "Direct link to status-accessKeyScope-virtualClusters-virtualCluster")VirtualCluster is the name of the tenant cluster to access. You can specify \* to select all tenant clusters. #### `clusters` object\[] [​](#status-accessKeyScope-clusters "Direct link to status-accessKeyScope-clusters")Clusters specifies the project cluster the access key is allowed to access. ##### `cluster` string [​](#status-accessKeyScope-clusters-cluster "Direct link to status-accessKeyScope-clusters-cluster")Cluster is the name of the cluster to access. You can specify \* to select all clusters. #### `rules` object\[] [​](#status-accessKeyScope-rules "Direct link to status-accessKeyScope-rules")DEPRECATED: Use Projects, Spaces and VirtualClusters instead Rules specifies the rules that should apply to the access key. ##### `verbs` string\[] [​](#status-accessKeyScope-rules-verbs "Direct link to status-accessKeyScope-rules-verbs")The verbs that match this rule. An empty list implies every verb. ##### `resources` object\[] [​](#status-accessKeyScope-rules-resources "Direct link to status-accessKeyScope-rules-resources")Resources that this rule matches. An empty list implies all kinds in all API groups. ##### `group` string [​](#status-accessKeyScope-rules-resources-group "Direct link to status-accessKeyScope-rules-resources-group")Group is the name of the API group that contains the resources. The empty string represents the core API group. ##### `resources` string\[] [​](#status-accessKeyScope-rules-resources-resources "Direct link to status-accessKeyScope-rules-resources-resources")Resources is a list of resources this rule applies to.For example: 'pods' matches pods. 'pods/log' matches the log subresource of pods. '*' matches all resources and their subresources. 'pods/*' matches all subresources of pods. '\*/scale' matches all scale subresources.If wildcard is present, the validation rule will ensure resources do not overlap with each other.An empty list implies all resources and subresources in this API groups apply. ##### `resourceNames` string\[] [​](#status-accessKeyScope-rules-resources-resourceNames "Direct link to status-accessKeyScope-rules-resources-resourceNames")ResourceNames is a list of resource instance names that the policy matches. Using this field requires Resources to be specified. An empty list implies that every instance of the resource is matched. ##### `namespaces` string\[] [​](#status-accessKeyScope-rules-namespaces "Direct link to status-accessKeyScope-rules-namespaces")Namespaces that this rule matches. The empty string "" matches non-namespaced resources. An empty list implies every namespace. ##### `nonResourceURLs` string\[] [​](#status-accessKeyScope-rules-nonResourceURLs "Direct link to status-accessKeyScope-rules-nonResourceURLs")NonResourceURLs is a set of URL paths that should be checked. *s are allowed, but only as the full, final step in the path. Examples: "/metrics" - Log requests for apiserver metrics "/healthz*" - Log all health checks ##### `requestTargets` string\[] [​](#status-accessKeyScope-rules-requestTargets "Direct link to status-accessKeyScope-rules-requestTargets")RequestTargets is a list of request targets that are allowed. An empty list implies every request. ##### `cluster` string [​](#status-accessKeyScope-rules-cluster "Direct link to status-accessKeyScope-rules-cluster")Cluster that this rule matches. Only applies to cluster requests. If this is set, no requests for non cluster requests are allowed. An empty cluster means no restrictions will apply. ##### `virtualClusters` object\[] [​](#status-accessKeyScope-rules-virtualClusters "Direct link to status-accessKeyScope-rules-virtualClusters")VirtualClusters that this rule matches. Only applies to tenant cluster requests. An empty list means no restrictions will apply. ##### `name` string [​](#status-accessKeyScope-rules-virtualClusters-name "Direct link to status-accessKeyScope-rules-virtualClusters-name")Name of the tenant cluster. Empty means all tenant clusters. ##### `namespace` string [​](#status-accessKeyScope-rules-virtualClusters-namespace "Direct link to status-accessKeyScope-rules-virtualClusters-namespace")Namespace of the tenant cluster. Empty means all namespaces. #### `allowLoftCli` boolean [​](#status-accessKeyScope-allowLoftCli "Direct link to status-accessKeyScope-allowLoftCli")AllowLoftCLI allows certain read-only management requests to make sure loft cli works correctly with this specific access key.Deprecated: Use the `roles` field instead``` # Example: roles: - role: loftCLI ``` ### `accessKeyType` string [​](#status-accessKeyType "Direct link to status-accessKeyType")The type of the currently used access key ### `subject` string [​](#status-subject "Direct link to status-subject")The subject of the currently logged in user ### `uid` string [​](#status-uid "Direct link to status-uid")UID is the user uid ### `groups` string\[] [​](#status-groups "Direct link to status-groups")The groups of the currently logged in user ### `chatAuthToken` string [​](#status-chatAuthToken "Direct link to status-chatAuthToken")ChatAuthToken is the token used to authenticate with the in-product chat widget in the UI ### `instanceID` string [​](#status-instanceID "Direct link to status-instanceID")InstanceID is the loft instance id ### `loftHost` string [​](#status-loftHost "Direct link to status-loftHost")LoftHost is the host of the loft instance ### `projectNamespacePrefix` string [​](#status-projectNamespacePrefix "Direct link to status-projectNamespacePrefix")ProjectNamespacePrefix is the prefix used to name project namespaces after defaulting has been applied ## Create: Self[​](#create-self "Direct link to Create: Self") You can either use curl or kubectl to create a new Self. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: Self metadata: {} spec: {} status: accessKey: my-access-key accessKeyType: Login groups: - loft:admins - system:authenticated - loft:authenticated - loft:user:* - loft:user:admin subject: admin ``` Then create the Self `my-object` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: Self metadata: {} spec: {} status: accessKey: my-access-key accessKeyType: Login groups: - loft:admins - system:authenticated - loft:authenticated - loft:user:* - loft:user:admin subject: admin ``` Run the following curl command to create a new Self `my-object`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/selves" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Global Secret Global Secrets can be used to share sensitive information across users, teams and connected clusters. You can either access shared secrets through the Loft CLI or sync them directly to a project secret. ## Global Secret example[​](#global-secret-example "Direct link to Global Secret example") An example Global Secret: ``` apiVersion: management.loft.sh/v1 kind: SharedSecret metadata: name: my-global-secret namespace: loft spec: data: password: cGFzc3dvcmQ= description: Secret Data is base64 encoded. displayName: My Global Secret status: {} ``` ## Global Secret reference[​](#global-secret-reference "Direct link to Global Secret reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes a shared secret ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `data` object [​](#spec-data "Direct link to spec-data")Data contains the secret data. Each key must consist of alphanumeric characters, '-', '\_' or '.'. The serialized form of the secret data is a base64 encoded string, representing the arbitrary (possibly non-string) data value here. Described in ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams which will be transformed to Roles and RoleBindings #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ## `status` object [​](#status "Direct link to status") ### `conditions` object [​](#status-conditions "Direct link to status-conditions")Conditions holds several conditions the project might be in ## Retrieve: Global Secrets[​](#retrieve-global-secrets "Direct link to Retrieve: Global Secrets") You can either use curl or kubectl to retrieve Global Secrets. * kubectl * curl ### Retrieve a list of Global Secrets[​](#retrieve-a-list-of-global-secrets "Direct link to Retrieve a list of Global Secrets") Run the following command to list all Global Secretsin namespace `loft`: ``` kubectl get sharedsecrets.management.loft.sh -n loft -o yaml ``` ### Retrieve a single Global Secret by name[​](#retrieve-a-single-global-secret-by-name "Direct link to Retrieve a single Global Secret by name") Run the following kubectl command to get Global Secret `my-global-secret`in namespace `loft`: ``` kubectl get sharedsecrets.management.loft.sh my-global-secret -n loft -o yaml ``` ### Retrieve a list of Global Secrets[​](#retrieve-a-list-of-global-secrets-1 "Direct link to Retrieve a list of Global Secrets") Run the following curl command to list all Global Secretsin namespace `loft`: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/loft/sharedsecrets" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single Global Secret by name[​](#get-a-single-global-secret-by-name "Direct link to Get a single Global Secret by name") Run the following curl command to get Global Secret `my-global-secret`in namespace `loft`: ``` # Exchange my-global-secret in the url below with the name of the Global Secret curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/loft/sharedsecrets/my-global-secret" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: Global Secret[​](#create-global-secret "Direct link to Create: Global Secret") You can either use curl or kubectl to create a new Global Secret. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: SharedSecret metadata: name: my-global-secret namespace: loft spec: data: password: cGFzc3dvcmQ= description: Secret Data is base64 encoded. displayName: My Global Secret status: {} ``` Then create the Global Secret `my-global-secret`in namespace `loft` with: ``` kubectl create -f object.yaml -n loft ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: SharedSecret metadata: name: my-global-secret namespace: loft spec: data: password: cGFzc3dvcmQ= description: Secret Data is base64 encoded. displayName: My Global Secret status: {} ``` Run the following curl command to create a new Global Secret `my-global-secret`in namespace `loft`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/loft/sharedsecrets" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: Global Secret[​](#update-global-secret "Direct link to Update: Global Secret") You can either use curl or kubectl to update Global Secrets. * kubectl * curl ### Update Global Secret[​](#update-global-secret "Direct link to Update Global Secret") Run the following command to update Global Secret `my-global-secret`in namespace `loft`: ``` kubectl edit sharedsecrets.management.loft.sh my-global-secret -n loft ``` Then edit the object and upon save, kubectl will update the resource. ### Patch Global Secret[​](#patch-global-secret "Direct link to Patch Global Secret") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the Global Secret `my-global-secret`in namespace `loft` via a patch: ``` kubectl patch sharedsecrets.management.loft.sh my-global-secret -n loft \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update Global Secret[​](#update-global-secret-1 "Direct link to Update Global Secret") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: SharedSecret metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-global-secret namespace: loft resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: data: password: cGFzc3dvcmQ= description: Secret Data is base64 encoded. displayName: My Global Secret status: {} ``` Run the following curl command to update a single Global Secret `my-global-secret`in namespace `loft`: ``` # Replace the my-global-secret in the url below with the name of the Global Secret you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/loft/sharedsecrets/my-global-secret" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch Global Secret[​](#patch-global-secret-1 "Direct link to Patch Global Secret") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the Global Secret `my-global-secret`in namespace `loft` via a patch: ``` # Replace the my-global-secret in the url below with the name of the Global Secret you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/loft/sharedsecrets/my-global-secret" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: Global Secret[​](#delete-global-secret "Direct link to Delete: Global Secret") You can either use curl or kubectl to delete Global Secrets. * kubectl * curl Run the following command to delete Global Secret `my-global-secret`in namespace `loft`: ``` kubectl delete sharedsecrets.management.loft.sh my-global-secret -n loft ``` Run the following curl command to delete Global Secret `my-global-secret`in namespace `loft`: ``` # Replace the my-global-secret in the url below with the name of the Global Secret you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/loft/sharedsecrets/my-global-secret" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Space Instance Spaces are regular Kubernetes namespaces managed by the platform that are owned by a user or team. In Kubernetes, namespaces provide a mechanism for isolating groups of resources within a single cluster. ## Space example[​](#space-example "Direct link to Space example") An example Space: ``` apiVersion: management.loft.sh/v1 kind: SpaceInstance metadata: name: my-space namespace: loft-p-my-project spec: clusterRef: {} displayName: my-display-name owner: user: my-user parameters: 'my-parameter: my-value' templateRef: name: my-space-template status: {} ``` ## Space reference[​](#space-reference "Direct link to Space reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes a space instance ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `templateRef` object [​](#spec-templateRef "Direct link to spec-templateRef")TemplateRef holds the space template reference #### `name` string [​](#spec-templateRef-name "Direct link to spec-templateRef-name")Name holds the name of the template to reference. #### `version` string [​](#spec-templateRef-version "Direct link to spec-templateRef-version")Version holds the template version to use. Version is expected to be in semantic versioning format. Alternatively, you can also exchange major, minor or patch with an 'x' to tell Loft to automatically select the latest major, minor or patch version. #### `syncOnce` boolean [​](#spec-templateRef-syncOnce "Direct link to spec-templateRef-syncOnce")SyncOnce tells the controller to sync the instance once with the template. This is useful if you want to sync an instance after a template was changed. To automatically sync an instance with a template, use 'x.x.x' as version instead. ### `template` object [​](#spec-template "Direct link to spec-template")Template is the inline template to use for space creation. This is mutually exclusive with templateRef. #### `metadata` object [​](#spec-template-metadata "Direct link to spec-template-metadata") ##### `labels` object [​](#spec-template-metadata-labels "Direct link to spec-template-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-template-metadata-annotations "Direct link to spec-template-metadata-annotations")Annotations are annotations on the object #### `instanceTemplate` object [​](#spec-template-instanceTemplate "Direct link to spec-template-instanceTemplate")InstanceTemplate holds the space instance template ##### `metadata` object [​](#spec-template-instanceTemplate-metadata "Direct link to spec-template-instanceTemplate-metadata") ##### `labels` object [​](#spec-template-instanceTemplate-metadata-labels "Direct link to spec-template-instanceTemplate-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-template-instanceTemplate-metadata-annotations "Direct link to spec-template-instanceTemplate-metadata-annotations")Annotations are annotations on the object #### `objects` string [​](#spec-template-objects "Direct link to spec-template-objects")Objects are Kubernetes style YAMLs that should get deployed into the tenant cluster #### `charts` object\[] [​](#spec-template-charts "Direct link to spec-template-charts")Charts are helm charts that should get deployed ##### `name` string [​](#spec-template-charts-name "Direct link to spec-template-charts-name")Name is the chart name in the repository ##### `version` string [​](#spec-template-charts-version "Direct link to spec-template-charts-version")Version is the chart version in the repository ##### `repoURL` string [​](#spec-template-charts-repoURL "Direct link to spec-template-charts-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#spec-template-charts-username "Direct link to spec-template-charts-username")The username that is required for this repository ##### `usernameRef` object [​](#spec-template-charts-usernameRef "Direct link to spec-template-charts-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#spec-template-charts-usernameRef-projectSecretRef "Direct link to spec-template-charts-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-charts-usernameRef-projectSecretRef-project "Direct link to spec-template-charts-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-charts-usernameRef-projectSecretRef-name "Direct link to spec-template-charts-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-charts-usernameRef-projectSecretRef-key "Direct link to spec-template-charts-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#spec-template-charts-password "Direct link to spec-template-charts-password")The password that is required for this repository ##### `passwordRef` object [​](#spec-template-charts-passwordRef "Direct link to spec-template-charts-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#spec-template-charts-passwordRef-projectSecretRef "Direct link to spec-template-charts-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-charts-passwordRef-projectSecretRef-project "Direct link to spec-template-charts-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-charts-passwordRef-projectSecretRef-name "Direct link to spec-template-charts-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-charts-passwordRef-projectSecretRef-key "Direct link to spec-template-charts-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#spec-template-charts-insecureSkipTlsVerify "Direct link to spec-template-charts-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `releaseName` string [​](#spec-template-charts-releaseName "Direct link to spec-template-charts-releaseName")ReleaseName is the preferred release name of the app ##### `releaseNamespace` string [​](#spec-template-charts-releaseNamespace "Direct link to spec-template-charts-releaseNamespace")ReleaseNamespace is the preferred release namespace of the app ##### `values` string [​](#spec-template-charts-values "Direct link to spec-template-charts-values")Values are the values that should get passed to the chart ##### `wait` boolean [​](#spec-template-charts-wait "Direct link to spec-template-charts-wait")Wait determines if Loft should wait during deploy for the app to become ready ##### `timeout` string [​](#spec-template-charts-timeout "Direct link to spec-template-charts-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) #### `apps` object\[] [​](#spec-template-apps "Direct link to spec-template-apps")Apps specifies the apps that should get deployed by this template ##### `name` string [​](#spec-template-apps-name "Direct link to spec-template-apps-name")Name of the target app ##### `namespace` string [​](#spec-template-apps-namespace "Direct link to spec-template-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#spec-template-apps-releaseName "Direct link to spec-template-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#spec-template-apps-version "Direct link to spec-template-apps-version")Version of the app ##### `hash` string [​](#spec-template-apps-hash "Direct link to spec-template-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#spec-template-apps-parameters "Direct link to spec-template-apps-parameters")Parameters to use for the app #### `access` object [​](#spec-template-access "Direct link to spec-template-access")The space access ##### `defaultClusterRole` string [​](#spec-template-access-defaultClusterRole "Direct link to spec-template-access-defaultClusterRole")Specifies which cluster role should get applied to users or teams that do not match a rule below. ##### `rules` object\[] [​](#spec-template-access-rules "Direct link to spec-template-access-rules")Rules defines which users and teams should have which access to the tenant cluster. If no rule matches an authenticated incoming user, the user will get cluster admin access. ##### `users` string\[] [​](#spec-template-access-rules-users "Direct link to spec-template-access-rules-users")Users this rule matches. \* means all users. ##### `teams` string\[] [​](#spec-template-access-rules-teams "Direct link to spec-template-access-rules-teams")Teams that this rule matches. ##### `clusterRole` string [​](#spec-template-access-rules-clusterRole "Direct link to spec-template-access-rules-clusterRole")ClusterRole is the cluster role that should be assigned to the ### `clusterRef` object [​](#spec-clusterRef "Direct link to spec-clusterRef")ClusterRef is the reference to the connected cluster holding this space #### `cluster` string [​](#spec-clusterRef-cluster "Direct link to spec-clusterRef-cluster")Cluster is the connected cluster the space will be created in #### `namespace` string [​](#spec-clusterRef-namespace "Direct link to spec-clusterRef-namespace")Namespace is the namespace inside the connected cluster holding the space ### `parameters` string [​](#spec-parameters "Direct link to spec-parameters")Parameters are values to pass to the template. The values should be encoded as YAML string where each parameter is represented as a top-level field key. ### `extraAccessRules` object\[] [​](#spec-extraAccessRules "Direct link to spec-extraAccessRules")ExtraAccessRules defines extra rules which users and teams should have which access to the tenant cluster. #### `users` string\[] [​](#spec-extraAccessRules-users "Direct link to spec-extraAccessRules-users")Users this rule matches. \* means all users. #### `teams` string\[] [​](#spec-extraAccessRules-teams "Direct link to spec-extraAccessRules-teams")Teams that this rule matches. #### `clusterRole` string [​](#spec-extraAccessRules-clusterRole "Direct link to spec-extraAccessRules-clusterRole")ClusterRole is the cluster role that should be assigned to the ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ## `status` object [​](#status "Direct link to status") ### `phase` string [​](#status-phase "Direct link to status-phase")Phase describes the current phase the space instance is in ### `reason` string [​](#status-reason "Direct link to status-reason")Reason describes the reason in machine-readable form ### `message` string [​](#status-message "Direct link to status-message")Message describes the reason in human-readable form ### `conditions` object [​](#status-conditions "Direct link to status-conditions")Conditions holds several conditions the tenant cluster might be in ### `spaceObjects` object [​](#status-spaceObjects "Direct link to status-spaceObjects")SpaceObjects are the objects that were applied within the tenant cluster space #### `lastAppliedObjects` string [​](#status-spaceObjects-lastAppliedObjects "Direct link to status-spaceObjects-lastAppliedObjects")LastAppliedObjects holds the status for the objects that were applied #### `charts` object\[] [​](#status-spaceObjects-charts "Direct link to status-spaceObjects-charts")Charts are the charts that were applied ##### `name` string [​](#status-spaceObjects-charts-name "Direct link to status-spaceObjects-charts-name")Name of the chart that was applied ##### `namespace` string [​](#status-spaceObjects-charts-namespace "Direct link to status-spaceObjects-charts-namespace")Namespace of the chart that was applied ##### `lastAppliedChartConfigHash` string [​](#status-spaceObjects-charts-lastAppliedChartConfigHash "Direct link to status-spaceObjects-charts-lastAppliedChartConfigHash")LastAppliedChartConfigHash is the last applied configuration #### `apps` object\[] [​](#status-spaceObjects-apps "Direct link to status-spaceObjects-apps")Apps are the apps that were applied ##### `name` string [​](#status-spaceObjects-apps-name "Direct link to status-spaceObjects-apps-name")Name of the target app ##### `namespace` string [​](#status-spaceObjects-apps-namespace "Direct link to status-spaceObjects-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#status-spaceObjects-apps-releaseName "Direct link to status-spaceObjects-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#status-spaceObjects-apps-version "Direct link to status-spaceObjects-apps-version")Version of the app ##### `hash` string [​](#status-spaceObjects-apps-hash "Direct link to status-spaceObjects-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#status-spaceObjects-apps-parameters "Direct link to status-spaceObjects-apps-parameters")Parameters to use for the app ### `space` object [​](#status-space "Direct link to status-space")Space is the template rendered with all the parameters #### `metadata` object [​](#status-space-metadata "Direct link to status-space-metadata") ##### `labels` object [​](#status-space-metadata-labels "Direct link to status-space-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#status-space-metadata-annotations "Direct link to status-space-metadata-annotations")Annotations are annotations on the object #### `instanceTemplate` object [​](#status-space-instanceTemplate "Direct link to status-space-instanceTemplate")InstanceTemplate holds the space instance template ##### `metadata` object [​](#status-space-instanceTemplate-metadata "Direct link to status-space-instanceTemplate-metadata") ##### `labels` object [​](#status-space-instanceTemplate-metadata-labels "Direct link to status-space-instanceTemplate-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#status-space-instanceTemplate-metadata-annotations "Direct link to status-space-instanceTemplate-metadata-annotations")Annotations are annotations on the object #### `objects` string [​](#status-space-objects "Direct link to status-space-objects")Objects are Kubernetes style YAMLs that should get deployed into the tenant cluster #### `charts` object\[] [​](#status-space-charts "Direct link to status-space-charts")Charts are helm charts that should get deployed ##### `name` string [​](#status-space-charts-name "Direct link to status-space-charts-name")Name is the chart name in the repository ##### `version` string [​](#status-space-charts-version "Direct link to status-space-charts-version")Version is the chart version in the repository ##### `repoURL` string [​](#status-space-charts-repoURL "Direct link to status-space-charts-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#status-space-charts-username "Direct link to status-space-charts-username")The username that is required for this repository ##### `usernameRef` object [​](#status-space-charts-usernameRef "Direct link to status-space-charts-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#status-space-charts-usernameRef-projectSecretRef "Direct link to status-space-charts-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#status-space-charts-usernameRef-projectSecretRef-project "Direct link to status-space-charts-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#status-space-charts-usernameRef-projectSecretRef-name "Direct link to status-space-charts-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#status-space-charts-usernameRef-projectSecretRef-key "Direct link to status-space-charts-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#status-space-charts-password "Direct link to status-space-charts-password")The password that is required for this repository ##### `passwordRef` object [​](#status-space-charts-passwordRef "Direct link to status-space-charts-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#status-space-charts-passwordRef-projectSecretRef "Direct link to status-space-charts-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#status-space-charts-passwordRef-projectSecretRef-project "Direct link to status-space-charts-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#status-space-charts-passwordRef-projectSecretRef-name "Direct link to status-space-charts-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#status-space-charts-passwordRef-projectSecretRef-key "Direct link to status-space-charts-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#status-space-charts-insecureSkipTlsVerify "Direct link to status-space-charts-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `releaseName` string [​](#status-space-charts-releaseName "Direct link to status-space-charts-releaseName")ReleaseName is the preferred release name of the app ##### `releaseNamespace` string [​](#status-space-charts-releaseNamespace "Direct link to status-space-charts-releaseNamespace")ReleaseNamespace is the preferred release namespace of the app ##### `values` string [​](#status-space-charts-values "Direct link to status-space-charts-values")Values are the values that should get passed to the chart ##### `wait` boolean [​](#status-space-charts-wait "Direct link to status-space-charts-wait")Wait determines if Loft should wait during deploy for the app to become ready ##### `timeout` string [​](#status-space-charts-timeout "Direct link to status-space-charts-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) #### `apps` object\[] [​](#status-space-apps "Direct link to status-space-apps")Apps specifies the apps that should get deployed by this template ##### `name` string [​](#status-space-apps-name "Direct link to status-space-apps-name")Name of the target app ##### `namespace` string [​](#status-space-apps-namespace "Direct link to status-space-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#status-space-apps-releaseName "Direct link to status-space-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#status-space-apps-version "Direct link to status-space-apps-version")Version of the app ##### `hash` string [​](#status-space-apps-hash "Direct link to status-space-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#status-space-apps-parameters "Direct link to status-space-apps-parameters")Parameters to use for the app #### `access` object [​](#status-space-access "Direct link to status-space-access")The space access ##### `defaultClusterRole` string [​](#status-space-access-defaultClusterRole "Direct link to status-space-access-defaultClusterRole")Specifies which cluster role should get applied to users or teams that do not match a rule below. ##### `rules` object\[] [​](#status-space-access-rules "Direct link to status-space-access-rules")Rules defines which users and teams should have which access to the tenant cluster. If no rule matches an authenticated incoming user, the user will get cluster admin access. ##### `users` string\[] [​](#status-space-access-rules-users "Direct link to status-space-access-rules-users")Users this rule matches. \* means all users. ##### `teams` string\[] [​](#status-space-access-rules-teams "Direct link to status-space-access-rules-teams")Teams that this rule matches. ##### `clusterRole` string [​](#status-space-access-rules-clusterRole "Direct link to status-space-access-rules-clusterRole")ClusterRole is the cluster role that should be assigned to the ### `ignoreReconciliation` boolean [​](#status-ignoreReconciliation "Direct link to status-ignoreReconciliation")IgnoreReconciliation tells the controller to ignore reconciliation for this instance -- this is primarily used when migrating tenant cluster instances from project to project; this prevents a situation where there are two tenant cluster instances representing the same tenant cluster which could cause issues with concurrent reconciliations of the same object. Once the tenant cluster instance has been cloned and placed into the new project, this (the "old") tenant cluster instance can safely be deleted. ## Retrieve: Spaces[​](#retrieve-spaces "Direct link to Retrieve: Spaces") You can either use curl or kubectl to retrieve Spaces. * kubectl * curl ### Retrieve a list of Spaces[​](#retrieve-a-list-of-spaces "Direct link to Retrieve a list of Spaces") Run the following command to list all Spaces in project `my-project`: ``` kubectl get spaceinstances.management.loft.sh -n p-my-project -o yaml ``` ### Retrieve a single Space by name[​](#retrieve-a-single-space-by-name "Direct link to Retrieve a single Space by name") Run the following kubectl command to get Space `my-space` in project `my-project`: ``` kubectl get spaceinstances.management.loft.sh my-space -n p-my-project -o yaml ``` ### Retrieve a list of Spaces[​](#retrieve-a-list-of-spaces-1 "Direct link to Retrieve a list of Spaces") Run the following curl command to list all Spaces in project `my-project`: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/spaceinstances" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single Space by name[​](#get-a-single-space-by-name "Direct link to Get a single Space by name") Run the following curl command to get Space `my-space` in project `my-project`: ``` # Exchange my-space in the url below with the name of the Space curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/spaceinstances/my-space" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: Space[​](#create-space "Direct link to Create: Space") You can either use curl or kubectl to create a new Space. Make sure to set the project in the `metadata.namespace` field you want to create the Space in. If your project has the id `my-project`, the corresponding namespace would be `p-my-project`. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: SpaceInstance metadata: name: my-space namespace: loft-p-my-project spec: clusterRef: {} displayName: my-display-name owner: user: my-user parameters: 'my-parameter: my-value' templateRef: name: my-space-template status: {} ``` Then create the Space `my-space` in project `my-project` with: ``` kubectl create -f object.yaml -n p-my-project ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: SpaceInstance metadata: name: my-space namespace: loft-p-my-project spec: clusterRef: {} displayName: my-display-name owner: user: my-user parameters: 'my-parameter: my-value' templateRef: name: my-space-template status: {} ``` Run the following curl command to create a new Space `my-space` in project `my-project`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/spaceinstances" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: Space[​](#update-space "Direct link to Update: Space") You can either use curl or kubectl to update Spaces. * kubectl * curl ### Update Space[​](#update-space "Direct link to Update Space") Run the following command to update Space `my-space` in project `my-project`: ``` kubectl edit spaceinstances.management.loft.sh my-space -n p-my-project ``` Then edit the object and upon save, kubectl will update the resource. ### Patch Space[​](#patch-space "Direct link to Patch Space") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the Space `my-space` in project `my-project` via a patch: ``` kubectl patch spaceinstances.management.loft.sh my-space -n p-my-project \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update Space[​](#update-space-1 "Direct link to Update Space") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: SpaceInstance metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-space namespace: loft-p-my-project resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: clusterRef: {} displayName: my-display-name owner: user: my-user parameters: 'my-parameter: my-value' templateRef: name: my-space-template status: {} ``` Run the following curl command to update a single Space `my-space` in project `my-project`: ``` # Replace the my-space in the url below with the name of the Space you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/spaceinstances/my-space" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch Space[​](#patch-space-1 "Direct link to Patch Space") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the Space `my-space` in project `my-project` via a patch: ``` # Replace the my-space in the url below with the name of the Space you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/spaceinstances/my-space" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: Space[​](#delete-space "Direct link to Delete: Space") You can either use curl or kubectl to delete Spaces. * kubectl * curl Run the following command to delete Space `my-space` in project `my-project`: ``` kubectl delete spaceinstances.management.loft.sh my-space -n p-my-project ``` Run the following curl command to delete Space `my-space` in project `my-project`: ``` # Replace the my-space in the url below with the name of the Space you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/spaceinstances/my-space" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Space Template Space templates can be used to create predefined space configurations that have certain applications already deployed inside them or other predefined configuration applied. They are a powerful tool to create new predefined environments on demand. ## Space Template example[​](#space-template-example "Direct link to Space Template example") An example Space Template: ``` apiVersion: management.loft.sh/v1 kind: SpaceTemplate metadata: name: my-space-template spec: access: - users: - '*' verbs: - get description: This space templates deploys an isolated space displayName: Isolated Space Template template: instanceTemplate: metadata: {} metadata: {} objects: |- apiVersion: v1 kind: ResourceQuota metadata: name: loft-resource-quota spec: hard: count/configmaps: '100' count/endpoints: '40' count/persistentvolumeclaims: '20' count/pods: '20' count/secrets: '100' count/services: '20' limits.cpu: '20' limits.ephemeral-storage: 160Gi limits.memory: 40Gi requests.cpu: '10' requests.ephemeral-storage: 60Gi requests.memory: 20Gi requests.storage: 100Gi services.loadbalancers: '1' services.nodeports: '0' status: {} ``` ## Space Template reference[​](#space-template-reference "Direct link to Space Template reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that is shown in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes the space template ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `template` object [​](#spec-template "Direct link to spec-template")Template holds the space template #### `metadata` object [​](#spec-template-metadata "Direct link to spec-template-metadata") ##### `labels` object [​](#spec-template-metadata-labels "Direct link to spec-template-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-template-metadata-annotations "Direct link to spec-template-metadata-annotations")Annotations are annotations on the object #### `instanceTemplate` object [​](#spec-template-instanceTemplate "Direct link to spec-template-instanceTemplate")InstanceTemplate holds the space instance template ##### `metadata` object [​](#spec-template-instanceTemplate-metadata "Direct link to spec-template-instanceTemplate-metadata") ##### `labels` object [​](#spec-template-instanceTemplate-metadata-labels "Direct link to spec-template-instanceTemplate-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-template-instanceTemplate-metadata-annotations "Direct link to spec-template-instanceTemplate-metadata-annotations")Annotations are annotations on the object #### `objects` string [​](#spec-template-objects "Direct link to spec-template-objects")Objects are Kubernetes style YAMLs that should get deployed into the tenant cluster #### `charts` object\[] [​](#spec-template-charts "Direct link to spec-template-charts")Charts are helm charts that should get deployed ##### `name` string [​](#spec-template-charts-name "Direct link to spec-template-charts-name")Name is the chart name in the repository ##### `version` string [​](#spec-template-charts-version "Direct link to spec-template-charts-version")Version is the chart version in the repository ##### `repoURL` string [​](#spec-template-charts-repoURL "Direct link to spec-template-charts-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#spec-template-charts-username "Direct link to spec-template-charts-username")The username that is required for this repository ##### `usernameRef` object [​](#spec-template-charts-usernameRef "Direct link to spec-template-charts-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#spec-template-charts-usernameRef-projectSecretRef "Direct link to spec-template-charts-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-charts-usernameRef-projectSecretRef-project "Direct link to spec-template-charts-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-charts-usernameRef-projectSecretRef-name "Direct link to spec-template-charts-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-charts-usernameRef-projectSecretRef-key "Direct link to spec-template-charts-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#spec-template-charts-password "Direct link to spec-template-charts-password")The password that is required for this repository ##### `passwordRef` object [​](#spec-template-charts-passwordRef "Direct link to spec-template-charts-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#spec-template-charts-passwordRef-projectSecretRef "Direct link to spec-template-charts-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-charts-passwordRef-projectSecretRef-project "Direct link to spec-template-charts-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-charts-passwordRef-projectSecretRef-name "Direct link to spec-template-charts-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-charts-passwordRef-projectSecretRef-key "Direct link to spec-template-charts-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#spec-template-charts-insecureSkipTlsVerify "Direct link to spec-template-charts-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `releaseName` string [​](#spec-template-charts-releaseName "Direct link to spec-template-charts-releaseName")ReleaseName is the preferred release name of the app ##### `releaseNamespace` string [​](#spec-template-charts-releaseNamespace "Direct link to spec-template-charts-releaseNamespace")ReleaseNamespace is the preferred release namespace of the app ##### `values` string [​](#spec-template-charts-values "Direct link to spec-template-charts-values")Values are the values that should get passed to the chart ##### `wait` boolean [​](#spec-template-charts-wait "Direct link to spec-template-charts-wait")Wait determines if Loft should wait during deploy for the app to become ready ##### `timeout` string [​](#spec-template-charts-timeout "Direct link to spec-template-charts-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) #### `apps` object\[] [​](#spec-template-apps "Direct link to spec-template-apps")Apps specifies the apps that should get deployed by this template ##### `name` string [​](#spec-template-apps-name "Direct link to spec-template-apps-name")Name of the target app ##### `namespace` string [​](#spec-template-apps-namespace "Direct link to spec-template-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#spec-template-apps-releaseName "Direct link to spec-template-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#spec-template-apps-version "Direct link to spec-template-apps-version")Version of the app ##### `hash` string [​](#spec-template-apps-hash "Direct link to spec-template-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#spec-template-apps-parameters "Direct link to spec-template-apps-parameters")Parameters to use for the app #### `access` object [​](#spec-template-access "Direct link to spec-template-access")The space access ##### `defaultClusterRole` string [​](#spec-template-access-defaultClusterRole "Direct link to spec-template-access-defaultClusterRole")Specifies which cluster role should get applied to users or teams that do not match a rule below. ##### `rules` object\[] [​](#spec-template-access-rules "Direct link to spec-template-access-rules")Rules defines which users and teams should have which access to the tenant cluster. If no rule matches an authenticated incoming user, the user will get cluster admin access. ##### `users` string\[] [​](#spec-template-access-rules-users "Direct link to spec-template-access-rules-users")Users this rule matches. \* means all users. ##### `teams` string\[] [​](#spec-template-access-rules-teams "Direct link to spec-template-access-rules-teams")Teams that this rule matches. ##### `clusterRole` string [​](#spec-template-access-rules-clusterRole "Direct link to spec-template-access-rules-clusterRole")ClusterRole is the cluster role that should be assigned to the ### `parameters` object\[] [​](#spec-parameters "Direct link to spec-parameters")Parameters define additional app parameters that will set helm values #### `variable` string [​](#spec-parameters-variable "Direct link to spec-parameters-variable")Variable is the path of the variable. Can be foo or foo.bar for nested objects. #### `label` string [​](#spec-parameters-label "Direct link to spec-parameters-label")Label is the label to show for this parameter #### `description` string [​](#spec-parameters-description "Direct link to spec-parameters-description")Description is the description to show for this parameter #### `type` string [​](#spec-parameters-type "Direct link to spec-parameters-type")Type of the parameter. Can be one of: string, multiline, boolean, number and password #### `options` string\[] [​](#spec-parameters-options "Direct link to spec-parameters-options")Options is a slice of strings, where each string represents a mutually exclusive choice. #### `min` integer [​](#spec-parameters-min "Direct link to spec-parameters-min")Min is the minimum number if type is number #### `max` integer [​](#spec-parameters-max "Direct link to spec-parameters-max")Max is the maximum number if type is number #### `required` boolean [​](#spec-parameters-required "Direct link to spec-parameters-required")Required specifies if this parameter is required #### `defaultValue` string [​](#spec-parameters-defaultValue "Direct link to spec-parameters-defaultValue")DefaultValue is the default value if none is specified #### `placeholder` string [​](#spec-parameters-placeholder "Direct link to spec-parameters-placeholder")Placeholder shown in the UI #### `invalidation` string [​](#spec-parameters-invalidation "Direct link to spec-parameters-invalidation")Invalidation regex that if matched will reject the input #### `validation` string [​](#spec-parameters-validation "Direct link to spec-parameters-validation")Validation regex that if matched will allow the input #### `section` string [​](#spec-parameters-section "Direct link to spec-parameters-section")Section where this app should be displayed. Apps with the same section name will be grouped together ### `versions` object\[] [​](#spec-versions "Direct link to spec-versions")Versions are different space template versions that can be referenced as well #### `template` object [​](#spec-versions-template "Direct link to spec-versions-template")Template holds the space template ##### `metadata` object [​](#spec-versions-template-metadata "Direct link to spec-versions-template-metadata") ##### `labels` object [​](#spec-versions-template-metadata-labels "Direct link to spec-versions-template-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-versions-template-metadata-annotations "Direct link to spec-versions-template-metadata-annotations")Annotations are annotations on the object ##### `instanceTemplate` object [​](#spec-versions-template-instanceTemplate "Direct link to spec-versions-template-instanceTemplate")InstanceTemplate holds the space instance template ##### `metadata` object [​](#spec-versions-template-instanceTemplate-metadata "Direct link to spec-versions-template-instanceTemplate-metadata") ##### `labels` object [​](#spec-versions-template-instanceTemplate-metadata-labels "Direct link to spec-versions-template-instanceTemplate-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-versions-template-instanceTemplate-metadata-annotations "Direct link to spec-versions-template-instanceTemplate-metadata-annotations")Annotations are annotations on the object ##### `objects` string [​](#spec-versions-template-objects "Direct link to spec-versions-template-objects")Objects are Kubernetes style YAMLs that should get deployed into the tenant cluster ##### `charts` object\[] [​](#spec-versions-template-charts "Direct link to spec-versions-template-charts")Charts are helm charts that should get deployed ##### `name` string [​](#spec-versions-template-charts-name "Direct link to spec-versions-template-charts-name")Name is the chart name in the repository ##### `version` string [​](#spec-versions-template-charts-version "Direct link to spec-versions-template-charts-version")Version is the chart version in the repository ##### `repoURL` string [​](#spec-versions-template-charts-repoURL "Direct link to spec-versions-template-charts-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#spec-versions-template-charts-username "Direct link to spec-versions-template-charts-username")The username that is required for this repository ##### `usernameRef` object [​](#spec-versions-template-charts-usernameRef "Direct link to spec-versions-template-charts-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#spec-versions-template-charts-usernameRef-projectSecretRef "Direct link to spec-versions-template-charts-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-versions-template-charts-usernameRef-projectSecretRef-project "Direct link to spec-versions-template-charts-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-versions-template-charts-usernameRef-projectSecretRef-name "Direct link to spec-versions-template-charts-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-versions-template-charts-usernameRef-projectSecretRef-key "Direct link to spec-versions-template-charts-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#spec-versions-template-charts-password "Direct link to spec-versions-template-charts-password")The password that is required for this repository ##### `passwordRef` object [​](#spec-versions-template-charts-passwordRef "Direct link to spec-versions-template-charts-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#spec-versions-template-charts-passwordRef-projectSecretRef "Direct link to spec-versions-template-charts-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-versions-template-charts-passwordRef-projectSecretRef-project "Direct link to spec-versions-template-charts-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-versions-template-charts-passwordRef-projectSecretRef-name "Direct link to spec-versions-template-charts-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-versions-template-charts-passwordRef-projectSecretRef-key "Direct link to spec-versions-template-charts-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#spec-versions-template-charts-insecureSkipTlsVerify "Direct link to spec-versions-template-charts-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `releaseName` string [​](#spec-versions-template-charts-releaseName "Direct link to spec-versions-template-charts-releaseName")ReleaseName is the preferred release name of the app ##### `releaseNamespace` string [​](#spec-versions-template-charts-releaseNamespace "Direct link to spec-versions-template-charts-releaseNamespace")ReleaseNamespace is the preferred release namespace of the app ##### `values` string [​](#spec-versions-template-charts-values "Direct link to spec-versions-template-charts-values")Values are the values that should get passed to the chart ##### `wait` boolean [​](#spec-versions-template-charts-wait "Direct link to spec-versions-template-charts-wait")Wait determines if Loft should wait during deploy for the app to become ready ##### `timeout` string [​](#spec-versions-template-charts-timeout "Direct link to spec-versions-template-charts-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) ##### `apps` object\[] [​](#spec-versions-template-apps "Direct link to spec-versions-template-apps")Apps specifies the apps that should get deployed by this template ##### `name` string [​](#spec-versions-template-apps-name "Direct link to spec-versions-template-apps-name")Name of the target app ##### `namespace` string [​](#spec-versions-template-apps-namespace "Direct link to spec-versions-template-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#spec-versions-template-apps-releaseName "Direct link to spec-versions-template-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#spec-versions-template-apps-version "Direct link to spec-versions-template-apps-version")Version of the app ##### `hash` string [​](#spec-versions-template-apps-hash "Direct link to spec-versions-template-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#spec-versions-template-apps-parameters "Direct link to spec-versions-template-apps-parameters")Parameters to use for the app ##### `access` object [​](#spec-versions-template-access "Direct link to spec-versions-template-access")The space access ##### `defaultClusterRole` string [​](#spec-versions-template-access-defaultClusterRole "Direct link to spec-versions-template-access-defaultClusterRole")Specifies which cluster role should get applied to users or teams that do not match a rule below. ##### `rules` object\[] [​](#spec-versions-template-access-rules "Direct link to spec-versions-template-access-rules")Rules defines which users and teams should have which access to the tenant cluster. If no rule matches an authenticated incoming user, the user will get cluster admin access. ##### `users` string\[] [​](#spec-versions-template-access-rules-users "Direct link to spec-versions-template-access-rules-users")Users this rule matches. \* means all users. ##### `teams` string\[] [​](#spec-versions-template-access-rules-teams "Direct link to spec-versions-template-access-rules-teams")Teams that this rule matches. ##### `clusterRole` string [​](#spec-versions-template-access-rules-clusterRole "Direct link to spec-versions-template-access-rules-clusterRole")ClusterRole is the cluster role that should be assigned to the #### `parameters` object\[] [​](#spec-versions-parameters "Direct link to spec-versions-parameters")Parameters define additional app parameters that will set helm values ##### `variable` string [​](#spec-versions-parameters-variable "Direct link to spec-versions-parameters-variable")Variable is the path of the variable. Can be foo or foo.bar for nested objects. ##### `label` string [​](#spec-versions-parameters-label "Direct link to spec-versions-parameters-label")Label is the label to show for this parameter ##### `description` string [​](#spec-versions-parameters-description "Direct link to spec-versions-parameters-description")Description is the description to show for this parameter ##### `type` string [​](#spec-versions-parameters-type "Direct link to spec-versions-parameters-type")Type of the parameter. Can be one of: string, multiline, boolean, number and password ##### `options` string\[] [​](#spec-versions-parameters-options "Direct link to spec-versions-parameters-options")Options is a slice of strings, where each string represents a mutually exclusive choice. ##### `min` integer [​](#spec-versions-parameters-min "Direct link to spec-versions-parameters-min")Min is the minimum number if type is number ##### `max` integer [​](#spec-versions-parameters-max "Direct link to spec-versions-parameters-max")Max is the maximum number if type is number ##### `required` boolean [​](#spec-versions-parameters-required "Direct link to spec-versions-parameters-required")Required specifies if this parameter is required ##### `defaultValue` string [​](#spec-versions-parameters-defaultValue "Direct link to spec-versions-parameters-defaultValue")DefaultValue is the default value if none is specified ##### `placeholder` string [​](#spec-versions-parameters-placeholder "Direct link to spec-versions-parameters-placeholder")Placeholder shown in the UI ##### `invalidation` string [​](#spec-versions-parameters-invalidation "Direct link to spec-versions-parameters-invalidation")Invalidation regex that if matched will reject the input ##### `validation` string [​](#spec-versions-parameters-validation "Direct link to spec-versions-parameters-validation")Validation regex that if matched will allow the input ##### `section` string [​](#spec-versions-parameters-section "Direct link to spec-versions-parameters-section")Section where this app should be displayed. Apps with the same section name will be grouped together #### `version` string [​](#spec-versions-version "Direct link to spec-versions-version")Version is the version. Needs to be in X.X.X format. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ## `status` object [​](#status "Direct link to status") ## Retrieve: Space Templates[​](#retrieve-space-templates "Direct link to Retrieve: Space Templates") You can either use curl or kubectl to retrieve Space Templates. * kubectl * curl ### Retrieve a list of Space Templates[​](#retrieve-a-list-of-space-templates "Direct link to Retrieve a list of Space Templates") Run the following command to list all Space Templates: ``` kubectl get spacetemplates.management.loft.sh -o yaml ``` ### Retrieve a single Space Template by name[​](#retrieve-a-single-space-template-by-name "Direct link to Retrieve a single Space Template by name") Run the following kubectl command to get Space Template `my-space-template`: ``` kubectl get spacetemplates.management.loft.sh my-space-template -o yaml ``` ### Retrieve a list of Space Templates[​](#retrieve-a-list-of-space-templates-1 "Direct link to Retrieve a list of Space Templates") Run the following curl command to list all Space Templates: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/spacetemplates" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single Space Template by name[​](#get-a-single-space-template-by-name "Direct link to Get a single Space Template by name") Run the following curl command to get Space Template `my-space-template`: ``` # Exchange my-space-template in the url below with the name of the Space Template curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/spacetemplates/my-space-template" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: Space Template[​](#create-space-template "Direct link to Create: Space Template") You can either use curl or kubectl to create a new Space Template. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: SpaceTemplate metadata: name: my-space-template spec: access: - users: - '*' verbs: - get description: This space templates deploys an isolated space displayName: Isolated Space Template template: instanceTemplate: metadata: {} metadata: {} objects: |- apiVersion: v1 kind: ResourceQuota metadata: name: loft-resource-quota spec: hard: count/configmaps: '100' count/endpoints: '40' count/persistentvolumeclaims: '20' count/pods: '20' count/secrets: '100' count/services: '20' limits.cpu: '20' limits.ephemeral-storage: 160Gi limits.memory: 40Gi requests.cpu: '10' requests.ephemeral-storage: 60Gi requests.memory: 20Gi requests.storage: 100Gi services.loadbalancers: '1' services.nodeports: '0' status: {} ``` Then create the Space Template `my-space-template` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: SpaceTemplate metadata: name: my-space-template spec: access: - users: - '*' verbs: - get description: This space templates deploys an isolated space displayName: Isolated Space Template template: instanceTemplate: metadata: {} metadata: {} objects: |- apiVersion: v1 kind: ResourceQuota metadata: name: loft-resource-quota spec: hard: count/configmaps: '100' count/endpoints: '40' count/persistentvolumeclaims: '20' count/pods: '20' count/secrets: '100' count/services: '20' limits.cpu: '20' limits.ephemeral-storage: 160Gi limits.memory: 40Gi requests.cpu: '10' requests.ephemeral-storage: 60Gi requests.memory: 20Gi requests.storage: 100Gi services.loadbalancers: '1' services.nodeports: '0' status: {} ``` Run the following curl command to create a new Space Template `my-space-template`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/spacetemplates" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: Space Template[​](#update-space-template "Direct link to Update: Space Template") You can either use curl or kubectl to update Space Templates. * kubectl * curl ### Update Space Template[​](#update-space-template "Direct link to Update Space Template") Run the following command to update Space Template `my-space-template`: ``` kubectl edit spacetemplates.management.loft.sh my-space-template ``` Then edit the object and upon save, kubectl will update the resource. ### Patch Space Template[​](#patch-space-template "Direct link to Patch Space Template") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the Space Template `my-space-template` via a patch: ``` kubectl patch spacetemplates.management.loft.sh my-space-template \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update Space Template[​](#update-space-template-1 "Direct link to Update Space Template") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: SpaceTemplate metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-space-template resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: access: - users: - '*' verbs: - get description: This space templates deploys an isolated space displayName: Isolated Space Template template: instanceTemplate: metadata: {} metadata: {} objects: |- apiVersion: v1 kind: ResourceQuota metadata: name: loft-resource-quota spec: hard: count/configmaps: '100' count/endpoints: '40' count/persistentvolumeclaims: '20' count/pods: '20' count/secrets: '100' count/services: '20' limits.cpu: '20' limits.ephemeral-storage: 160Gi limits.memory: 40Gi requests.cpu: '10' requests.ephemeral-storage: 60Gi requests.memory: 20Gi requests.storage: 100Gi services.loadbalancers: '1' services.nodeports: '0' status: {} ``` Run the following curl command to update a single Space Template `my-space-template`: ``` # Replace the my-space-template in the url below with the name of the Space Template you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/spacetemplates/my-space-template" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch Space Template[​](#patch-space-template-1 "Direct link to Patch Space Template") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the Space Template `my-space-template` via a patch: ``` # Replace the my-space-template in the url below with the name of the Space Template you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/spacetemplates/my-space-template" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: Space Template[​](#delete-space-template "Direct link to Delete: Space Template") You can either use curl or kubectl to delete Space Templates. * kubectl * curl Run the following command to delete Space Template `my-space-template`: ``` kubectl delete spacetemplates.management.loft.sh my-space-template ``` Run the following curl command to delete Space Template `my-space-template`: ``` # Replace the my-space-template in the url below with the name of the Space Template you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/spacetemplates/my-space-template" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # SSH Key SSHKey holds a public SSH key that can be attached to auto-provisioned or manually joined nodes. ## SSHKey example[​](#sshkey-example "Direct link to SSHKey example") An example SSHKey: ``` apiVersion: management.loft.sh/v1 kind: SSHKey metadata: name: my-ssh-key spec: description: Development access key displayName: My SSH Key publicKey: ssh-ed25519 AAAA... user@host status: {} ``` ## SSHKey reference[​](#sshkey-reference "Direct link to SSHKey reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes an SSH key ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ### `publicKey` string [​](#spec-publicKey "Direct link to spec-publicKey")PublicKey is the public SSH key ## `status` object [​](#status "Direct link to status") ## Retrieve: SSHKeys[​](#retrieve-sshkeys "Direct link to Retrieve: SSHKeys") You can either use curl or kubectl to retrieve SSHKeys. * kubectl * curl ### Retrieve a list of SSHKeys[​](#retrieve-a-list-of-sshkeys "Direct link to Retrieve a list of SSHKeys") Run the following command to list all SSHKeys: ``` kubectl get sshkeys.management.loft.sh -o yaml ``` ### Retrieve a single SSHKey by name[​](#retrieve-a-single-sshkey-by-name "Direct link to Retrieve a single SSHKey by name") Run the following kubectl command to get SSHKey `my-ssh-key`: ``` kubectl get sshkeys.management.loft.sh my-ssh-key -o yaml ``` ### Retrieve a list of SSHKeys[​](#retrieve-a-list-of-sshkeys-1 "Direct link to Retrieve a list of SSHKeys") Run the following curl command to list all SSHKeys: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/sshkeys" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single SSHKey by name[​](#get-a-single-sshkey-by-name "Direct link to Get a single SSHKey by name") Run the following curl command to get SSHKey `my-ssh-key`: ``` # Exchange my-ssh-key in the url below with the name of the SSHKey curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/sshkeys/my-ssh-key" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: SSHKey[​](#create-sshkey "Direct link to Create: SSHKey") You can either use curl or kubectl to create a new SSHKey. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: SSHKey metadata: name: my-ssh-key spec: description: Development access key displayName: My SSH Key publicKey: ssh-ed25519 AAAA... user@host status: {} ``` Then create the SSHKey `my-ssh-key` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: SSHKey metadata: name: my-ssh-key spec: description: Development access key displayName: My SSH Key publicKey: ssh-ed25519 AAAA... user@host status: {} ``` Run the following curl command to create a new SSHKey `my-ssh-key`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/sshkeys" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: SSHKey[​](#update-sshkey "Direct link to Update: SSHKey") You can either use curl or kubectl to update SSHKeys. * kubectl * curl ### Update SSHKey[​](#update-sshkey "Direct link to Update SSHKey") Run the following command to update SSHKey `my-ssh-key`: ``` kubectl edit sshkeys.management.loft.sh my-ssh-key ``` Then edit the object and upon save, kubectl will update the resource. ### Patch SSHKey[​](#patch-sshkey "Direct link to Patch SSHKey") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the SSHKey `my-ssh-key` via a patch: ``` kubectl patch sshkeys.management.loft.sh my-ssh-key \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update SSHKey[​](#update-sshkey-1 "Direct link to Update SSHKey") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: SSHKey metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-ssh-key resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: description: Development access key displayName: My SSH Key publicKey: ssh-ed25519 AAAA... user@host status: {} ``` Run the following curl command to update a single SSHKey `my-ssh-key`: ``` # Replace the my-ssh-key in the url below with the name of the SSHKey you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/sshkeys/my-ssh-key" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch SSHKey[​](#patch-sshkey-1 "Direct link to Patch SSHKey") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the SSHKey `my-ssh-key` via a patch: ``` # Replace the my-ssh-key in the url below with the name of the SSHKey you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/sshkeys/my-ssh-key" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: SSHKey[​](#delete-sshkey "Direct link to Delete: SSHKey") You can either use curl or kubectl to delete SSHKeys. * kubectl * curl Run the following command to delete SSHKey `my-ssh-key`: ``` kubectl delete sshkeys.management.loft.sh my-ssh-key ``` Run the following curl command to delete SSHKey `my-ssh-key`: ``` # Replace the my-ssh-key in the url below with the name of the SSHKey you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/sshkeys/my-ssh-key" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Stack Instance StackInstance deploys one inline or referenced task graph to a tenant cluster or control plane cluster and reports aggregate and per-task status. See [What are Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md) for the concept and [Use a Stack](https://vcluster.com/docs/platform/use-platform/apps/use-stacks.md) for the installation and monitoring workflow. ## StackInstance example[​](#stackinstance-example "Direct link to StackInstance example") An example StackInstance: ``` apiVersion: management.loft.sh/v1 kind: StackInstance metadata: name: example-platform namespace: p-my-project spec: destination: virtualCluster: name: development displayName: Example application platform owner: user: admin prunePolicy: Retain templateRef: name: example-platform status: {} ``` ## StackInstance reference[​](#stackinstance-reference "Direct link to StackInstance reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes the stack instance ### `destination` object [​](#spec-destination "Direct link to spec-destination")Destination is where every materialized application is deployed: a tenant cluster or a control plane cluster. Immutable after creation. #### `virtualCluster` object [​](#spec-destination-virtualCluster "Direct link to spec-destination-virtualCluster")VirtualCluster names the tenant cluster to deploy into. Mutually exclusive with Cluster. ##### `name` string [​](#spec-destination-virtualCluster-name "Direct link to spec-destination-virtualCluster-name")Name of the tenant cluster. #### `cluster` object [​](#spec-destination-cluster "Direct link to spec-destination-cluster")Cluster names the connected cluster to deploy into. Mutually exclusive with VirtualCluster. ##### `name` string [​](#spec-destination-cluster-name "Direct link to spec-destination-cluster-name")Name of the connected cluster. ### `template` object [​](#spec-template "Direct link to spec-template")Template defines the stack inline. It has the same shape as a StackTemplate's parameters and tasks, so a template's payload can be copy-pasted here and back. Exactly one of Template or TemplateRef must be set (validated at admission). #### `parameters` object\[] [​](#spec-template-parameters "Direct link to spec-template-parameters")Parameters declares the typed parameters (with defaults) a stack accepts, reused from AppParameter. ##### `variable` string [​](#spec-template-parameters-variable "Direct link to spec-template-parameters-variable")Variable is the path of the variable. Can be foo or foo.bar for nested objects. ##### `label` string [​](#spec-template-parameters-label "Direct link to spec-template-parameters-label")Label is the label to show for this parameter ##### `description` string [​](#spec-template-parameters-description "Direct link to spec-template-parameters-description")Description is the description to show for this parameter ##### `type` string [​](#spec-template-parameters-type "Direct link to spec-template-parameters-type")Type of the parameter. Can be one of: string, multiline, boolean, number and password ##### `options` string\[] [​](#spec-template-parameters-options "Direct link to spec-template-parameters-options")Options is a slice of strings, where each string represents a mutually exclusive choice. ##### `min` integer [​](#spec-template-parameters-min "Direct link to spec-template-parameters-min")Min is the minimum number if type is number ##### `max` integer [​](#spec-template-parameters-max "Direct link to spec-template-parameters-max")Max is the maximum number if type is number ##### `required` boolean [​](#spec-template-parameters-required "Direct link to spec-template-parameters-required")Required specifies if this parameter is required ##### `defaultValue` string [​](#spec-template-parameters-defaultValue "Direct link to spec-template-parameters-defaultValue")DefaultValue is the default value if none is specified ##### `placeholder` string [​](#spec-template-parameters-placeholder "Direct link to spec-template-parameters-placeholder")Placeholder shown in the UI ##### `invalidation` string [​](#spec-template-parameters-invalidation "Direct link to spec-template-parameters-invalidation")Invalidation regex that if matched will reject the input ##### `validation` string [​](#spec-template-parameters-validation "Direct link to spec-template-parameters-validation")Validation regex that if matched will allow the input ##### `section` string [​](#spec-template-parameters-section "Direct link to spec-template-parameters-section")Section where this app should be displayed. Apps with the same section name will be grouped together #### `tasks` object\[] [​](#spec-template-tasks "Direct link to spec-template-tasks")Tasks is the DAG blueprint. Each task follows the same task-level mutual-exclusivity rule (ArgoCDApplication XOR App), validated at admission. ##### `name` required string [​](#spec-template-tasks-name "Direct link to spec-template-tasks-name")Name is the stable identifier of the task. DNS-label-safe, unique within the stack. A task that declares outputs may use letters and digits only: its outputs are referenced as {{ .Outputs.task.name }}, and the template syntax cannot address a name containing "-". ##### `dependsOn` string\[] [​](#spec-template-tasks-dependsOn "Direct link to spec-template-tasks-dependsOn")DependsOn lists task names that must reach readiness before this task starts. ##### `argoCDApplication` object [​](#spec-template-tasks-argoCDApplication "Direct link to spec-template-tasks-argoCDApplication")ArgoCDApplication defines this task as an Argo CD application. Exactly one of ArgoCDApplication or App must be set per task (validated at admission). ##### `template` object [​](#spec-template-tasks-argoCDApplication-template "Direct link to spec-template-tasks-argoCDApplication-template")Template is an inline ArgoCD application template definition. Exactly one of Template or TemplateRef must be set per task (validated at admission). ##### `metadata` object [​](#spec-template-tasks-argoCDApplication-template-metadata "Direct link to spec-template-tasks-argoCDApplication-template-metadata") ##### `labels` object [​](#spec-template-tasks-argoCDApplication-template-metadata-labels "Direct link to spec-template-tasks-argoCDApplication-template-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-template-tasks-argoCDApplication-template-metadata-annotations "Direct link to spec-template-tasks-argoCDApplication-template-metadata-annotations")Annotations are annotations on the object ##### `spec` object [​](#spec-template-tasks-argoCDApplication-template-spec "Direct link to spec-template-tasks-argoCDApplication-template-spec")Spec is the spec of the Argo CD application ##### `source` object [​](#spec-template-tasks-argoCDApplication-template-spec-source "Direct link to spec-template-tasks-argoCDApplication-template-spec-source") ##### `repoURL` required string [​](#spec-template-tasks-argoCDApplication-template-spec-source-repoURL "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-repoURL") ##### `path` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-path "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-path") ##### `targetRevision` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-targetRevision "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-targetRevision") ##### `helm` object [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm") ##### `valueFiles` string\[] [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-valueFiles "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-valueFiles") ##### `parameters` object\[] [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-parameters "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-parameters") ##### `name` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-parameters-name "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-parameters-name") ##### `value` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-parameters-value "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-parameters-value") ##### `forceString` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-parameters-forceString "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-parameters-forceString") ##### `releaseName` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-releaseName "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-releaseName") ##### `values` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-values "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-values") ##### `fileParameters` object\[] [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-fileParameters "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-fileParameters") ##### `name` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-fileParameters-name "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-fileParameters-name") ##### `path` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-fileParameters-path "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-fileParameters-path") ##### `version` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-version "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-version") ##### `passCredentials` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-passCredentials "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-passCredentials") ##### `ignoreMissingValueFiles` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-ignoreMissingValueFiles "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-ignoreMissingValueFiles") ##### `skipCrds` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-skipCrds "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-skipCrds") ##### `valuesObject` object [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-valuesObject "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-valuesObject") ##### `namespace` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-namespace "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-namespace") ##### `kubeVersion` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-kubeVersion "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-kubeVersion") ##### `apiVersions` string\[] [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-apiVersions "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-apiVersions") ##### `skipTests` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-skipTests "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-skipTests") ##### `skipSchemaValidation` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-helm-skipSchemaValidation "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-helm-skipSchemaValidation") ##### `kustomize` object [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize") ##### `namePrefix` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-namePrefix "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-namePrefix") ##### `nameSuffix` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-nameSuffix "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-nameSuffix") ##### `images` object [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-images "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-images") ##### `commonLabels` object [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-commonLabels "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-commonLabels") ##### `version` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-version "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-version") ##### `commonAnnotations` object [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-commonAnnotations "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-commonAnnotations") ##### `forceCommonLabels` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-forceCommonLabels "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-forceCommonLabels") ##### `forceCommonAnnotations` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-forceCommonAnnotations "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-forceCommonAnnotations") ##### `namespace` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-namespace "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-namespace") ##### `commonAnnotationsEnvsubst` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-commonAnnotationsEnvsubst "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-commonAnnotationsEnvsubst") ##### `replicas` object [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-replicas "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-replicas") ##### `patches` object [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-patches "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-patches") ##### `components` string\[] [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-components "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-components") ##### `ignoreMissingComponents` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-ignoreMissingComponents "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-ignoreMissingComponents") ##### `labelWithoutSelector` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-labelWithoutSelector "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-labelWithoutSelector") ##### `kubeVersion` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-kubeVersion "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-kubeVersion") ##### `apiVersions` string\[] [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-apiVersions "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-apiVersions") ##### `labelIncludeTemplates` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-kustomize-labelIncludeTemplates "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-kustomize-labelIncludeTemplates") ##### `directory` object [​](#spec-template-tasks-argoCDApplication-template-spec-source-directory "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-directory") ##### `recurse` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-directory-recurse "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-directory-recurse") ##### `jsonnet` object [​](#spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet") ##### `extVars` object\[] [​](#spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars") ##### `name` required string [​](#spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars-name "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars-name") ##### `value` required string [​](#spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars-value "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars-value") ##### `code` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars-code "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars-code") ##### `tlas` object\[] [​](#spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas") ##### `name` required string [​](#spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas-name "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas-name") ##### `value` required string [​](#spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas-value "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas-value") ##### `code` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas-code "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas-code") ##### `libs` string\[] [​](#spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-libs "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-directory-jsonnet-libs") ##### `exclude` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-directory-exclude "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-directory-exclude") ##### `include` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-directory-include "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-directory-include") ##### `plugin` object [​](#spec-template-tasks-argoCDApplication-template-spec-source-plugin "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-plugin") ##### `name` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-plugin-name "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-plugin-name") ##### `env` object [​](#spec-template-tasks-argoCDApplication-template-spec-source-plugin-env "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-plugin-env") ##### `parameters` object [​](#spec-template-tasks-argoCDApplication-template-spec-source-plugin-parameters "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-plugin-parameters") ##### `chart` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-chart "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-chart") ##### `ref` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-ref "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-ref") ##### `name` string [​](#spec-template-tasks-argoCDApplication-template-spec-source-name "Direct link to spec-template-tasks-argoCDApplication-template-spec-source-name") ##### `destination` required object [​](#spec-template-tasks-argoCDApplication-template-spec-destination "Direct link to spec-template-tasks-argoCDApplication-template-spec-destination") ##### `server` string [​](#spec-template-tasks-argoCDApplication-template-spec-destination-server "Direct link to spec-template-tasks-argoCDApplication-template-spec-destination-server") ##### `namespace` string [​](#spec-template-tasks-argoCDApplication-template-spec-destination-namespace "Direct link to spec-template-tasks-argoCDApplication-template-spec-destination-namespace") ##### `name` string [​](#spec-template-tasks-argoCDApplication-template-spec-destination-name "Direct link to spec-template-tasks-argoCDApplication-template-spec-destination-name") ##### `project` required string [​](#spec-template-tasks-argoCDApplication-template-spec-project "Direct link to spec-template-tasks-argoCDApplication-template-spec-project") ##### `syncPolicy` object [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy") ##### `automated` object [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-automated "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-automated") ##### `prune` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-automated-prune "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-automated-prune") ##### `selfHeal` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-automated-selfHeal "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-automated-selfHeal") ##### `allowEmpty` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-automated-allowEmpty "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-automated-allowEmpty") ##### `enabled` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-automated-enabled "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-automated-enabled") ##### `syncOptions` object [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-syncOptions "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-syncOptions") ##### `retry` object [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-retry "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-retry") ##### `limit` integer [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-retry-limit "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-retry-limit") ##### `backoff` object [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff") ##### `duration` string [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff-duration "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff-duration") ##### `factor` integer [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff-factor "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff-factor") ##### `maxDuration` string [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff-maxDuration "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff-maxDuration") ##### `refresh` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-retry-refresh "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-retry-refresh") ##### `managedNamespaceMetadata` object [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-managedNamespaceMetadata "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-managedNamespaceMetadata") ##### `labels` object [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-managedNamespaceMetadata-labels "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-managedNamespaceMetadata-labels") ##### `annotations` object [​](#spec-template-tasks-argoCDApplication-template-spec-syncPolicy-managedNamespaceMetadata-annotations "Direct link to spec-template-tasks-argoCDApplication-template-spec-syncPolicy-managedNamespaceMetadata-annotations") ##### `ignoreDifferences` object [​](#spec-template-tasks-argoCDApplication-template-spec-ignoreDifferences "Direct link to spec-template-tasks-argoCDApplication-template-spec-ignoreDifferences") ##### `info` object\[] [​](#spec-template-tasks-argoCDApplication-template-spec-info "Direct link to spec-template-tasks-argoCDApplication-template-spec-info") ##### `name` required string [​](#spec-template-tasks-argoCDApplication-template-spec-info-name "Direct link to spec-template-tasks-argoCDApplication-template-spec-info-name") ##### `value` required string [​](#spec-template-tasks-argoCDApplication-template-spec-info-value "Direct link to spec-template-tasks-argoCDApplication-template-spec-info-value") ##### `revisionHistoryLimit` integer [​](#spec-template-tasks-argoCDApplication-template-spec-revisionHistoryLimit "Direct link to spec-template-tasks-argoCDApplication-template-spec-revisionHistoryLimit") ##### `sources` object [​](#spec-template-tasks-argoCDApplication-template-spec-sources "Direct link to spec-template-tasks-argoCDApplication-template-spec-sources") ##### `sourceHydrator` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator") ##### `drySource` required object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource") ##### `repoURL` required string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-repoURL "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-repoURL") ##### `targetRevision` required string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-targetRevision "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-targetRevision") ##### `path` required string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-path "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-path") ##### `helm` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm") ##### `valueFiles` string\[] [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-valueFiles "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-valueFiles") ##### `parameters` object\[] [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters") ##### `name` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters-name "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters-name") ##### `value` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters-value "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters-value") ##### `forceString` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters-forceString "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters-forceString") ##### `releaseName` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-releaseName "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-releaseName") ##### `values` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-values "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-values") ##### `fileParameters` object\[] [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-fileParameters "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-fileParameters") ##### `name` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-fileParameters-name "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-fileParameters-name") ##### `path` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-fileParameters-path "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-fileParameters-path") ##### `version` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-version "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-version") ##### `passCredentials` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-passCredentials "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-passCredentials") ##### `ignoreMissingValueFiles` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-ignoreMissingValueFiles "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-ignoreMissingValueFiles") ##### `skipCrds` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-skipCrds "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-skipCrds") ##### `valuesObject` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-valuesObject "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-valuesObject") ##### `namespace` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-namespace "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-namespace") ##### `kubeVersion` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-kubeVersion "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-kubeVersion") ##### `apiVersions` string\[] [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-apiVersions "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-apiVersions") ##### `skipTests` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-skipTests "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-skipTests") ##### `skipSchemaValidation` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-skipSchemaValidation "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-skipSchemaValidation") ##### `kustomize` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize") ##### `namePrefix` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-namePrefix "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-namePrefix") ##### `nameSuffix` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-nameSuffix "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-nameSuffix") ##### `images` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-images "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-images") ##### `commonLabels` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-commonLabels "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-commonLabels") ##### `version` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-version "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-version") ##### `commonAnnotations` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-commonAnnotations "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-commonAnnotations") ##### `forceCommonLabels` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-forceCommonLabels "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-forceCommonLabels") ##### `forceCommonAnnotations` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-forceCommonAnnotations "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-forceCommonAnnotations") ##### `namespace` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-namespace "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-namespace") ##### `commonAnnotationsEnvsubst` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-commonAnnotationsEnvsubst "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-commonAnnotationsEnvsubst") ##### `replicas` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-replicas "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-replicas") ##### `patches` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-patches "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-patches") ##### `components` string\[] [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-components "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-components") ##### `ignoreMissingComponents` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-ignoreMissingComponents "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-ignoreMissingComponents") ##### `labelWithoutSelector` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-labelWithoutSelector "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-labelWithoutSelector") ##### `kubeVersion` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-kubeVersion "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-kubeVersion") ##### `apiVersions` string\[] [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-apiVersions "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-apiVersions") ##### `labelIncludeTemplates` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-labelIncludeTemplates "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-labelIncludeTemplates") ##### `directory` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory") ##### `recurse` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-recurse "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-recurse") ##### `jsonnet` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet") ##### `extVars` object\[] [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars") ##### `name` required string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-name "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-name") ##### `value` required string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-value "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-value") ##### `code` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-code "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-code") ##### `tlas` object\[] [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas") ##### `name` required string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-name "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-name") ##### `value` required string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-value "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-value") ##### `code` boolean [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-code "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-code") ##### `libs` string\[] [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-libs "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-libs") ##### `exclude` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-exclude "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-exclude") ##### `include` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-include "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-include") ##### `plugin` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin") ##### `name` string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin-name "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin-name") ##### `env` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin-env "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin-env") ##### `parameters` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin-parameters "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin-parameters") ##### `syncSource` required object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-syncSource "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-syncSource") ##### `targetBranch` required string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-syncSource-targetBranch "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-syncSource-targetBranch") ##### `path` required string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-syncSource-path "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-syncSource-path") ##### `hydrateTo` object [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-hydrateTo "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-hydrateTo") ##### `targetBranch` required string [​](#spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-hydrateTo-targetBranch "Direct link to spec-template-tasks-argoCDApplication-template-spec-sourceHydrator-hydrateTo-targetBranch") ##### `templateRef` object [​](#spec-template-tasks-argoCDApplication-templateRef "Direct link to spec-template-tasks-argoCDApplication-templateRef")TemplateRef references an ArgoCDApplicationTemplate (per-application blueprint). Exactly one of Template or TemplateRef must be set per task (validated at admission). ##### `name` string [​](#spec-template-tasks-argoCDApplication-templateRef-name "Direct link to spec-template-tasks-argoCDApplication-templateRef-name")Name holds the name of the blueprint template to reference. ##### `parameters` object [​](#spec-template-tasks-argoCDApplication-parameters "Direct link to spec-template-tasks-argoCDApplication-parameters")Parameters are rendered as part of the task, then passed to the referenced ArgoCDApplicationTemplate. Only valid with TemplateRef; admission rejects the field on a task with an inline Template, whose values belong in the definition itself. ##### `app` object [​](#spec-template-tasks-app "Direct link to spec-template-tasks-app")App defines this task as a platform App. Exactly one of ArgoCDApplication or App must be set per task (validated at admission). ##### `template` object [​](#spec-template-tasks-app-template "Direct link to spec-template-tasks-app-template")Template is an inline app definition. Exactly one of Template or TemplateRef must be set per task (validated at admission). ##### `metadata` object [​](#spec-template-tasks-app-template-metadata "Direct link to spec-template-tasks-app-template-metadata") ##### `labels` object [​](#spec-template-tasks-app-template-metadata-labels "Direct link to spec-template-tasks-app-template-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-template-tasks-app-template-metadata-annotations "Direct link to spec-template-tasks-app-template-metadata-annotations")Annotations are annotations on the object ##### `spec` object [​](#spec-template-tasks-app-template-spec "Direct link to spec-template-tasks-app-template-spec")Spec is the app definition of the child app instance. ##### `displayName` string [​](#spec-template-tasks-app-template-spec-displayName "Direct link to spec-template-tasks-app-template-spec-displayName")DisplayName is the name shown for the app instance in the UI. ##### `description` string [​](#spec-template-tasks-app-template-spec-description "Direct link to spec-template-tasks-app-template-spec-description")Description describes the app instance. ##### `releaseName` string [​](#spec-template-tasks-app-template-spec-releaseName "Direct link to spec-template-tasks-app-template-spec-releaseName")ReleaseName is the helm release name for the app instance. If empty, it defaults to the generated child name. ##### `defaultNamespace` string [​](#spec-template-tasks-app-template-spec-defaultNamespace "Direct link to spec-template-tasks-app-template-spec-defaultNamespace")DefaultNamespace is the default namespace this app should installed in. ##### `readme` string [​](#spec-template-tasks-app-template-spec-readme "Direct link to spec-template-tasks-app-template-spec-readme")Readme is a longer markdown string that describes the app. ##### `icon` string [​](#spec-template-tasks-app-template-spec-icon "Direct link to spec-template-tasks-app-template-spec-icon")Icon holds an URL to the app icon ##### `config` object [​](#spec-template-tasks-app-template-spec-config "Direct link to spec-template-tasks-app-template-spec-config")Config is the helm config to use to deploy the helm release ##### `chart` object [​](#spec-template-tasks-app-template-spec-config-chart "Direct link to spec-template-tasks-app-template-spec-config-chart")Chart holds information about a chart that should get deployed ##### `name` string [​](#spec-template-tasks-app-template-spec-config-chart-name "Direct link to spec-template-tasks-app-template-spec-config-chart-name")Name is the chart name in the repository ##### `version` string [​](#spec-template-tasks-app-template-spec-config-chart-version "Direct link to spec-template-tasks-app-template-spec-config-chart-version")Version is the chart version in the repository ##### `repoURL` string [​](#spec-template-tasks-app-template-spec-config-chart-repoURL "Direct link to spec-template-tasks-app-template-spec-config-chart-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#spec-template-tasks-app-template-spec-config-chart-username "Direct link to spec-template-tasks-app-template-spec-config-chart-username")The username that is required for this repository ##### `usernameRef` object [​](#spec-template-tasks-app-template-spec-config-chart-usernameRef "Direct link to spec-template-tasks-app-template-spec-config-chart-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#spec-template-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef "Direct link to spec-template-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef-project "Direct link to spec-template-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef-name "Direct link to spec-template-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef-key "Direct link to spec-template-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#spec-template-tasks-app-template-spec-config-chart-password "Direct link to spec-template-tasks-app-template-spec-config-chart-password")The password that is required for this repository ##### `passwordRef` object [​](#spec-template-tasks-app-template-spec-config-chart-passwordRef "Direct link to spec-template-tasks-app-template-spec-config-chart-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#spec-template-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef "Direct link to spec-template-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef-project "Direct link to spec-template-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef-name "Direct link to spec-template-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef-key "Direct link to spec-template-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#spec-template-tasks-app-template-spec-config-chart-insecureSkipTlsVerify "Direct link to spec-template-tasks-app-template-spec-config-chart-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `manifests` string [​](#spec-template-tasks-app-template-spec-config-manifests "Direct link to spec-template-tasks-app-template-spec-config-manifests")Manifests holds kube manifests that will be deployed as a chart ##### `bash` object [​](#spec-template-tasks-app-template-spec-config-bash "Direct link to spec-template-tasks-app-template-spec-config-bash")Bash holds the bash script to execute in a container in the target ##### `script` string [​](#spec-template-tasks-app-template-spec-config-bash-script "Direct link to spec-template-tasks-app-template-spec-config-bash-script")Script is the script to execute. ##### `image` string [​](#spec-template-tasks-app-template-spec-config-bash-image "Direct link to spec-template-tasks-app-template-spec-config-bash-image")Image is the image to use for this app ##### `clusterRole` string [​](#spec-template-tasks-app-template-spec-config-bash-clusterRole "Direct link to spec-template-tasks-app-template-spec-config-bash-clusterRole")ClusterRole is the cluster role to use for this job ##### `podSecurityContext` object [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext")PodSecurityContext for the bash pod. ##### `seLinuxOptions` object [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions")The SELinux context to be applied to all containers. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `user` string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-user "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-user")User is a SELinux user label that applies to the container. ##### `role` string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-role "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-role")Role is a SELinux role label that applies to the container. ##### `type` string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-type "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-type")Type is a SELinux type label that applies to the container. ##### `level` string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-level "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-level")Level is SELinux level label that applies to the container. ##### `windowsOptions` object [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions")The Windows specific settings applied to all containers. If unspecified, the options within a container's SecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. ##### `gmsaCredentialSpecName` string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpecName "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpecName")GMSACredentialSpecName is the name of the GMSA credential spec to use. ##### `gmsaCredentialSpec` string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpec "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpec")GMSACredentialSpec is where the GMSA admission webhook () inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. ##### `runAsUserName` string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-runAsUserName "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-runAsUserName")The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `hostProcess` boolean [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-hostProcess "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-hostProcess")HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. ##### `runAsUser` integer [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-runAsUser "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-runAsUser")The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `runAsGroup` integer [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-runAsGroup "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-runAsGroup")The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `runAsNonRoot` boolean [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-runAsNonRoot "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-runAsNonRoot")Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `supplementalGroups` integer\[] [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-supplementalGroups "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-supplementalGroups")A list of groups applied to the first process run in each container, in addition to the container's primary GID and fsGroup (if specified). If the SupplementalGroupsPolicy feature is enabled, the supplementalGroupsPolicy field determines whether these are in addition to or instead of any group memberships defined in the container image. If unspecified, no additional groups are added, though group memberships defined in the container image may still be used, depending on the supplementalGroupsPolicy field. Note that this field cannot be set when spec.os.name is windows. ##### `supplementalGroupsPolicy` string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-supplementalGroupsPolicy "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-supplementalGroupsPolicy")Defines how supplemental groups of the first container processes are calculated. Valid values are "Merge" and "Strict". If not specified, "Merge" is used. (Alpha) Using the field requires the SupplementalGroupsPolicy feature gate to be enabled and the container runtime must implement support for this feature. Note that this field cannot be set when spec.os.name is windows. TODO: update the default value to "Merge" when spec.os.name is not windows in v1.34 ##### `fsGroup` integer [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-fsGroup "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-fsGroup")A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod:1) The owning GID will be the FSGroup 2) The setgid bit is set (new files created in the volume will be owned by FSGroup) 3) The permission bits are OR'd with rw-rw----If unset, the Kubelet will not modify the ownership and permissions of any volume. Note that this field cannot be set when spec.os.name is windows. ##### `sysctls` object\[] [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-sysctls "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-sysctls")Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported sysctls (by the container runtime) might fail to launch. Note that this field cannot be set when spec.os.name is windows. ##### `name` required string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-sysctls-name "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-sysctls-name")Name of a property to set ##### `value` required string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-sysctls-value "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-sysctls-value")Value of a property to set ##### `fsGroupChangePolicy` string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-fsGroupChangePolicy "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-fsGroupChangePolicy")fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. Note that this field cannot be set when spec.os.name is windows. ##### `seccompProfile` object [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seccompProfile "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seccompProfile")The seccomp options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seccompProfile-type "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seccompProfile-type")type indicates which kind of seccomp profile will be applied. Valid options are:Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. ##### `localhostProfile` string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seccompProfile-localhostProfile "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seccompProfile-localhostProfile")localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. ##### `appArmorProfile` object [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-appArmorProfile "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-appArmorProfile")appArmorProfile is the AppArmor options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-appArmorProfile-type "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-appArmorProfile-type")type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. ##### `localhostProfile` string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-appArmorProfile-localhostProfile "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-appArmorProfile-localhostProfile")localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". ##### `seLinuxChangePolicy` string [​](#spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxChangePolicy "Direct link to spec-template-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxChangePolicy")seLinuxChangePolicy defines how the container's SELinux label is applied to all volumes used by the Pod. It has no effect on nodes that do not support SELinux or to volumes does not support SELinux. Valid values are "MountOption" and "Recursive"."Recursive" means relabeling of all files on all Pod volumes by the container runtime. This may be slow for large volumes, but allows mixing privileged and unprivileged Pods sharing the same volume on the same node."MountOption" mounts all eligible Pod volumes with `-o context` mount option. This requires all Pods that share the same volume to use the same SELinux label. It is not possible to share the same volume among privileged and unprivileged Pods. Eligible volumes are in-tree FibreChannel and iSCSI volumes, and all CSI volumes whose CSI driver announces SELinux support by setting spec.seLinuxMount: true in their CSIDriver instance. Other volumes are always re-labelled recursively. "MountOption" value is allowed only when SELinuxMount feature gate is enabled.If not specified and SELinuxMount feature gate is enabled, "MountOption" is used. If not specified and SELinuxMount feature gate is disabled, "MountOption" is used for ReadWriteOncePod volumes and "Recursive" for all other volumes.This field affects only Pods that have SELinux label set, either in PodSecurityContext or in SecurityContext of all containers.All Pods that use the same volume should use the same seLinuxChangePolicy, otherwise some pods can get stuck in ContainerCreating state. Note that this field cannot be set when spec.os.name is windows. ##### `securityContext` object [​](#spec-template-tasks-app-template-spec-config-bash-securityContext "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext")SecurityContext for the bash container. ##### `capabilities` object [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-capabilities "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-capabilities")The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container runtime. Note that this field cannot be set when spec.os.name is windows. ##### `add` string\[] [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-capabilities-add "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-capabilities-add")Added capabilities ##### `drop` string\[] [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-capabilities-drop "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-capabilities-drop")Removed capabilities ##### `privileged` boolean [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-privileged "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-privileged")Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false. Note that this field cannot be set when spec.os.name is windows. ##### `seLinuxOptions` object [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions")The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `user` string [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-user "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-user")User is a SELinux user label that applies to the container. ##### `role` string [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-role "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-role")Role is a SELinux role label that applies to the container. ##### `type` string [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-type "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-type")Type is a SELinux type label that applies to the container. ##### `level` string [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-level "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-level")Level is SELinux level label that applies to the container. ##### `windowsOptions` object [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-windowsOptions "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-windowsOptions")The Windows specific settings applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. ##### `gmsaCredentialSpecName` string [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-windowsOptions-gmsaCredentialSpecName "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-windowsOptions-gmsaCredentialSpecName")GMSACredentialSpecName is the name of the GMSA credential spec to use. ##### `gmsaCredentialSpec` string [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-windowsOptions-gmsaCredentialSpec "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-windowsOptions-gmsaCredentialSpec")GMSACredentialSpec is where the GMSA admission webhook () inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. ##### `runAsUserName` string [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-windowsOptions-runAsUserName "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-windowsOptions-runAsUserName")The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `hostProcess` boolean [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-windowsOptions-hostProcess "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-windowsOptions-hostProcess")HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. ##### `runAsUser` integer [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-runAsUser "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-runAsUser")The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `runAsGroup` integer [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-runAsGroup "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-runAsGroup")The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `runAsNonRoot` boolean [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-runAsNonRoot "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-runAsNonRoot")Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `readOnlyRootFilesystem` boolean [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-readOnlyRootFilesystem "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-readOnlyRootFilesystem")Whether this container has a read-only root filesystem. Default is false. Note that this field cannot be set when spec.os.name is windows. ##### `allowPrivilegeEscalation` boolean [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-allowPrivilegeEscalation "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-allowPrivilegeEscalation")AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no\_new\_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is:1) run as Privileged 2) has CAP\_SYS\_ADMIN Note that this field cannot be set when spec.os.name is windows. ##### `procMount` string [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-procMount "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-procMount")procMount denotes the type of proc mount to use for the containers. The default value is Default which uses the container runtime defaults for readonly paths and masked paths. Note that this field cannot be set when spec.os.name is windows. ##### `seccompProfile` object [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-seccompProfile "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-seccompProfile")The seccomp options to use by this container. If seccomp options are provided at both the pod & container level, the container options override the pod options. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-seccompProfile-type "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-seccompProfile-type")type indicates which kind of seccomp profile will be applied. Valid options are:Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. ##### `localhostProfile` string [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-seccompProfile-localhostProfile "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-seccompProfile-localhostProfile")localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. ##### `appArmorProfile` object [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-appArmorProfile "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-appArmorProfile")appArmorProfile is the AppArmor options to use by this container. If set, this profile overrides the pod's appArmorProfile. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-appArmorProfile-type "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-appArmorProfile-type")type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. ##### `localhostProfile` string [​](#spec-template-tasks-app-template-spec-config-bash-securityContext-appArmorProfile-localhostProfile "Direct link to spec-template-tasks-app-template-spec-config-bash-securityContext-appArmorProfile-localhostProfile")localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". ##### `values` string [​](#spec-template-tasks-app-template-spec-config-values "Direct link to spec-template-tasks-app-template-spec-config-values")Values is the set of extra Values added to the chart. These values merge with the default values inside of the chart. You can use golang templating in here with values from parameters. ##### `parameters` string [​](#spec-template-tasks-app-template-spec-config-parameters "Direct link to spec-template-tasks-app-template-spec-config-parameters")Parameters are additional helm chart values that will get merged with config and are then used to deploy the helm chart. ##### `annotations` object [​](#spec-template-tasks-app-template-spec-config-annotations "Direct link to spec-template-tasks-app-template-spec-config-annotations")Annotations are extra annotations for this helm release ##### `wait` boolean [​](#spec-template-tasks-app-template-spec-wait "Direct link to spec-template-tasks-app-template-spec-wait")Wait determines if Loft should wait during deploy for the app to become ready ##### `timeout` string [​](#spec-template-tasks-app-template-spec-timeout "Direct link to spec-template-tasks-app-template-spec-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) ##### `parameters` object\[] [​](#spec-template-tasks-app-template-spec-parameters "Direct link to spec-template-tasks-app-template-spec-parameters")Parameters define additional app parameters that will set helm values ##### `variable` string [​](#spec-template-tasks-app-template-spec-parameters-variable "Direct link to spec-template-tasks-app-template-spec-parameters-variable")Variable is the path of the variable. Can be foo or foo.bar for nested objects. ##### `label` string [​](#spec-template-tasks-app-template-spec-parameters-label "Direct link to spec-template-tasks-app-template-spec-parameters-label")Label is the label to show for this parameter ##### `description` string [​](#spec-template-tasks-app-template-spec-parameters-description "Direct link to spec-template-tasks-app-template-spec-parameters-description")Description is the description to show for this parameter ##### `type` string [​](#spec-template-tasks-app-template-spec-parameters-type "Direct link to spec-template-tasks-app-template-spec-parameters-type")Type of the parameter. Can be one of: string, multiline, boolean, number and password ##### `options` string\[] [​](#spec-template-tasks-app-template-spec-parameters-options "Direct link to spec-template-tasks-app-template-spec-parameters-options")Options is a slice of strings, where each string represents a mutually exclusive choice. ##### `min` integer [​](#spec-template-tasks-app-template-spec-parameters-min "Direct link to spec-template-tasks-app-template-spec-parameters-min")Min is the minimum number if type is number ##### `max` integer [​](#spec-template-tasks-app-template-spec-parameters-max "Direct link to spec-template-tasks-app-template-spec-parameters-max")Max is the maximum number if type is number ##### `required` boolean [​](#spec-template-tasks-app-template-spec-parameters-required "Direct link to spec-template-tasks-app-template-spec-parameters-required")Required specifies if this parameter is required ##### `defaultValue` string [​](#spec-template-tasks-app-template-spec-parameters-defaultValue "Direct link to spec-template-tasks-app-template-spec-parameters-defaultValue")DefaultValue is the default value if none is specified ##### `placeholder` string [​](#spec-template-tasks-app-template-spec-parameters-placeholder "Direct link to spec-template-tasks-app-template-spec-parameters-placeholder")Placeholder shown in the UI ##### `invalidation` string [​](#spec-template-tasks-app-template-spec-parameters-invalidation "Direct link to spec-template-tasks-app-template-spec-parameters-invalidation")Invalidation regex that if matched will reject the input ##### `validation` string [​](#spec-template-tasks-app-template-spec-parameters-validation "Direct link to spec-template-tasks-app-template-spec-parameters-validation")Validation regex that if matched will allow the input ##### `section` string [​](#spec-template-tasks-app-template-spec-parameters-section "Direct link to spec-template-tasks-app-template-spec-parameters-section")Section where this app should be displayed. Apps with the same section name will be grouped together ##### `streamContainer` object [​](#spec-template-tasks-app-template-spec-streamContainer "Direct link to spec-template-tasks-app-template-spec-streamContainer")DEPRECATED: Use config.bash instead StreamContainer can be used to stream a containers logs instead of the helm output. ##### `selector` object [​](#spec-template-tasks-app-template-spec-streamContainer-selector "Direct link to spec-template-tasks-app-template-spec-streamContainer-selector")Label selector for pods. The newest matching pod will be used to stream logs from ##### `matchLabels` object [​](#spec-template-tasks-app-template-spec-streamContainer-selector-matchLabels "Direct link to spec-template-tasks-app-template-spec-streamContainer-selector-matchLabels")matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. ##### `matchExpressions` object\[] [​](#spec-template-tasks-app-template-spec-streamContainer-selector-matchExpressions "Direct link to spec-template-tasks-app-template-spec-streamContainer-selector-matchExpressions")matchExpressions is a list of label selector requirements. The requirements are ANDed. ##### `key` required string [​](#spec-template-tasks-app-template-spec-streamContainer-selector-matchExpressions-key "Direct link to spec-template-tasks-app-template-spec-streamContainer-selector-matchExpressions-key")key is the label key that the selector applies to. ##### `operator` required string [​](#spec-template-tasks-app-template-spec-streamContainer-selector-matchExpressions-operator "Direct link to spec-template-tasks-app-template-spec-streamContainer-selector-matchExpressions-operator")operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. ##### `values` string\[] [​](#spec-template-tasks-app-template-spec-streamContainer-selector-matchExpressions-values "Direct link to spec-template-tasks-app-template-spec-streamContainer-selector-matchExpressions-values")values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. ##### `container` string [​](#spec-template-tasks-app-template-spec-streamContainer-container "Direct link to spec-template-tasks-app-template-spec-streamContainer-container")Container is the container name to use ##### `templateRef` object [​](#spec-template-tasks-app-templateRef "Direct link to spec-template-tasks-app-templateRef")TemplateRef references a named app by name and version. Exactly one of Template or TemplateRef must be set per task (validated at admission). ##### `name` string [​](#spec-template-tasks-app-templateRef-name "Direct link to spec-template-tasks-app-templateRef-name")Name holds the name of the app to reference. ##### `version` string [​](#spec-template-tasks-app-templateRef-version "Direct link to spec-template-tasks-app-templateRef-version")Version of the app to deploy. If empty, the latest version is deployed. ##### `parameters` object [​](#spec-template-tasks-app-parameters "Direct link to spec-template-tasks-app-parameters")Parameters are passed to the referenced app. Only valid with TemplateRef; admission rejects the field on a task with an inline Template, whose values belong in the definition itself. ##### `timeout` object [​](#spec-template-tasks-timeout "Direct link to spec-template-tasks-timeout")Timeout overrides Defaults.TaskTimeout for this task. ##### `Duration` required object [​](#spec-template-tasks-timeout-Duration "Direct link to spec-template-tasks-timeout-Duration") ##### `outputs` object\[] [​](#spec-template-tasks-outputs "Direct link to spec-template-tasks-outputs")Outputs declares named values this task publishes once it is Healthy. Later tasks consume them in their specs as {{ .Outputs.task.name }} and must list this task in dependsOn (validated at admission). ##### `name` required string [​](#spec-template-tasks-outputs-name "Direct link to spec-template-tasks-outputs-name")Name identifies the output. Letters and digits only, unique within the task: the output is referenced as {{ .Outputs.task.name }}, and the template syntax cannot address a name containing "-". ##### `fromSecret` object [​](#spec-template-tasks-outputs-fromSecret "Direct link to spec-template-tasks-outputs-fromSecret")FromSecret reads the value from a Secret key on the destination cluster. Exactly one of fromSecret or fromResource must be set per output. ##### `namespace` required string [​](#spec-template-tasks-outputs-fromSecret-namespace "Direct link to spec-template-tasks-outputs-fromSecret-namespace")Namespace of the Secret. Must be a namespace this stack deploys into. ##### `name` required string [​](#spec-template-tasks-outputs-fromSecret-name "Direct link to spec-template-tasks-outputs-fromSecret-name")Name of the Secret. ##### `key` required string [​](#spec-template-tasks-outputs-fromSecret-key "Direct link to spec-template-tasks-outputs-fromSecret-key")Key inside the Secret's data. ##### `fromResource` object [​](#spec-template-tasks-outputs-fromResource "Direct link to spec-template-tasks-outputs-fromResource")FromResource reads a single field from a resource on the destination cluster. Exactly one of fromSecret or fromResource must be set per output. ##### `apiVersion` required string [​](#spec-template-tasks-outputs-fromResource-apiVersion "Direct link to spec-template-tasks-outputs-fromResource-apiVersion")APIVersion of the resource, e.g. "v1" or "apps/v1". ##### `kind` required string [​](#spec-template-tasks-outputs-fromResource-kind "Direct link to spec-template-tasks-outputs-fromResource-kind")Kind of the resource, e.g. "Service". ##### `namespace` required string [​](#spec-template-tasks-outputs-fromResource-namespace "Direct link to spec-template-tasks-outputs-fromResource-namespace")Namespace of the resource. Must be a namespace this stack deploys into; cluster-scoped resources cannot be read. ##### `name` required string [​](#spec-template-tasks-outputs-fromResource-name "Direct link to spec-template-tasks-outputs-fromResource-name")Name of the resource. ##### `jsonPath` required string [​](#spec-template-tasks-outputs-fromResource-jsonPath "Direct link to spec-template-tasks-outputs-fromResource-jsonPath")JSONPath selects the field, in the kubectl template syntax, e.g. "{.spec.clusterIP}". It must select exactly one scalar value. #### `publishedOutputs` object\[] [​](#spec-template-publishedOutputs "Direct link to spec-template-publishedOutputs")PublishedOutputs selects which captured task outputs the stack exposes to its users, optionally under a different name. The values come from the captured task outputs; nothing extra is read from the cluster. They are read through the instance's outputs subresource, never from its status. ##### `name` required string [​](#spec-template-publishedOutputs-name "Direct link to spec-template-publishedOutputs-name")Name is the public name of the output. Unique within the stack; renaming the task output is allowed and expected. ##### `fromTask` required object [​](#spec-template-publishedOutputs-fromTask "Direct link to spec-template-publishedOutputs-fromTask")FromTask selects which captured task output to publish. It must name an existing task and one of its declared outputs (validated at admission). ##### `task` required string [​](#spec-template-publishedOutputs-fromTask-task "Direct link to spec-template-publishedOutputs-fromTask-task")Task is the task name. ##### `output` required string [​](#spec-template-publishedOutputs-fromTask-output "Direct link to spec-template-publishedOutputs-fromTask-output")Output is the output name declared on that task. ### `templateRef` object [​](#spec-templateRef "Direct link to spec-templateRef")TemplateRef references a cluster-scoped StackTemplate, resolved live each reconcile. Exactly one of Template or TemplateRef must be set (validated at admission). #### `name` required string [​](#spec-templateRef-name "Direct link to spec-templateRef-name")Name holds the name of the StackTemplate to reference. ### `parameters` object [​](#spec-parameters "Direct link to spec-parameters")Parameters are the values the template's tasks reference as .Values.(name), the same way an app reads its parameters. Declarations provide defaults and validation but are not a whitelist: values without a matching declaration are passed through and may be referenced too. ### `defaults` object [​](#spec-defaults "Direct link to spec-defaults")Defaults are applied to all tasks unless overridden per-task. #### `taskTimeout` object [​](#spec-defaults-taskTimeout "Direct link to spec-defaults-taskTimeout")TaskTimeout is the default per-task timeout. The empty value is treated as 10m. ##### `Duration` required object [​](#spec-defaults-taskTimeout-Duration "Direct link to spec-defaults-taskTimeout-Duration") ### `prunePolicy` string [​](#spec-prunePolicy "Direct link to spec-prunePolicy")PrunePolicy controls what happens to an owned application whose task is removed from the resolved task set. Leaving it out is treated as Retain. ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ## `status` object [​](#status "Direct link to status") ### `phase` string [​](#status-phase "Direct link to status-phase")Phase is the aggregate phase of the StackInstance. Consumers must tolerate unknown values. ### `conditions` object [​](#status-conditions "Direct link to status-conditions")Conditions holds several conditions the StackInstance might be in. ### `tasks` object\[] [​](#status-tasks "Direct link to status-tasks")Tasks is the denormalized per-task status, including not-yet-materialized tasks. It is empty when the instance stopped before it could work out its task set, such as a missing destination or a template that does not resolve. #### `name` required string [​](#status-tasks-name "Direct link to status-tasks-name")Name is the task name. #### `phase` required string [​](#status-tasks-phase "Direct link to status-tasks-phase")Phase is the per-task phase. It is type-agnostic: Healthy/Failed report the task's resolved readiness, not an ArgoCD-specific state. #### `type` string [​](#status-tasks-type "Direct link to status-tasks-type")Type is the task's child kind (argoCDApplication or app). Empty on status objects written before this field existed. #### `dependsOn` string\[] [​](#status-tasks-dependsOn "Direct link to status-tasks-dependsOn")DependsOn are the resolved dependency edges for this task (inline AND templateRef), so the UI can render the graph from this object alone. #### `applicationName` string [​](#status-tasks-applicationName "Direct link to status-tasks-applicationName")ApplicationName is the materialized child object name (empty until created). #### `synced` boolean [​](#status-tasks-synced "Direct link to status-tasks-synced")Synced mirrors the child ArgoCD sync state. Set only for argoCDApplication tasks; empty for other task types. #### `health` string [​](#status-tasks-health "Direct link to status-tasks-health")Health mirrors the child ArgoCD health. Set only for argoCDApplication tasks; empty for other task types. #### `reason` string [​](#status-tasks-reason "Direct link to status-tasks-reason")Reason describes the reason in machine-readable form why the task is in the current phase. Empty while the task is healthy. #### `message` string [​](#status-tasks-message "Direct link to status-tasks-message")Message describes the reason in human-readable form why the task is in the current phase. #### `lastTransitionTime` object [​](#status-tasks-lastTransitionTime "Direct link to status-tasks-lastTransitionTime")LastTransitionTime is when this task last changed phase. #### `timeoutAt` object [​](#status-tasks-timeoutAt "Direct link to status-tasks-timeoutAt")TimeoutAt is when the stack fails the task if it is still progressing. Set only while the task is Progressing, and moved when the task starts a new wait, such as capturing its outputs. ### `orphanedApplications` object\[] [​](#status-orphanedApplications "Direct link to status-orphanedApplications")OrphanedApplications lists owned children whose task is no longer in the resolved task set (template drift). Reported, not deleted, unless prunePolicy is Prune. #### `applicationName` required string [​](#status-orphanedApplications-applicationName "Direct link to status-orphanedApplications-applicationName")ApplicationName is the child's Kubernetes object name (what Prune acts on). The name is retained for traceability; with more than one task type it may name a non-Application child, so read Type to know the kind. #### `type` string [​](#status-orphanedApplications-type "Direct link to status-orphanedApplications-type")Type is the orphan's child kind (argoCDApplication or app), stamped from the adapter that listed it. Empty on status objects written before this field existed. #### `taskName` string [​](#status-orphanedApplications-taskName "Direct link to status-orphanedApplications-taskName")TaskName is the name of the task this child was created for. That task is no longer in the resolved task set. #### `dependsOn` string\[] [​](#status-orphanedApplications-dependsOn "Direct link to status-orphanedApplications-dependsOn")DependsOn are the orphan's resolved dependency edges, captured when the task left the resolved set and carried forward here. They give Prune the reverse-dependency order so a child something still depends on is never deleted first. ### `observedGeneration` integer [​](#status-observedGeneration "Direct link to status-observedGeneration")ObservedGeneration is the generation the controller last reconciled. ## Retrieve: StackInstances[​](#retrieve-stackinstances "Direct link to Retrieve: StackInstances") You can either use curl or kubectl to retrieve StackInstances. * kubectl * curl ### Retrieve a list of StackInstances[​](#retrieve-a-list-of-stackinstances "Direct link to Retrieve a list of StackInstances") Run the following command to list all StackInstances in project `my-project`: ``` kubectl get stackinstances.management.loft.sh -n p-my-project -o yaml ``` ### Retrieve a single StackInstance by name[​](#retrieve-a-single-stackinstance-by-name "Direct link to Retrieve a single StackInstance by name") Run the following kubectl command to get StackInstance `example-platform` in project `my-project`: ``` kubectl get stackinstances.management.loft.sh example-platform -n p-my-project -o yaml ``` ### Retrieve a list of StackInstances[​](#retrieve-a-list-of-stackinstances-1 "Direct link to Retrieve a list of StackInstances") Run the following curl command to list all StackInstances in project `my-project`: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/stackinstances" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single StackInstance by name[​](#get-a-single-stackinstance-by-name "Direct link to Get a single StackInstance by name") Run the following curl command to get StackInstance `example-platform` in project `my-project`: ``` # Exchange example-platform in the url below with the name of the StackInstance curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/stackinstances/example-platform" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: StackInstance[​](#create-stackinstance "Direct link to Create: StackInstance") You can either use curl or kubectl to create a new StackInstance. Make sure to set the project in the `metadata.namespace` field you want to create the StackInstance in. If your project has the id `my-project`, the corresponding namespace would be `p-my-project`. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: StackInstance metadata: name: example-platform namespace: p-my-project spec: destination: virtualCluster: name: development displayName: Example application platform owner: user: admin prunePolicy: Retain templateRef: name: example-platform status: {} ``` Then create the StackInstance `example-platform` in project `my-project` with: ``` kubectl create -f object.yaml -n p-my-project ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: StackInstance metadata: name: example-platform namespace: p-my-project spec: destination: virtualCluster: name: development displayName: Example application platform owner: user: admin prunePolicy: Retain templateRef: name: example-platform status: {} ``` Run the following curl command to create a new StackInstance `example-platform` in project `my-project`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/stackinstances" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: StackInstance[​](#update-stackinstance "Direct link to Update: StackInstance") You can either use curl or kubectl to update StackInstances. * kubectl * curl ### Update StackInstance[​](#update-stackinstance "Direct link to Update StackInstance") Run the following command to update StackInstance `example-platform` in project `my-project`: ``` kubectl edit stackinstances.management.loft.sh example-platform -n p-my-project ``` Then edit the object and upon save, kubectl will update the resource. ### Patch StackInstance[​](#patch-stackinstance "Direct link to Patch StackInstance") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the StackInstance `example-platform` in project `my-project` via a patch: ``` kubectl patch stackinstances.management.loft.sh example-platform -n p-my-project \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update StackInstance[​](#update-stackinstance-1 "Direct link to Update StackInstance") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: StackInstance metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: example-platform namespace: p-my-project resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: destination: virtualCluster: name: development displayName: Example application platform owner: user: admin prunePolicy: Retain templateRef: name: example-platform status: {} ``` Run the following curl command to update a single StackInstance `example-platform` in project `my-project`: ``` # Replace the example-platform in the url below with the name of the StackInstance you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/stackinstances/example-platform" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch StackInstance[​](#patch-stackinstance-1 "Direct link to Patch StackInstance") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the StackInstance `example-platform` in project `my-project` via a patch: ``` # Replace the example-platform in the url below with the name of the StackInstance you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/stackinstances/example-platform" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: StackInstance[​](#delete-stackinstance "Direct link to Delete: StackInstance") You can either use curl or kubectl to delete StackInstances. * kubectl * curl Run the following command to delete StackInstance `example-platform` in project `my-project`: ``` kubectl delete stackinstances.management.loft.sh example-platform -n p-my-project ``` Run the following curl command to delete StackInstance `example-platform` in project `my-project`: ``` # Replace the example-platform in the url below with the name of the StackInstance you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/stackinstances/example-platform" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Retrieve Stack Instance Outputs Retrieve the outputs a StackTemplate publishes for a StackInstance. Reading this subresource requires separate `get` permission on `stackinstances/outputs`. See [Stack permissions](https://vcluster.com/docs/platform/administer/users-permissions/permissions/stacks.md#permission-map) and [What are Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md) for the output-capture concept. ## StackInstanceOutputs example[​](#stackinstanceoutputs-example "Direct link to StackInstanceOutputs example") An example StackInstanceOutputs: ``` apiVersion: management.loft.sh/v1 kind: StackInstanceOutputs metadata: name: example-platform namespace: p-my-project outputs: - name: databaseAddress sensitive: false state: Available task: database value: 10.0.0.12 ``` ## StackInstanceOutputs reference[​](#stackinstanceoutputs-reference "Direct link to StackInstanceOutputs reference") ## `outputs` object\[] [​](#outputs "Direct link to outputs")Outputs are the stack's published outputs, in the order publishedOutputs declares them. Task outputs the stack does not publish are not listed here. ### `name` required string [​](#outputs-name "Direct link to outputs-name")Name is the public name the stack publishes the value under. ### `task` required string [​](#outputs-task "Direct link to outputs-task")Task is the task that produced the value. ### `sensitive` required boolean [​](#outputs-sensitive "Direct link to outputs-sensitive")Sensitive reflects the output's declared source type, not its content: a value read via fromSecret is sensitive, one read via fromResource is not. To have a value treated as sensitive, read it from a Secret via fromSecret. Always serialized. ### `state` required string [​](#outputs-state "Direct link to outputs-state")State reports whether the value can be read yet. ### `reason` string [​](#outputs-reason "Direct link to outputs-reason")Reason is the cause of the state in one word. An Available output carries one only when its task failed after the value was captured, meaning the value served is the last good one. ### `message` string [​](#outputs-message "Direct link to outputs-message")Message explains the reason in words. It names the task, the output or the failure behind it. ### `value` string [​](#outputs-value "Direct link to outputs-value")Value is the captured value. It is set when State is Available, including when the captured value is an empty string. ## StackInstanceOutputs (Get)[​](#stackinstanceoutputs-get "Direct link to StackInstanceOutputs (Get)") Retrieve the published outputs of a StackInstance. Values captured from Secrets are marked sensitive. * kubectl * curl Run the following command: ``` # Exchange example-platform in the url below with the name of the StackInstance kubectl get --raw "/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/stackinstances/example-platform/outputs" ``` Run the following curl command: ``` # Exchange example-platform in the url below with the name of the StackInstance curl -s -X GET --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/stackinstances/example-platform/outputs" \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Stack Template StackTemplate is a reusable, parameterized task graph that a StackInstance deploys to a tenant cluster or control plane cluster. See [What are Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md) for the concept and [Create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) for the full task, parameter, and output syntax. ## StackTemplate example[​](#stacktemplate-example "Direct link to StackTemplate example") An example StackTemplate: ``` apiVersion: management.loft.sh/v1 kind: StackTemplate metadata: name: example-platform spec: description: Deploys a database before the API that uses it. displayName: Example application platform tasks: - app: templateRef: name: postgresql name: database - app: templateRef: name: example-api dependsOn: - database name: api status: {} ``` ## StackTemplate reference[​](#stacktemplate-reference "Direct link to StackTemplate reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the catalog/UI name. ### `description` string [​](#spec-description "Direct link to spec-description")Description describes the stack template. ### `icon` string [​](#spec-icon "Direct link to spec-icon")Icon is a logo/png reference for the catalog. ### `parameters` object\[] [​](#spec-parameters "Direct link to spec-parameters")Parameters declares the typed parameters (with defaults) a stack accepts, reused from AppParameter. #### `variable` string [​](#spec-parameters-variable "Direct link to spec-parameters-variable")Variable is the path of the variable. Can be foo or foo.bar for nested objects. #### `label` string [​](#spec-parameters-label "Direct link to spec-parameters-label")Label is the label to show for this parameter #### `description` string [​](#spec-parameters-description "Direct link to spec-parameters-description")Description is the description to show for this parameter #### `type` string [​](#spec-parameters-type "Direct link to spec-parameters-type")Type of the parameter. Can be one of: string, multiline, boolean, number and password #### `options` string\[] [​](#spec-parameters-options "Direct link to spec-parameters-options")Options is a slice of strings, where each string represents a mutually exclusive choice. #### `min` integer [​](#spec-parameters-min "Direct link to spec-parameters-min")Min is the minimum number if type is number #### `max` integer [​](#spec-parameters-max "Direct link to spec-parameters-max")Max is the maximum number if type is number #### `required` boolean [​](#spec-parameters-required "Direct link to spec-parameters-required")Required specifies if this parameter is required #### `defaultValue` string [​](#spec-parameters-defaultValue "Direct link to spec-parameters-defaultValue")DefaultValue is the default value if none is specified #### `placeholder` string [​](#spec-parameters-placeholder "Direct link to spec-parameters-placeholder")Placeholder shown in the UI #### `invalidation` string [​](#spec-parameters-invalidation "Direct link to spec-parameters-invalidation")Invalidation regex that if matched will reject the input #### `validation` string [​](#spec-parameters-validation "Direct link to spec-parameters-validation")Validation regex that if matched will allow the input #### `section` string [​](#spec-parameters-section "Direct link to spec-parameters-section")Section where this app should be displayed. Apps with the same section name will be grouped together ### `tasks` object\[] [​](#spec-tasks "Direct link to spec-tasks")Tasks is the DAG blueprint. Each task follows the same task-level mutual-exclusivity rule (ArgoCDApplication XOR App), validated at admission. #### `name` required string [​](#spec-tasks-name "Direct link to spec-tasks-name")Name is the stable identifier of the task. DNS-label-safe, unique within the stack. A task that declares outputs may use letters and digits only: its outputs are referenced as {{ .Outputs.task.name }}, and the template syntax cannot address a name containing "-". #### `dependsOn` string\[] [​](#spec-tasks-dependsOn "Direct link to spec-tasks-dependsOn")DependsOn lists task names that must reach readiness before this task starts. #### `argoCDApplication` object [​](#spec-tasks-argoCDApplication "Direct link to spec-tasks-argoCDApplication")ArgoCDApplication defines this task as an Argo CD application. Exactly one of ArgoCDApplication or App must be set per task (validated at admission). ##### `template` object [​](#spec-tasks-argoCDApplication-template "Direct link to spec-tasks-argoCDApplication-template")Template is an inline ArgoCD application template definition. Exactly one of Template or TemplateRef must be set per task (validated at admission). ##### `metadata` object [​](#spec-tasks-argoCDApplication-template-metadata "Direct link to spec-tasks-argoCDApplication-template-metadata") ##### `labels` object [​](#spec-tasks-argoCDApplication-template-metadata-labels "Direct link to spec-tasks-argoCDApplication-template-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-tasks-argoCDApplication-template-metadata-annotations "Direct link to spec-tasks-argoCDApplication-template-metadata-annotations")Annotations are annotations on the object ##### `spec` object [​](#spec-tasks-argoCDApplication-template-spec "Direct link to spec-tasks-argoCDApplication-template-spec")Spec is the spec of the Argo CD application ##### `source` object [​](#spec-tasks-argoCDApplication-template-spec-source "Direct link to spec-tasks-argoCDApplication-template-spec-source") ##### `repoURL` required string [​](#spec-tasks-argoCDApplication-template-spec-source-repoURL "Direct link to spec-tasks-argoCDApplication-template-spec-source-repoURL") ##### `path` string [​](#spec-tasks-argoCDApplication-template-spec-source-path "Direct link to spec-tasks-argoCDApplication-template-spec-source-path") ##### `targetRevision` string [​](#spec-tasks-argoCDApplication-template-spec-source-targetRevision "Direct link to spec-tasks-argoCDApplication-template-spec-source-targetRevision") ##### `helm` object [​](#spec-tasks-argoCDApplication-template-spec-source-helm "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm") ##### `valueFiles` string\[] [​](#spec-tasks-argoCDApplication-template-spec-source-helm-valueFiles "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-valueFiles") ##### `parameters` object\[] [​](#spec-tasks-argoCDApplication-template-spec-source-helm-parameters "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-parameters") ##### `name` string [​](#spec-tasks-argoCDApplication-template-spec-source-helm-parameters-name "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-parameters-name") ##### `value` string [​](#spec-tasks-argoCDApplication-template-spec-source-helm-parameters-value "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-parameters-value") ##### `forceString` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-helm-parameters-forceString "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-parameters-forceString") ##### `releaseName` string [​](#spec-tasks-argoCDApplication-template-spec-source-helm-releaseName "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-releaseName") ##### `values` string [​](#spec-tasks-argoCDApplication-template-spec-source-helm-values "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-values") ##### `fileParameters` object\[] [​](#spec-tasks-argoCDApplication-template-spec-source-helm-fileParameters "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-fileParameters") ##### `name` string [​](#spec-tasks-argoCDApplication-template-spec-source-helm-fileParameters-name "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-fileParameters-name") ##### `path` string [​](#spec-tasks-argoCDApplication-template-spec-source-helm-fileParameters-path "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-fileParameters-path") ##### `version` string [​](#spec-tasks-argoCDApplication-template-spec-source-helm-version "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-version") ##### `passCredentials` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-helm-passCredentials "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-passCredentials") ##### `ignoreMissingValueFiles` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-helm-ignoreMissingValueFiles "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-ignoreMissingValueFiles") ##### `skipCrds` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-helm-skipCrds "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-skipCrds") ##### `valuesObject` object [​](#spec-tasks-argoCDApplication-template-spec-source-helm-valuesObject "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-valuesObject") ##### `namespace` string [​](#spec-tasks-argoCDApplication-template-spec-source-helm-namespace "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-namespace") ##### `kubeVersion` string [​](#spec-tasks-argoCDApplication-template-spec-source-helm-kubeVersion "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-kubeVersion") ##### `apiVersions` string\[] [​](#spec-tasks-argoCDApplication-template-spec-source-helm-apiVersions "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-apiVersions") ##### `skipTests` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-helm-skipTests "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-skipTests") ##### `skipSchemaValidation` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-helm-skipSchemaValidation "Direct link to spec-tasks-argoCDApplication-template-spec-source-helm-skipSchemaValidation") ##### `kustomize` object [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize") ##### `namePrefix` string [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-namePrefix "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-namePrefix") ##### `nameSuffix` string [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-nameSuffix "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-nameSuffix") ##### `images` object [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-images "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-images") ##### `commonLabels` object [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-commonLabels "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-commonLabels") ##### `version` string [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-version "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-version") ##### `commonAnnotations` object [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-commonAnnotations "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-commonAnnotations") ##### `forceCommonLabels` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-forceCommonLabels "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-forceCommonLabels") ##### `forceCommonAnnotations` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-forceCommonAnnotations "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-forceCommonAnnotations") ##### `namespace` string [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-namespace "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-namespace") ##### `commonAnnotationsEnvsubst` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-commonAnnotationsEnvsubst "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-commonAnnotationsEnvsubst") ##### `replicas` object [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-replicas "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-replicas") ##### `patches` object [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-patches "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-patches") ##### `components` string\[] [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-components "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-components") ##### `ignoreMissingComponents` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-ignoreMissingComponents "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-ignoreMissingComponents") ##### `labelWithoutSelector` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-labelWithoutSelector "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-labelWithoutSelector") ##### `kubeVersion` string [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-kubeVersion "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-kubeVersion") ##### `apiVersions` string\[] [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-apiVersions "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-apiVersions") ##### `labelIncludeTemplates` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-kustomize-labelIncludeTemplates "Direct link to spec-tasks-argoCDApplication-template-spec-source-kustomize-labelIncludeTemplates") ##### `directory` object [​](#spec-tasks-argoCDApplication-template-spec-source-directory "Direct link to spec-tasks-argoCDApplication-template-spec-source-directory") ##### `recurse` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-directory-recurse "Direct link to spec-tasks-argoCDApplication-template-spec-source-directory-recurse") ##### `jsonnet` object [​](#spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet "Direct link to spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet") ##### `extVars` object\[] [​](#spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars "Direct link to spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars") ##### `name` required string [​](#spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars-name "Direct link to spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars-name") ##### `value` required string [​](#spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars-value "Direct link to spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars-value") ##### `code` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars-code "Direct link to spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-extVars-code") ##### `tlas` object\[] [​](#spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas "Direct link to spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas") ##### `name` required string [​](#spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas-name "Direct link to spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas-name") ##### `value` required string [​](#spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas-value "Direct link to spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas-value") ##### `code` boolean [​](#spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas-code "Direct link to spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-tlas-code") ##### `libs` string\[] [​](#spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-libs "Direct link to spec-tasks-argoCDApplication-template-spec-source-directory-jsonnet-libs") ##### `exclude` string [​](#spec-tasks-argoCDApplication-template-spec-source-directory-exclude "Direct link to spec-tasks-argoCDApplication-template-spec-source-directory-exclude") ##### `include` string [​](#spec-tasks-argoCDApplication-template-spec-source-directory-include "Direct link to spec-tasks-argoCDApplication-template-spec-source-directory-include") ##### `plugin` object [​](#spec-tasks-argoCDApplication-template-spec-source-plugin "Direct link to spec-tasks-argoCDApplication-template-spec-source-plugin") ##### `name` string [​](#spec-tasks-argoCDApplication-template-spec-source-plugin-name "Direct link to spec-tasks-argoCDApplication-template-spec-source-plugin-name") ##### `env` object [​](#spec-tasks-argoCDApplication-template-spec-source-plugin-env "Direct link to spec-tasks-argoCDApplication-template-spec-source-plugin-env") ##### `parameters` object [​](#spec-tasks-argoCDApplication-template-spec-source-plugin-parameters "Direct link to spec-tasks-argoCDApplication-template-spec-source-plugin-parameters") ##### `chart` string [​](#spec-tasks-argoCDApplication-template-spec-source-chart "Direct link to spec-tasks-argoCDApplication-template-spec-source-chart") ##### `ref` string [​](#spec-tasks-argoCDApplication-template-spec-source-ref "Direct link to spec-tasks-argoCDApplication-template-spec-source-ref") ##### `name` string [​](#spec-tasks-argoCDApplication-template-spec-source-name "Direct link to spec-tasks-argoCDApplication-template-spec-source-name") ##### `destination` required object [​](#spec-tasks-argoCDApplication-template-spec-destination "Direct link to spec-tasks-argoCDApplication-template-spec-destination") ##### `server` string [​](#spec-tasks-argoCDApplication-template-spec-destination-server "Direct link to spec-tasks-argoCDApplication-template-spec-destination-server") ##### `namespace` string [​](#spec-tasks-argoCDApplication-template-spec-destination-namespace "Direct link to spec-tasks-argoCDApplication-template-spec-destination-namespace") ##### `name` string [​](#spec-tasks-argoCDApplication-template-spec-destination-name "Direct link to spec-tasks-argoCDApplication-template-spec-destination-name") ##### `project` required string [​](#spec-tasks-argoCDApplication-template-spec-project "Direct link to spec-tasks-argoCDApplication-template-spec-project") ##### `syncPolicy` object [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy") ##### `automated` object [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-automated "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-automated") ##### `prune` boolean [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-automated-prune "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-automated-prune") ##### `selfHeal` boolean [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-automated-selfHeal "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-automated-selfHeal") ##### `allowEmpty` boolean [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-automated-allowEmpty "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-automated-allowEmpty") ##### `enabled` boolean [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-automated-enabled "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-automated-enabled") ##### `syncOptions` object [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-syncOptions "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-syncOptions") ##### `retry` object [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-retry "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-retry") ##### `limit` integer [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-retry-limit "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-retry-limit") ##### `backoff` object [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff") ##### `duration` string [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff-duration "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff-duration") ##### `factor` integer [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff-factor "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff-factor") ##### `maxDuration` string [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff-maxDuration "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-retry-backoff-maxDuration") ##### `refresh` boolean [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-retry-refresh "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-retry-refresh") ##### `managedNamespaceMetadata` object [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-managedNamespaceMetadata "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-managedNamespaceMetadata") ##### `labels` object [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-managedNamespaceMetadata-labels "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-managedNamespaceMetadata-labels") ##### `annotations` object [​](#spec-tasks-argoCDApplication-template-spec-syncPolicy-managedNamespaceMetadata-annotations "Direct link to spec-tasks-argoCDApplication-template-spec-syncPolicy-managedNamespaceMetadata-annotations") ##### `ignoreDifferences` object [​](#spec-tasks-argoCDApplication-template-spec-ignoreDifferences "Direct link to spec-tasks-argoCDApplication-template-spec-ignoreDifferences") ##### `info` object\[] [​](#spec-tasks-argoCDApplication-template-spec-info "Direct link to spec-tasks-argoCDApplication-template-spec-info") ##### `name` required string [​](#spec-tasks-argoCDApplication-template-spec-info-name "Direct link to spec-tasks-argoCDApplication-template-spec-info-name") ##### `value` required string [​](#spec-tasks-argoCDApplication-template-spec-info-value "Direct link to spec-tasks-argoCDApplication-template-spec-info-value") ##### `revisionHistoryLimit` integer [​](#spec-tasks-argoCDApplication-template-spec-revisionHistoryLimit "Direct link to spec-tasks-argoCDApplication-template-spec-revisionHistoryLimit") ##### `sources` object [​](#spec-tasks-argoCDApplication-template-spec-sources "Direct link to spec-tasks-argoCDApplication-template-spec-sources") ##### `sourceHydrator` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator") ##### `drySource` required object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource") ##### `repoURL` required string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-repoURL "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-repoURL") ##### `targetRevision` required string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-targetRevision "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-targetRevision") ##### `path` required string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-path "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-path") ##### `helm` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm") ##### `valueFiles` string\[] [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-valueFiles "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-valueFiles") ##### `parameters` object\[] [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters") ##### `name` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters-name "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters-name") ##### `value` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters-value "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters-value") ##### `forceString` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters-forceString "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-parameters-forceString") ##### `releaseName` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-releaseName "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-releaseName") ##### `values` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-values "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-values") ##### `fileParameters` object\[] [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-fileParameters "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-fileParameters") ##### `name` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-fileParameters-name "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-fileParameters-name") ##### `path` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-fileParameters-path "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-fileParameters-path") ##### `version` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-version "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-version") ##### `passCredentials` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-passCredentials "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-passCredentials") ##### `ignoreMissingValueFiles` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-ignoreMissingValueFiles "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-ignoreMissingValueFiles") ##### `skipCrds` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-skipCrds "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-skipCrds") ##### `valuesObject` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-valuesObject "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-valuesObject") ##### `namespace` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-namespace "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-namespace") ##### `kubeVersion` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-kubeVersion "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-kubeVersion") ##### `apiVersions` string\[] [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-apiVersions "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-apiVersions") ##### `skipTests` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-skipTests "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-skipTests") ##### `skipSchemaValidation` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-skipSchemaValidation "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-helm-skipSchemaValidation") ##### `kustomize` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize") ##### `namePrefix` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-namePrefix "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-namePrefix") ##### `nameSuffix` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-nameSuffix "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-nameSuffix") ##### `images` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-images "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-images") ##### `commonLabels` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-commonLabels "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-commonLabels") ##### `version` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-version "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-version") ##### `commonAnnotations` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-commonAnnotations "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-commonAnnotations") ##### `forceCommonLabels` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-forceCommonLabels "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-forceCommonLabels") ##### `forceCommonAnnotations` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-forceCommonAnnotations "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-forceCommonAnnotations") ##### `namespace` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-namespace "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-namespace") ##### `commonAnnotationsEnvsubst` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-commonAnnotationsEnvsubst "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-commonAnnotationsEnvsubst") ##### `replicas` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-replicas "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-replicas") ##### `patches` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-patches "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-patches") ##### `components` string\[] [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-components "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-components") ##### `ignoreMissingComponents` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-ignoreMissingComponents "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-ignoreMissingComponents") ##### `labelWithoutSelector` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-labelWithoutSelector "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-labelWithoutSelector") ##### `kubeVersion` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-kubeVersion "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-kubeVersion") ##### `apiVersions` string\[] [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-apiVersions "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-apiVersions") ##### `labelIncludeTemplates` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-labelIncludeTemplates "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-kustomize-labelIncludeTemplates") ##### `directory` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory") ##### `recurse` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-recurse "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-recurse") ##### `jsonnet` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet") ##### `extVars` object\[] [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars") ##### `name` required string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-name "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-name") ##### `value` required string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-value "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-value") ##### `code` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-code "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-extVars-code") ##### `tlas` object\[] [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas") ##### `name` required string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-name "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-name") ##### `value` required string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-value "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-value") ##### `code` boolean [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-code "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-tlas-code") ##### `libs` string\[] [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-libs "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-jsonnet-libs") ##### `exclude` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-exclude "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-exclude") ##### `include` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-include "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-directory-include") ##### `plugin` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin") ##### `name` string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin-name "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin-name") ##### `env` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin-env "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin-env") ##### `parameters` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin-parameters "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-drySource-plugin-parameters") ##### `syncSource` required object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-syncSource "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-syncSource") ##### `targetBranch` required string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-syncSource-targetBranch "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-syncSource-targetBranch") ##### `path` required string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-syncSource-path "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-syncSource-path") ##### `hydrateTo` object [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-hydrateTo "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-hydrateTo") ##### `targetBranch` required string [​](#spec-tasks-argoCDApplication-template-spec-sourceHydrator-hydrateTo-targetBranch "Direct link to spec-tasks-argoCDApplication-template-spec-sourceHydrator-hydrateTo-targetBranch") ##### `templateRef` object [​](#spec-tasks-argoCDApplication-templateRef "Direct link to spec-tasks-argoCDApplication-templateRef")TemplateRef references an ArgoCDApplicationTemplate (per-application blueprint). Exactly one of Template or TemplateRef must be set per task (validated at admission). ##### `name` string [​](#spec-tasks-argoCDApplication-templateRef-name "Direct link to spec-tasks-argoCDApplication-templateRef-name")Name holds the name of the blueprint template to reference. ##### `parameters` object [​](#spec-tasks-argoCDApplication-parameters "Direct link to spec-tasks-argoCDApplication-parameters")Parameters are rendered as part of the task, then passed to the referenced ArgoCDApplicationTemplate. Only valid with TemplateRef; admission rejects the field on a task with an inline Template, whose values belong in the definition itself. #### `app` object [​](#spec-tasks-app "Direct link to spec-tasks-app")App defines this task as a platform App. Exactly one of ArgoCDApplication or App must be set per task (validated at admission). ##### `template` object [​](#spec-tasks-app-template "Direct link to spec-tasks-app-template")Template is an inline app definition. Exactly one of Template or TemplateRef must be set per task (validated at admission). ##### `metadata` object [​](#spec-tasks-app-template-metadata "Direct link to spec-tasks-app-template-metadata") ##### `labels` object [​](#spec-tasks-app-template-metadata-labels "Direct link to spec-tasks-app-template-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-tasks-app-template-metadata-annotations "Direct link to spec-tasks-app-template-metadata-annotations")Annotations are annotations on the object ##### `spec` object [​](#spec-tasks-app-template-spec "Direct link to spec-tasks-app-template-spec")Spec is the app definition of the child app instance. ##### `displayName` string [​](#spec-tasks-app-template-spec-displayName "Direct link to spec-tasks-app-template-spec-displayName")DisplayName is the name shown for the app instance in the UI. ##### `description` string [​](#spec-tasks-app-template-spec-description "Direct link to spec-tasks-app-template-spec-description")Description describes the app instance. ##### `releaseName` string [​](#spec-tasks-app-template-spec-releaseName "Direct link to spec-tasks-app-template-spec-releaseName")ReleaseName is the helm release name for the app instance. If empty, it defaults to the generated child name. ##### `defaultNamespace` string [​](#spec-tasks-app-template-spec-defaultNamespace "Direct link to spec-tasks-app-template-spec-defaultNamespace")DefaultNamespace is the default namespace this app should installed in. ##### `readme` string [​](#spec-tasks-app-template-spec-readme "Direct link to spec-tasks-app-template-spec-readme")Readme is a longer markdown string that describes the app. ##### `icon` string [​](#spec-tasks-app-template-spec-icon "Direct link to spec-tasks-app-template-spec-icon")Icon holds an URL to the app icon ##### `config` object [​](#spec-tasks-app-template-spec-config "Direct link to spec-tasks-app-template-spec-config")Config is the helm config to use to deploy the helm release ##### `chart` object [​](#spec-tasks-app-template-spec-config-chart "Direct link to spec-tasks-app-template-spec-config-chart")Chart holds information about a chart that should get deployed ##### `name` string [​](#spec-tasks-app-template-spec-config-chart-name "Direct link to spec-tasks-app-template-spec-config-chart-name")Name is the chart name in the repository ##### `version` string [​](#spec-tasks-app-template-spec-config-chart-version "Direct link to spec-tasks-app-template-spec-config-chart-version")Version is the chart version in the repository ##### `repoURL` string [​](#spec-tasks-app-template-spec-config-chart-repoURL "Direct link to spec-tasks-app-template-spec-config-chart-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#spec-tasks-app-template-spec-config-chart-username "Direct link to spec-tasks-app-template-spec-config-chart-username")The username that is required for this repository ##### `usernameRef` object [​](#spec-tasks-app-template-spec-config-chart-usernameRef "Direct link to spec-tasks-app-template-spec-config-chart-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#spec-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef "Direct link to spec-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef-project "Direct link to spec-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef-name "Direct link to spec-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef-key "Direct link to spec-tasks-app-template-spec-config-chart-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#spec-tasks-app-template-spec-config-chart-password "Direct link to spec-tasks-app-template-spec-config-chart-password")The password that is required for this repository ##### `passwordRef` object [​](#spec-tasks-app-template-spec-config-chart-passwordRef "Direct link to spec-tasks-app-template-spec-config-chart-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#spec-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef "Direct link to spec-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef-project "Direct link to spec-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef-name "Direct link to spec-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef-key "Direct link to spec-tasks-app-template-spec-config-chart-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#spec-tasks-app-template-spec-config-chart-insecureSkipTlsVerify "Direct link to spec-tasks-app-template-spec-config-chart-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `manifests` string [​](#spec-tasks-app-template-spec-config-manifests "Direct link to spec-tasks-app-template-spec-config-manifests")Manifests holds kube manifests that will be deployed as a chart ##### `bash` object [​](#spec-tasks-app-template-spec-config-bash "Direct link to spec-tasks-app-template-spec-config-bash")Bash holds the bash script to execute in a container in the target ##### `script` string [​](#spec-tasks-app-template-spec-config-bash-script "Direct link to spec-tasks-app-template-spec-config-bash-script")Script is the script to execute. ##### `image` string [​](#spec-tasks-app-template-spec-config-bash-image "Direct link to spec-tasks-app-template-spec-config-bash-image")Image is the image to use for this app ##### `clusterRole` string [​](#spec-tasks-app-template-spec-config-bash-clusterRole "Direct link to spec-tasks-app-template-spec-config-bash-clusterRole")ClusterRole is the cluster role to use for this job ##### `podSecurityContext` object [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext")PodSecurityContext for the bash pod. ##### `seLinuxOptions` object [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions")The SELinux context to be applied to all containers. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `user` string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-user "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-user")User is a SELinux user label that applies to the container. ##### `role` string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-role "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-role")Role is a SELinux role label that applies to the container. ##### `type` string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-type "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-type")Type is a SELinux type label that applies to the container. ##### `level` string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-level "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxOptions-level")Level is SELinux level label that applies to the container. ##### `windowsOptions` object [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions")The Windows specific settings applied to all containers. If unspecified, the options within a container's SecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. ##### `gmsaCredentialSpecName` string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpecName "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpecName")GMSACredentialSpecName is the name of the GMSA credential spec to use. ##### `gmsaCredentialSpec` string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpec "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-gmsaCredentialSpec")GMSACredentialSpec is where the GMSA admission webhook () inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. ##### `runAsUserName` string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-runAsUserName "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-runAsUserName")The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `hostProcess` boolean [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-hostProcess "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-windowsOptions-hostProcess")HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. ##### `runAsUser` integer [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-runAsUser "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-runAsUser")The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `runAsGroup` integer [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-runAsGroup "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-runAsGroup")The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows. ##### `runAsNonRoot` boolean [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-runAsNonRoot "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-runAsNonRoot")Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `supplementalGroups` integer\[] [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-supplementalGroups "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-supplementalGroups")A list of groups applied to the first process run in each container, in addition to the container's primary GID and fsGroup (if specified). If the SupplementalGroupsPolicy feature is enabled, the supplementalGroupsPolicy field determines whether these are in addition to or instead of any group memberships defined in the container image. If unspecified, no additional groups are added, though group memberships defined in the container image may still be used, depending on the supplementalGroupsPolicy field. Note that this field cannot be set when spec.os.name is windows. ##### `supplementalGroupsPolicy` string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-supplementalGroupsPolicy "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-supplementalGroupsPolicy")Defines how supplemental groups of the first container processes are calculated. Valid values are "Merge" and "Strict". If not specified, "Merge" is used. (Alpha) Using the field requires the SupplementalGroupsPolicy feature gate to be enabled and the container runtime must implement support for this feature. Note that this field cannot be set when spec.os.name is windows. TODO: update the default value to "Merge" when spec.os.name is not windows in v1.34 ##### `fsGroup` integer [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-fsGroup "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-fsGroup")A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod:1) The owning GID will be the FSGroup 2) The setgid bit is set (new files created in the volume will be owned by FSGroup) 3) The permission bits are OR'd with rw-rw----If unset, the Kubelet will not modify the ownership and permissions of any volume. Note that this field cannot be set when spec.os.name is windows. ##### `sysctls` object\[] [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-sysctls "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-sysctls")Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported sysctls (by the container runtime) might fail to launch. Note that this field cannot be set when spec.os.name is windows. ##### `name` required string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-sysctls-name "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-sysctls-name")Name of a property to set ##### `value` required string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-sysctls-value "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-sysctls-value")Value of a property to set ##### `fsGroupChangePolicy` string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-fsGroupChangePolicy "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-fsGroupChangePolicy")fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. Note that this field cannot be set when spec.os.name is windows. ##### `seccompProfile` object [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-seccompProfile "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-seccompProfile")The seccomp options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-seccompProfile-type "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-seccompProfile-type")type indicates which kind of seccomp profile will be applied. Valid options are:Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. ##### `localhostProfile` string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-seccompProfile-localhostProfile "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-seccompProfile-localhostProfile")localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. ##### `appArmorProfile` object [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-appArmorProfile "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-appArmorProfile")appArmorProfile is the AppArmor options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-appArmorProfile-type "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-appArmorProfile-type")type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. ##### `localhostProfile` string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-appArmorProfile-localhostProfile "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-appArmorProfile-localhostProfile")localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". ##### `seLinuxChangePolicy` string [​](#spec-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxChangePolicy "Direct link to spec-tasks-app-template-spec-config-bash-podSecurityContext-seLinuxChangePolicy")seLinuxChangePolicy defines how the container's SELinux label is applied to all volumes used by the Pod. It has no effect on nodes that do not support SELinux or to volumes does not support SELinux. Valid values are "MountOption" and "Recursive"."Recursive" means relabeling of all files on all Pod volumes by the container runtime. This may be slow for large volumes, but allows mixing privileged and unprivileged Pods sharing the same volume on the same node."MountOption" mounts all eligible Pod volumes with `-o context` mount option. This requires all Pods that share the same volume to use the same SELinux label. It is not possible to share the same volume among privileged and unprivileged Pods. Eligible volumes are in-tree FibreChannel and iSCSI volumes, and all CSI volumes whose CSI driver announces SELinux support by setting spec.seLinuxMount: true in their CSIDriver instance. Other volumes are always re-labelled recursively. "MountOption" value is allowed only when SELinuxMount feature gate is enabled.If not specified and SELinuxMount feature gate is enabled, "MountOption" is used. If not specified and SELinuxMount feature gate is disabled, "MountOption" is used for ReadWriteOncePod volumes and "Recursive" for all other volumes.This field affects only Pods that have SELinux label set, either in PodSecurityContext or in SecurityContext of all containers.All Pods that use the same volume should use the same seLinuxChangePolicy, otherwise some pods can get stuck in ContainerCreating state. Note that this field cannot be set when spec.os.name is windows. ##### `securityContext` object [​](#spec-tasks-app-template-spec-config-bash-securityContext "Direct link to spec-tasks-app-template-spec-config-bash-securityContext")SecurityContext for the bash container. ##### `capabilities` object [​](#spec-tasks-app-template-spec-config-bash-securityContext-capabilities "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-capabilities")The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container runtime. Note that this field cannot be set when spec.os.name is windows. ##### `add` string\[] [​](#spec-tasks-app-template-spec-config-bash-securityContext-capabilities-add "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-capabilities-add")Added capabilities ##### `drop` string\[] [​](#spec-tasks-app-template-spec-config-bash-securityContext-capabilities-drop "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-capabilities-drop")Removed capabilities ##### `privileged` boolean [​](#spec-tasks-app-template-spec-config-bash-securityContext-privileged "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-privileged")Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false. Note that this field cannot be set when spec.os.name is windows. ##### `seLinuxOptions` object [​](#spec-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions")The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `user` string [​](#spec-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-user "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-user")User is a SELinux user label that applies to the container. ##### `role` string [​](#spec-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-role "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-role")Role is a SELinux role label that applies to the container. ##### `type` string [​](#spec-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-type "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-type")Type is a SELinux type label that applies to the container. ##### `level` string [​](#spec-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-level "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-seLinuxOptions-level")Level is SELinux level label that applies to the container. ##### `windowsOptions` object [​](#spec-tasks-app-template-spec-config-bash-securityContext-windowsOptions "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-windowsOptions")The Windows specific settings applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux. ##### `gmsaCredentialSpecName` string [​](#spec-tasks-app-template-spec-config-bash-securityContext-windowsOptions-gmsaCredentialSpecName "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-windowsOptions-gmsaCredentialSpecName")GMSACredentialSpecName is the name of the GMSA credential spec to use. ##### `gmsaCredentialSpec` string [​](#spec-tasks-app-template-spec-config-bash-securityContext-windowsOptions-gmsaCredentialSpec "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-windowsOptions-gmsaCredentialSpec")GMSACredentialSpec is where the GMSA admission webhook () inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field. ##### `runAsUserName` string [​](#spec-tasks-app-template-spec-config-bash-securityContext-windowsOptions-runAsUserName "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-windowsOptions-runAsUserName")The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `hostProcess` boolean [​](#spec-tasks-app-template-spec-config-bash-securityContext-windowsOptions-hostProcess "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-windowsOptions-hostProcess")HostProcess determines if a container should be run as a 'Host Process' container. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true. ##### `runAsUser` integer [​](#spec-tasks-app-template-spec-config-bash-securityContext-runAsUser "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-runAsUser")The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `runAsGroup` integer [​](#spec-tasks-app-template-spec-config-bash-securityContext-runAsGroup "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-runAsGroup")The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows. ##### `runAsNonRoot` boolean [​](#spec-tasks-app-template-spec-config-bash-securityContext-runAsNonRoot "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-runAsNonRoot")Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. ##### `readOnlyRootFilesystem` boolean [​](#spec-tasks-app-template-spec-config-bash-securityContext-readOnlyRootFilesystem "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-readOnlyRootFilesystem")Whether this container has a read-only root filesystem. Default is false. Note that this field cannot be set when spec.os.name is windows. ##### `allowPrivilegeEscalation` boolean [​](#spec-tasks-app-template-spec-config-bash-securityContext-allowPrivilegeEscalation "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-allowPrivilegeEscalation")AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no\_new\_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is:1) run as Privileged 2) has CAP\_SYS\_ADMIN Note that this field cannot be set when spec.os.name is windows. ##### `procMount` string [​](#spec-tasks-app-template-spec-config-bash-securityContext-procMount "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-procMount")procMount denotes the type of proc mount to use for the containers. The default value is Default which uses the container runtime defaults for readonly paths and masked paths. Note that this field cannot be set when spec.os.name is windows. ##### `seccompProfile` object [​](#spec-tasks-app-template-spec-config-bash-securityContext-seccompProfile "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-seccompProfile")The seccomp options to use by this container. If seccomp options are provided at both the pod & container level, the container options override the pod options. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-tasks-app-template-spec-config-bash-securityContext-seccompProfile-type "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-seccompProfile-type")type indicates which kind of seccomp profile will be applied. Valid options are:Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied. ##### `localhostProfile` string [​](#spec-tasks-app-template-spec-config-bash-securityContext-seccompProfile-localhostProfile "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-seccompProfile-localhostProfile")localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must be set if type is "Localhost". Must NOT be set for any other type. ##### `appArmorProfile` object [​](#spec-tasks-app-template-spec-config-bash-securityContext-appArmorProfile "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-appArmorProfile")appArmorProfile is the AppArmor options to use by this container. If set, this profile overrides the pod's appArmorProfile. Note that this field cannot be set when spec.os.name is windows. ##### `type` required string [​](#spec-tasks-app-template-spec-config-bash-securityContext-appArmorProfile-type "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-appArmorProfile-type")type indicates which kind of AppArmor profile will be applied. Valid options are: Localhost - a profile pre-loaded on the node. RuntimeDefault - the container runtime's default profile. Unconfined - no AppArmor enforcement. ##### `localhostProfile` string [​](#spec-tasks-app-template-spec-config-bash-securityContext-appArmorProfile-localhostProfile "Direct link to spec-tasks-app-template-spec-config-bash-securityContext-appArmorProfile-localhostProfile")localhostProfile indicates a profile loaded on the node that should be used. The profile must be preconfigured on the node to work. Must match the loaded name of the profile. Must be set if and only if type is "Localhost". ##### `values` string [​](#spec-tasks-app-template-spec-config-values "Direct link to spec-tasks-app-template-spec-config-values")Values is the set of extra Values added to the chart. These values merge with the default values inside of the chart. You can use golang templating in here with values from parameters. ##### `parameters` string [​](#spec-tasks-app-template-spec-config-parameters "Direct link to spec-tasks-app-template-spec-config-parameters")Parameters are additional helm chart values that will get merged with config and are then used to deploy the helm chart. ##### `annotations` object [​](#spec-tasks-app-template-spec-config-annotations "Direct link to spec-tasks-app-template-spec-config-annotations")Annotations are extra annotations for this helm release ##### `wait` boolean [​](#spec-tasks-app-template-spec-wait "Direct link to spec-tasks-app-template-spec-wait")Wait determines if Loft should wait during deploy for the app to become ready ##### `timeout` string [​](#spec-tasks-app-template-spec-timeout "Direct link to spec-tasks-app-template-spec-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) ##### `parameters` object\[] [​](#spec-tasks-app-template-spec-parameters "Direct link to spec-tasks-app-template-spec-parameters")Parameters define additional app parameters that will set helm values ##### `variable` string [​](#spec-tasks-app-template-spec-parameters-variable "Direct link to spec-tasks-app-template-spec-parameters-variable")Variable is the path of the variable. Can be foo or foo.bar for nested objects. ##### `label` string [​](#spec-tasks-app-template-spec-parameters-label "Direct link to spec-tasks-app-template-spec-parameters-label")Label is the label to show for this parameter ##### `description` string [​](#spec-tasks-app-template-spec-parameters-description "Direct link to spec-tasks-app-template-spec-parameters-description")Description is the description to show for this parameter ##### `type` string [​](#spec-tasks-app-template-spec-parameters-type "Direct link to spec-tasks-app-template-spec-parameters-type")Type of the parameter. Can be one of: string, multiline, boolean, number and password ##### `options` string\[] [​](#spec-tasks-app-template-spec-parameters-options "Direct link to spec-tasks-app-template-spec-parameters-options")Options is a slice of strings, where each string represents a mutually exclusive choice. ##### `min` integer [​](#spec-tasks-app-template-spec-parameters-min "Direct link to spec-tasks-app-template-spec-parameters-min")Min is the minimum number if type is number ##### `max` integer [​](#spec-tasks-app-template-spec-parameters-max "Direct link to spec-tasks-app-template-spec-parameters-max")Max is the maximum number if type is number ##### `required` boolean [​](#spec-tasks-app-template-spec-parameters-required "Direct link to spec-tasks-app-template-spec-parameters-required")Required specifies if this parameter is required ##### `defaultValue` string [​](#spec-tasks-app-template-spec-parameters-defaultValue "Direct link to spec-tasks-app-template-spec-parameters-defaultValue")DefaultValue is the default value if none is specified ##### `placeholder` string [​](#spec-tasks-app-template-spec-parameters-placeholder "Direct link to spec-tasks-app-template-spec-parameters-placeholder")Placeholder shown in the UI ##### `invalidation` string [​](#spec-tasks-app-template-spec-parameters-invalidation "Direct link to spec-tasks-app-template-spec-parameters-invalidation")Invalidation regex that if matched will reject the input ##### `validation` string [​](#spec-tasks-app-template-spec-parameters-validation "Direct link to spec-tasks-app-template-spec-parameters-validation")Validation regex that if matched will allow the input ##### `section` string [​](#spec-tasks-app-template-spec-parameters-section "Direct link to spec-tasks-app-template-spec-parameters-section")Section where this app should be displayed. Apps with the same section name will be grouped together ##### `streamContainer` object [​](#spec-tasks-app-template-spec-streamContainer "Direct link to spec-tasks-app-template-spec-streamContainer")DEPRECATED: Use config.bash instead StreamContainer can be used to stream a containers logs instead of the helm output. ##### `selector` object [​](#spec-tasks-app-template-spec-streamContainer-selector "Direct link to spec-tasks-app-template-spec-streamContainer-selector")Label selector for pods. The newest matching pod will be used to stream logs from ##### `matchLabels` object [​](#spec-tasks-app-template-spec-streamContainer-selector-matchLabels "Direct link to spec-tasks-app-template-spec-streamContainer-selector-matchLabels")matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. ##### `matchExpressions` object\[] [​](#spec-tasks-app-template-spec-streamContainer-selector-matchExpressions "Direct link to spec-tasks-app-template-spec-streamContainer-selector-matchExpressions")matchExpressions is a list of label selector requirements. The requirements are ANDed. ##### `key` required string [​](#spec-tasks-app-template-spec-streamContainer-selector-matchExpressions-key "Direct link to spec-tasks-app-template-spec-streamContainer-selector-matchExpressions-key")key is the label key that the selector applies to. ##### `operator` required string [​](#spec-tasks-app-template-spec-streamContainer-selector-matchExpressions-operator "Direct link to spec-tasks-app-template-spec-streamContainer-selector-matchExpressions-operator")operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. ##### `values` string\[] [​](#spec-tasks-app-template-spec-streamContainer-selector-matchExpressions-values "Direct link to spec-tasks-app-template-spec-streamContainer-selector-matchExpressions-values")values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. ##### `container` string [​](#spec-tasks-app-template-spec-streamContainer-container "Direct link to spec-tasks-app-template-spec-streamContainer-container")Container is the container name to use ##### `templateRef` object [​](#spec-tasks-app-templateRef "Direct link to spec-tasks-app-templateRef")TemplateRef references a named app by name and version. Exactly one of Template or TemplateRef must be set per task (validated at admission). ##### `name` string [​](#spec-tasks-app-templateRef-name "Direct link to spec-tasks-app-templateRef-name")Name holds the name of the app to reference. ##### `version` string [​](#spec-tasks-app-templateRef-version "Direct link to spec-tasks-app-templateRef-version")Version of the app to deploy. If empty, the latest version is deployed. ##### `parameters` object [​](#spec-tasks-app-parameters "Direct link to spec-tasks-app-parameters")Parameters are passed to the referenced app. Only valid with TemplateRef; admission rejects the field on a task with an inline Template, whose values belong in the definition itself. #### `timeout` object [​](#spec-tasks-timeout "Direct link to spec-tasks-timeout")Timeout overrides Defaults.TaskTimeout for this task. ##### `Duration` required object [​](#spec-tasks-timeout-Duration "Direct link to spec-tasks-timeout-Duration") #### `outputs` object\[] [​](#spec-tasks-outputs "Direct link to spec-tasks-outputs")Outputs declares named values this task publishes once it is Healthy. Later tasks consume them in their specs as {{ .Outputs.task.name }} and must list this task in dependsOn (validated at admission). ##### `name` required string [​](#spec-tasks-outputs-name "Direct link to spec-tasks-outputs-name")Name identifies the output. Letters and digits only, unique within the task: the output is referenced as {{ .Outputs.task.name }}, and the template syntax cannot address a name containing "-". ##### `fromSecret` object [​](#spec-tasks-outputs-fromSecret "Direct link to spec-tasks-outputs-fromSecret")FromSecret reads the value from a Secret key on the destination cluster. Exactly one of fromSecret or fromResource must be set per output. ##### `namespace` required string [​](#spec-tasks-outputs-fromSecret-namespace "Direct link to spec-tasks-outputs-fromSecret-namespace")Namespace of the Secret. Must be a namespace this stack deploys into. ##### `name` required string [​](#spec-tasks-outputs-fromSecret-name "Direct link to spec-tasks-outputs-fromSecret-name")Name of the Secret. ##### `key` required string [​](#spec-tasks-outputs-fromSecret-key "Direct link to spec-tasks-outputs-fromSecret-key")Key inside the Secret's data. ##### `fromResource` object [​](#spec-tasks-outputs-fromResource "Direct link to spec-tasks-outputs-fromResource")FromResource reads a single field from a resource on the destination cluster. Exactly one of fromSecret or fromResource must be set per output. ##### `apiVersion` required string [​](#spec-tasks-outputs-fromResource-apiVersion "Direct link to spec-tasks-outputs-fromResource-apiVersion")APIVersion of the resource, e.g. "v1" or "apps/v1". ##### `kind` required string [​](#spec-tasks-outputs-fromResource-kind "Direct link to spec-tasks-outputs-fromResource-kind")Kind of the resource, e.g. "Service". ##### `namespace` required string [​](#spec-tasks-outputs-fromResource-namespace "Direct link to spec-tasks-outputs-fromResource-namespace")Namespace of the resource. Must be a namespace this stack deploys into; cluster-scoped resources cannot be read. ##### `name` required string [​](#spec-tasks-outputs-fromResource-name "Direct link to spec-tasks-outputs-fromResource-name")Name of the resource. ##### `jsonPath` required string [​](#spec-tasks-outputs-fromResource-jsonPath "Direct link to spec-tasks-outputs-fromResource-jsonPath")JSONPath selects the field, in the kubectl template syntax, e.g. "{.spec.clusterIP}". It must select exactly one scalar value. ### `publishedOutputs` object\[] [​](#spec-publishedOutputs "Direct link to spec-publishedOutputs")PublishedOutputs selects which captured task outputs the stack exposes to its users, optionally under a different name. The values come from the captured task outputs; nothing extra is read from the cluster. They are read through the instance's outputs subresource, never from its status. #### `name` required string [​](#spec-publishedOutputs-name "Direct link to spec-publishedOutputs-name")Name is the public name of the output. Unique within the stack; renaming the task output is allowed and expected. #### `fromTask` required object [​](#spec-publishedOutputs-fromTask "Direct link to spec-publishedOutputs-fromTask")FromTask selects which captured task output to publish. It must name an existing task and one of its declared outputs (validated at admission). ##### `task` required string [​](#spec-publishedOutputs-fromTask-task "Direct link to spec-publishedOutputs-fromTask-task")Task is the task name. ##### `output` required string [​](#spec-publishedOutputs-fromTask-output "Direct link to spec-publishedOutputs-fromTask-output")Output is the output name declared on that task. ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ## `status` object [​](#status "Direct link to status") ## Retrieve: StackTemplates[​](#retrieve-stacktemplates "Direct link to Retrieve: StackTemplates") You can either use curl or kubectl to retrieve StackTemplates. * kubectl * curl ### Retrieve a list of StackTemplates[​](#retrieve-a-list-of-stacktemplates "Direct link to Retrieve a list of StackTemplates") Run the following command to list all StackTemplates: ``` kubectl get stacktemplates.management.loft.sh -o yaml ``` ### Retrieve a single StackTemplate by name[​](#retrieve-a-single-stacktemplate-by-name "Direct link to Retrieve a single StackTemplate by name") Run the following kubectl command to get StackTemplate `example-platform`: ``` kubectl get stacktemplates.management.loft.sh example-platform -o yaml ``` ### Retrieve a list of StackTemplates[​](#retrieve-a-list-of-stacktemplates-1 "Direct link to Retrieve a list of StackTemplates") Run the following curl command to list all StackTemplates: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/stacktemplates" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single StackTemplate by name[​](#get-a-single-stacktemplate-by-name "Direct link to Get a single StackTemplate by name") Run the following curl command to get StackTemplate `example-platform`: ``` # Exchange example-platform in the url below with the name of the StackTemplate curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/stacktemplates/example-platform" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: StackTemplate[​](#create-stacktemplate "Direct link to Create: StackTemplate") You can either use curl or kubectl to create a new StackTemplate. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: StackTemplate metadata: name: example-platform spec: description: Deploys a database before the API that uses it. displayName: Example application platform tasks: - app: templateRef: name: postgresql name: database - app: templateRef: name: example-api dependsOn: - database name: api status: {} ``` Then create the StackTemplate `example-platform` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: StackTemplate metadata: name: example-platform spec: description: Deploys a database before the API that uses it. displayName: Example application platform tasks: - app: templateRef: name: postgresql name: database - app: templateRef: name: example-api dependsOn: - database name: api status: {} ``` Run the following curl command to create a new StackTemplate `example-platform`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/stacktemplates" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: StackTemplate[​](#update-stacktemplate "Direct link to Update: StackTemplate") You can either use curl or kubectl to update StackTemplates. * kubectl * curl ### Update StackTemplate[​](#update-stacktemplate "Direct link to Update StackTemplate") Run the following command to update StackTemplate `example-platform`: ``` kubectl edit stacktemplates.management.loft.sh example-platform ``` Then edit the object and upon save, kubectl will update the resource. ### Patch StackTemplate[​](#patch-stacktemplate "Direct link to Patch StackTemplate") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the StackTemplate `example-platform` via a patch: ``` kubectl patch stacktemplates.management.loft.sh example-platform \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update StackTemplate[​](#update-stacktemplate-1 "Direct link to Update StackTemplate") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: StackTemplate metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: example-platform resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: description: Deploys a database before the API that uses it. displayName: Example application platform tasks: - app: templateRef: name: postgresql name: database - app: templateRef: name: example-api dependsOn: - database name: api status: {} ``` Run the following curl command to update a single StackTemplate `example-platform`: ``` # Replace the example-platform in the url below with the name of the StackTemplate you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/stacktemplates/example-platform" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch StackTemplate[​](#patch-stacktemplate-1 "Direct link to Patch StackTemplate") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the StackTemplate `example-platform` via a patch: ``` # Replace the example-platform in the url below with the name of the StackTemplate you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/stacktemplates/example-platform" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: StackTemplate[​](#delete-stacktemplate "Direct link to Delete: StackTemplate") You can either use curl or kubectl to delete StackTemplates. * kubectl * curl Run the following command to delete StackTemplate `example-platform`: ``` kubectl delete stacktemplates.management.loft.sh example-platform ``` Run the following curl command to delete StackTemplate `example-platform`: ``` # Replace the example-platform in the url below with the name of the StackTemplate you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/stacktemplates/example-platform" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Team Teams are composed of multiple users and define a way to manage cluster access or other objects for multiple users at once. You can assign users automatically to teams by their groups, which can be synced from an authentication provider. Teams can also access the platform through their own access keys and own spaces or other objects. ## Platform-specific verbs[​](#platform-specific-verbs "Direct link to Platform-specific verbs") In addition to standard Kubernetes RBAC verbs (`get`, `list`, `create`, `update`, `patch`, `delete`), the Team resource supports these platform-specific verbs for access control: | Verb | Description | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `bind` | Allows binding permissions to a team, enabling assignment of access rules and roles to that team. Required when adding a team to ClusterAccess rules. | | `makeowner` | Allows transferring ownership of platform resources to a team. Used when changing the owner of projects, clusters, or other resources to a team. | ### Example access rule with bind verb[​](#example-access-rule-with-bind-verb "Direct link to Example access rule with bind verb") ``` apiVersion: management.loft.sh/v1 kind: Team metadata: name: platform-admins spec: access: - name: team-management verbs: - get - update - bind users: - admin ``` ## Team example[​](#team-example "Direct link to Team example") An example Team: ``` apiVersion: management.loft.sh/v1 kind: Team metadata: name: my-team spec: clusterRoles: - name: loft-management-admin description: All users in this team have full admin access to all clusters displayName: Global Admins groups: - loft:admins username: loftadmins status: {} ``` ## Team reference[​](#team-reference "Direct link to Team reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")The display name shown in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes a cluster access object ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `username` string [​](#spec-username "Direct link to spec-username")The username of the team that will be used for identification and docker registry namespace ### `users` string\[] [​](#spec-users "Direct link to spec-users")The loft users that belong to a team ### `groups` string\[] [​](#spec-groups "Direct link to spec-groups")The groups defined in a token that belong to a team ### `imagePullSecrets` object\[] [​](#spec-imagePullSecrets "Direct link to spec-imagePullSecrets")ImagePullSecrets holds secret references to image pull secrets the team has access to. #### `apiGroup` string [​](#spec-imagePullSecrets-apiGroup "Direct link to spec-imagePullSecrets-apiGroup")APIGroup is the api group of the secret #### `kind` string [​](#spec-imagePullSecrets-kind "Direct link to spec-imagePullSecrets-kind")Kind is the kind of the secret #### `secretName` string [​](#spec-imagePullSecrets-secretName "Direct link to spec-imagePullSecrets-secretName") #### `secretNamespace` string [​](#spec-imagePullSecrets-secretNamespace "Direct link to spec-imagePullSecrets-secretNamespace") #### `key` string [​](#spec-imagePullSecrets-key "Direct link to spec-imagePullSecrets-key") ### `clusterRoles` object\[] [​](#spec-clusterRoles "Direct link to spec-clusterRoles")ClusterRoles define the cluster roles that the users should have assigned in the cluster. #### `name` string [​](#spec-clusterRoles-name "Direct link to spec-clusterRoles-name")Name is the cluster role to assign ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ## `status` object [​](#status "Direct link to status") ## Retrieve: Teams[​](#retrieve-teams "Direct link to Retrieve: Teams") You can either use curl or kubectl to retrieve Teams. * kubectl * curl ### Retrieve a list of Teams[​](#retrieve-a-list-of-teams "Direct link to Retrieve a list of Teams") Run the following command to list all Teams: ``` kubectl get teams.management.loft.sh -o yaml ``` ### Retrieve a single Team by name[​](#retrieve-a-single-team-by-name "Direct link to Retrieve a single Team by name") Run the following kubectl command to get Team `my-team`: ``` kubectl get teams.management.loft.sh my-team -o yaml ``` ### Retrieve a list of Teams[​](#retrieve-a-list-of-teams-1 "Direct link to Retrieve a list of Teams") Run the following curl command to list all Teams: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/teams" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single Team by name[​](#get-a-single-team-by-name "Direct link to Get a single Team by name") Run the following curl command to get Team `my-team`: ``` # Exchange my-team in the url below with the name of the Team curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/teams/my-team" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: Team[​](#create-team "Direct link to Create: Team") You can either use curl or kubectl to create a new Team. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: Team metadata: name: my-team spec: clusterRoles: - name: loft-management-admin description: All users in this team have full admin access to all clusters displayName: Global Admins groups: - loft:admins username: loftadmins status: {} ``` Then create the Team `my-team` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: Team metadata: name: my-team spec: clusterRoles: - name: loft-management-admin description: All users in this team have full admin access to all clusters displayName: Global Admins groups: - loft:admins username: loftadmins status: {} ``` Run the following curl command to create a new Team `my-team`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/teams" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: Team[​](#update-team "Direct link to Update: Team") You can either use curl or kubectl to update Teams. * kubectl * curl ### Update Team[​](#update-team "Direct link to Update Team") Run the following command to update Team `my-team`: ``` kubectl edit teams.management.loft.sh my-team ``` Then edit the object and upon save, kubectl will update the resource. ### Patch Team[​](#patch-team "Direct link to Patch Team") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the Team `my-team` via a patch: ``` kubectl patch teams.management.loft.sh my-team \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update Team[​](#update-team-1 "Direct link to Update Team") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: Team metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-team resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: clusterRoles: - name: loft-management-admin description: All users in this team have full admin access to all clusters displayName: Global Admins groups: - loft:admins username: loftadmins status: {} ``` Run the following curl command to update a single Team `my-team`: ``` # Replace the my-team in the url below with the name of the Team you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/teams/my-team" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch Team[​](#patch-team-1 "Direct link to Patch Team") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the Team `my-team` via a patch: ``` # Replace the my-team in the url below with the name of the Team you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/teams/my-team" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: Team[​](#delete-team "Direct link to Delete: Team") You can either use curl or kubectl to delete Teams. * kubectl * curl Run the following command to delete Team `my-team`: ``` kubectl delete teams.management.loft.sh my-team ``` Run the following curl command to delete Team `my-team`: ``` # Replace the my-team in the url below with the name of the Team you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/teams/my-team" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # User Users that can access the platform. ## Platform-specific verbs[​](#platform-specific-verbs "Direct link to Platform-specific verbs") In addition to standard Kubernetes RBAC verbs (`get`, `list`, `create`, `update`, `patch`, `delete`), the User resource supports these platform-specific verbs for access control: | Verb | Description | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `bind` | Required when adding a user to ClusterAccess rules or other access rules. Allows granting the user permissions on platform resources. | | `makeowner` | Required when changing the owner of a resource to a user. Allows transferring ownership of projects, tenant clusters, spaces, or other platform resources. | ### Example access rule with makeowner verb[​](#example-access-rule-with-makeowner-verb "Direct link to Example access rule with makeowner verb") ``` apiVersion: management.loft.sh/v1 kind: User metadata: name: admin-user spec: access: - name: user-management verbs: - get - update - makeowner users: - admin-user ``` ## User example[​](#user-example "Direct link to User example") An example User: ``` apiVersion: management.loft.sh/v1 kind: User metadata: name: my-user spec: clusterRoles: - name: loft-management-admin displayName: My User email: myuser@test.com subject: my-user username: myuser status: {} ``` ## User reference[​](#user-reference "Direct link to User reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")The display name shown in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes a cluster access object ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `username` string [​](#spec-username "Direct link to spec-username")The username that is used to login ### `icon` string [​](#spec-icon "Direct link to spec-icon")The URL to an icon that should be shown for the user ### `email` string [​](#spec-email "Direct link to spec-email")The users email address ### `subject` string [​](#spec-subject "Direct link to spec-subject")The user subject as presented by the token ### `groups` string\[] [​](#spec-groups "Direct link to spec-groups")The groups the user has access to ### `ssoGroups` string\[] [​](#spec-ssoGroups "Direct link to spec-ssoGroups")SSOGroups is used to remember groups that were added from sso. ### `passwordRef` object [​](#spec-passwordRef "Direct link to spec-passwordRef")A reference to the user password #### `secretName` string [​](#spec-passwordRef-secretName "Direct link to spec-passwordRef-secretName") #### `secretNamespace` string [​](#spec-passwordRef-secretNamespace "Direct link to spec-passwordRef-secretNamespace") #### `key` string [​](#spec-passwordRef-key "Direct link to spec-passwordRef-key") ### `codesRef` object [​](#spec-codesRef "Direct link to spec-codesRef")A reference to the users access keys #### `secretName` string [​](#spec-codesRef-secretName "Direct link to spec-codesRef-secretName") #### `secretNamespace` string [​](#spec-codesRef-secretNamespace "Direct link to spec-codesRef-secretNamespace") #### `key` string [​](#spec-codesRef-key "Direct link to spec-codesRef-key") ### `imagePullSecrets` object\[] [​](#spec-imagePullSecrets "Direct link to spec-imagePullSecrets")ImagePullSecrets holds secret references to image pull secrets the user has access to. #### `apiGroup` string [​](#spec-imagePullSecrets-apiGroup "Direct link to spec-imagePullSecrets-apiGroup")APIGroup is the api group of the secret #### `kind` string [​](#spec-imagePullSecrets-kind "Direct link to spec-imagePullSecrets-kind")Kind is the kind of the secret #### `secretName` string [​](#spec-imagePullSecrets-secretName "Direct link to spec-imagePullSecrets-secretName") #### `secretNamespace` string [​](#spec-imagePullSecrets-secretNamespace "Direct link to spec-imagePullSecrets-secretNamespace") #### `key` string [​](#spec-imagePullSecrets-key "Direct link to spec-imagePullSecrets-key") ### `tokenGeneration` integer [​](#spec-tokenGeneration "Direct link to spec-tokenGeneration")TokenGeneration can be used to invalidate all user tokens ### `disabled` boolean [​](#spec-disabled "Direct link to spec-disabled")If disabled is true, a user will not be able to login anymore. All other user resources are unaffected and other users can still interact with this user ### `clusterRoles` object\[] [​](#spec-clusterRoles "Direct link to spec-clusterRoles")ClusterRoles define the cluster roles that the users should have assigned in the cluster. #### `name` string [​](#spec-clusterRoles-name "Direct link to spec-clusterRoles-name")Name is the cluster role to assign ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ### `extraClaims` object [​](#spec-extraClaims "Direct link to spec-extraClaims")ExtraClaims are additional claims that have been added to the user by an admin. ## `status` object [​](#status "Direct link to status") ### `teams` string\[] [​](#status-teams "Direct link to status-teams")Teams the user is currently part of ## Retrieve: Users[​](#retrieve-users "Direct link to Retrieve: Users") You can either use curl or kubectl to retrieve Users. * kubectl * curl ### Retrieve a list of Users[​](#retrieve-a-list-of-users "Direct link to Retrieve a list of Users") Run the following command to list all Users: ``` kubectl get users.management.loft.sh -o yaml ``` ### Retrieve a single User by name[​](#retrieve-a-single-user-by-name "Direct link to Retrieve a single User by name") Run the following kubectl command to get User `my-user`: ``` kubectl get users.management.loft.sh my-user -o yaml ``` ### Retrieve a list of Users[​](#retrieve-a-list-of-users-1 "Direct link to Retrieve a list of Users") Run the following curl command to list all Users: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/users" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single User by name[​](#get-a-single-user-by-name "Direct link to Get a single User by name") Run the following curl command to get User `my-user`: ``` # Exchange my-user in the url below with the name of the User curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/users/my-user" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: User[​](#create-user "Direct link to Create: User") You can either use curl or kubectl to create a new User. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: User metadata: name: my-user spec: clusterRoles: - name: loft-management-admin displayName: My User email: myuser@test.com subject: my-user username: myuser status: {} ``` Then create the User `my-user` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: User metadata: name: my-user spec: clusterRoles: - name: loft-management-admin displayName: My User email: myuser@test.com subject: my-user username: myuser status: {} ``` Run the following curl command to create a new User `my-user`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/users" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: User[​](#update-user "Direct link to Update: User") You can either use curl or kubectl to update Users. * kubectl * curl ### Update User[​](#update-user "Direct link to Update User") Run the following command to update User `my-user`: ``` kubectl edit users.management.loft.sh my-user ``` Then edit the object and upon save, kubectl will update the resource. ### Patch User[​](#patch-user "Direct link to Patch User") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the User `my-user` via a patch: ``` kubectl patch users.management.loft.sh my-user \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update User[​](#update-user-1 "Direct link to Update User") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: User metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-user resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: clusterRoles: - name: loft-management-admin displayName: My User email: myuser@test.com subject: my-user username: myuser status: {} ``` Run the following curl command to update a single User `my-user`: ``` # Replace the my-user in the url below with the name of the User you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/users/my-user" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch User[​](#patch-user-1 "Direct link to Patch User") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the User `my-user` via a patch: ``` # Replace the my-user in the url below with the name of the User you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/users/my-user" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: User[​](#delete-user "Direct link to Delete: User") You can either use curl or kubectl to delete Users. * kubectl * curl Run the following command to delete User `my-user`: ``` kubectl delete users.management.loft.sh my-user ``` Run the following curl command to delete User `my-user`: ``` # Replace the my-user in the url below with the name of the User you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/users/my-user" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Virtual Cluster Instance A virtual cluster is a fully functional Kubernetes cluster that runs inside the namespace of another Kubernetes cluster (host cluster). Virtual clusters are very useful if you are hitting the limits of namespaces and do not want to make special exceptions to the multi-tenancy configuration of the underlying cluster, e.g. a user needs their own CRD or user needs pods from 2 namespaces to communicate with each other but your standard NetworkPolicy does not allow this, then a virtual cluster may be perfect for this user. ## Virtual Cluster example[​](#virtual-cluster-example "Direct link to Virtual Cluster example") An example Virtual Cluster: ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterInstance metadata: name: my-virtual-cluster namespace: loft-p-my-project spec: clusterRef: {} displayName: my-display-name owner: user: my-user parameters: 'my-parameter: my-value' templateRef: name: my-virtual-cluster-template status: {} ``` ## Virtual Cluster reference[​](#virtual-cluster-reference "Direct link to Virtual Cluster reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that should be displayed in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes a tenant cluster instance ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `templateRef` object [​](#spec-templateRef "Direct link to spec-templateRef")TemplateRef holds the tenant cluster template reference #### `name` string [​](#spec-templateRef-name "Direct link to spec-templateRef-name")Name holds the name of the template to reference. #### `version` string [​](#spec-templateRef-version "Direct link to spec-templateRef-version")Version holds the template version to use. Version is expected to be in semantic versioning format. Alternatively, you can also exchange major, minor or patch with an 'x' to tell Loft to automatically select the latest major, minor or patch version. #### `syncOnce` boolean [​](#spec-templateRef-syncOnce "Direct link to spec-templateRef-syncOnce")SyncOnce tells the controller to sync the instance once with the template. This is useful if you want to sync an instance after a template was changed. To automatically sync an instance with a template, use 'x.x.x' as version instead. ### `template` object [​](#spec-template "Direct link to spec-template")Template is the inline template to use for tenant cluster creation. This is mutually exclusive with templateRef. #### `metadata` object [​](#spec-template-metadata "Direct link to spec-template-metadata") ##### `labels` object [​](#spec-template-metadata-labels "Direct link to spec-template-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-template-metadata-annotations "Direct link to spec-template-metadata-annotations")Annotations are annotations on the object #### `instanceTemplate` object [​](#spec-template-instanceTemplate "Direct link to spec-template-instanceTemplate")InstanceTemplate holds the tenant cluster instance template ##### `metadata` object [​](#spec-template-instanceTemplate-metadata "Direct link to spec-template-instanceTemplate-metadata") ##### `labels` object [​](#spec-template-instanceTemplate-metadata-labels "Direct link to spec-template-instanceTemplate-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-template-instanceTemplate-metadata-annotations "Direct link to spec-template-instanceTemplate-metadata-annotations")Annotations are annotations on the object #### `apps` object\[] [​](#spec-template-apps "Direct link to spec-template-apps")Apps specifies the apps that should get deployed by this template ##### `name` string [​](#spec-template-apps-name "Direct link to spec-template-apps-name")Name of the target app ##### `namespace` string [​](#spec-template-apps-namespace "Direct link to spec-template-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#spec-template-apps-releaseName "Direct link to spec-template-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#spec-template-apps-version "Direct link to spec-template-apps-version")Version of the app ##### `hash` string [​](#spec-template-apps-hash "Direct link to spec-template-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#spec-template-apps-parameters "Direct link to spec-template-apps-parameters")Parameters to use for the app #### `charts` object\[] [​](#spec-template-charts "Direct link to spec-template-charts")Charts are helm charts that should get deployed ##### `name` string [​](#spec-template-charts-name "Direct link to spec-template-charts-name")Name is the chart name in the repository ##### `version` string [​](#spec-template-charts-version "Direct link to spec-template-charts-version")Version is the chart version in the repository ##### `repoURL` string [​](#spec-template-charts-repoURL "Direct link to spec-template-charts-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#spec-template-charts-username "Direct link to spec-template-charts-username")The username that is required for this repository ##### `usernameRef` object [​](#spec-template-charts-usernameRef "Direct link to spec-template-charts-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#spec-template-charts-usernameRef-projectSecretRef "Direct link to spec-template-charts-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-charts-usernameRef-projectSecretRef-project "Direct link to spec-template-charts-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-charts-usernameRef-projectSecretRef-name "Direct link to spec-template-charts-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-charts-usernameRef-projectSecretRef-key "Direct link to spec-template-charts-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#spec-template-charts-password "Direct link to spec-template-charts-password")The password that is required for this repository ##### `passwordRef` object [​](#spec-template-charts-passwordRef "Direct link to spec-template-charts-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#spec-template-charts-passwordRef-projectSecretRef "Direct link to spec-template-charts-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-charts-passwordRef-projectSecretRef-project "Direct link to spec-template-charts-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-charts-passwordRef-projectSecretRef-name "Direct link to spec-template-charts-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-charts-passwordRef-projectSecretRef-key "Direct link to spec-template-charts-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#spec-template-charts-insecureSkipTlsVerify "Direct link to spec-template-charts-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `releaseName` string [​](#spec-template-charts-releaseName "Direct link to spec-template-charts-releaseName")ReleaseName is the preferred release name of the app ##### `releaseNamespace` string [​](#spec-template-charts-releaseNamespace "Direct link to spec-template-charts-releaseNamespace")ReleaseNamespace is the preferred release namespace of the app ##### `values` string [​](#spec-template-charts-values "Direct link to spec-template-charts-values")Values are the values that should get passed to the chart ##### `wait` boolean [​](#spec-template-charts-wait "Direct link to spec-template-charts-wait")Wait determines if Loft should wait during deploy for the app to become ready ##### `timeout` string [​](#spec-template-charts-timeout "Direct link to spec-template-charts-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) #### `objects` string [​](#spec-template-objects "Direct link to spec-template-objects")Objects are Kubernetes style YAMLs that should get deployed into the tenant cluster #### `access` object [​](#spec-template-access "Direct link to spec-template-access")Access defines the access of users and teams to the tenant cluster. ##### `defaultClusterRole` string [​](#spec-template-access-defaultClusterRole "Direct link to spec-template-access-defaultClusterRole")Specifies which cluster role should get applied to users or teams that do not match a rule below. ##### `rules` object\[] [​](#spec-template-access-rules "Direct link to spec-template-access-rules")Rules defines which users and teams should have which access to the tenant cluster. If no rule matches an authenticated incoming user, the user will get cluster admin access. ##### `users` string\[] [​](#spec-template-access-rules-users "Direct link to spec-template-access-rules-users")Users this rule matches. \* means all users. ##### `teams` string\[] [​](#spec-template-access-rules-teams "Direct link to spec-template-access-rules-teams")Teams that this rule matches. ##### `clusterRole` string [​](#spec-template-access-rules-clusterRole "Direct link to spec-template-access-rules-clusterRole")ClusterRole is the cluster role that should be assigned to the #### `pro` object [​](#spec-template-pro "Direct link to spec-template-pro")Pro defines the pro settings for the tenant cluster ##### `enabled` boolean [​](#spec-template-pro-enabled "Direct link to spec-template-pro-enabled")Enabled defines if the tenant cluster is a pro cluster or not #### `helmRelease` object [​](#spec-template-helmRelease "Direct link to spec-template-helmRelease")HelmRelease is the helm release configuration for the tenant cluster. ##### `chart` object [​](#spec-template-helmRelease-chart "Direct link to spec-template-helmRelease-chart")infos about what chart to deploy ##### `name` string [​](#spec-template-helmRelease-chart-name "Direct link to spec-template-helmRelease-chart-name")the name of the helm chart ##### `repo` string [​](#spec-template-helmRelease-chart-repo "Direct link to spec-template-helmRelease-chart-repo")the repo of the helm chart ##### `username` string [​](#spec-template-helmRelease-chart-username "Direct link to spec-template-helmRelease-chart-username")The username that is required for this repository ##### `password` string [​](#spec-template-helmRelease-chart-password "Direct link to spec-template-helmRelease-chart-password")The password that is required for this repository ##### `version` string [​](#spec-template-helmRelease-chart-version "Direct link to spec-template-helmRelease-chart-version")the version of the helm chart to use ##### `insecureSkipTlsVerify` boolean [​](#spec-template-helmRelease-chart-insecureSkipTlsVerify "Direct link to spec-template-helmRelease-chart-insecureSkipTlsVerify")InsecureSkipTlsVerify skips the TLS verification for the helm chart ##### `values` string [​](#spec-template-helmRelease-values "Direct link to spec-template-helmRelease-values")the values for the given chart #### `accessPoint` object [​](#spec-template-accessPoint "Direct link to spec-template-accessPoint")AccessPoint defines settings to expose the tenant cluster directly via an ingress rather than through the (default) Loft proxy ##### `ingress` object [​](#spec-template-accessPoint-ingress "Direct link to spec-template-accessPoint-ingress")Ingress defines tenant cluster access via ingress ##### `enabled` boolean [​](#spec-template-accessPoint-ingress-enabled "Direct link to spec-template-accessPoint-ingress-enabled")Enabled defines if the tenant cluster access point (via ingress) is enabled or not; requires the connected cluster to have the `loft.sh/ingress-suffix` annotation set to define the domain name suffix used for the ingress. #### `forwardToken` boolean [​](#spec-template-forwardToken "Direct link to spec-template-forwardToken")ForwardToken signals the proxy to pass through the used token to the virtual Kubernetes api server and do a TokenReview there. Deprecated: use ForwardTokenMode instead. When ForwardTokenMode is empty, ForwardToken==true is treated as ForwardTokenMode "TokenReview" and ForwardToken==false as "Off". #### `forwardTokenMode` string [​](#spec-template-forwardTokenMode "Direct link to spec-template-forwardTokenMode")ForwardTokenMode controls how the proxy forwards the caller's token to the virtual Kubernetes api server.- "Off": do not forward the token (default). - "TokenReview": run a TokenReview against the virtual cluster and proxy the request using impersonation headers for the resulting user (historical ForwardToken behavior). - "Passthrough": forward the raw bearer token directly to the virtual cluster api server without a TokenReview and without impersonation headers. The virtual cluster api server is responsible for authenticating and authorizing the request (e.g. via its own OIDC config). When empty, the deprecated ForwardToken bool is used to derive the mode. Both "TokenReview" and "Passthrough" skip the platform's own instance-access check and delegate authorization to the virtual cluster api server, so only enable them on virtual clusters that enforce their own authentication and authorization. Downgrade note: older platform versions understand only the ForwardToken bool. "Off" and "TokenReview" are kept in sync with it automatically, so they downgrade cleanly. "Passthrough" has no bool equivalent and is unavailable on older versions; it downgrades to forwarding disabled. #### `spaceTemplate` object [​](#spec-template-spaceTemplate "Direct link to spec-template-spaceTemplate")SpaceTemplate holds the space template ##### `metadata` object [​](#spec-template-spaceTemplate-metadata "Direct link to spec-template-spaceTemplate-metadata") ##### `labels` object [​](#spec-template-spaceTemplate-metadata-labels "Direct link to spec-template-spaceTemplate-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-template-spaceTemplate-metadata-annotations "Direct link to spec-template-spaceTemplate-metadata-annotations")Annotations are annotations on the object ##### `objects` string [​](#spec-template-spaceTemplate-objects "Direct link to spec-template-spaceTemplate-objects")Objects are Kubernetes style yamls that should get deployed into the tenant cluster namespace ##### `charts` object\[] [​](#spec-template-spaceTemplate-charts "Direct link to spec-template-spaceTemplate-charts")Charts are helm charts that should get deployed ##### `name` string [​](#spec-template-spaceTemplate-charts-name "Direct link to spec-template-spaceTemplate-charts-name")Name is the chart name in the repository ##### `version` string [​](#spec-template-spaceTemplate-charts-version "Direct link to spec-template-spaceTemplate-charts-version")Version is the chart version in the repository ##### `repoURL` string [​](#spec-template-spaceTemplate-charts-repoURL "Direct link to spec-template-spaceTemplate-charts-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#spec-template-spaceTemplate-charts-username "Direct link to spec-template-spaceTemplate-charts-username")The username that is required for this repository ##### `usernameRef` object [​](#spec-template-spaceTemplate-charts-usernameRef "Direct link to spec-template-spaceTemplate-charts-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#spec-template-spaceTemplate-charts-usernameRef-projectSecretRef "Direct link to spec-template-spaceTemplate-charts-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-spaceTemplate-charts-usernameRef-projectSecretRef-project "Direct link to spec-template-spaceTemplate-charts-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-spaceTemplate-charts-usernameRef-projectSecretRef-name "Direct link to spec-template-spaceTemplate-charts-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-spaceTemplate-charts-usernameRef-projectSecretRef-key "Direct link to spec-template-spaceTemplate-charts-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#spec-template-spaceTemplate-charts-password "Direct link to spec-template-spaceTemplate-charts-password")The password that is required for this repository ##### `passwordRef` object [​](#spec-template-spaceTemplate-charts-passwordRef "Direct link to spec-template-spaceTemplate-charts-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#spec-template-spaceTemplate-charts-passwordRef-projectSecretRef "Direct link to spec-template-spaceTemplate-charts-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-spaceTemplate-charts-passwordRef-projectSecretRef-project "Direct link to spec-template-spaceTemplate-charts-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-spaceTemplate-charts-passwordRef-projectSecretRef-name "Direct link to spec-template-spaceTemplate-charts-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-spaceTemplate-charts-passwordRef-projectSecretRef-key "Direct link to spec-template-spaceTemplate-charts-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#spec-template-spaceTemplate-charts-insecureSkipTlsVerify "Direct link to spec-template-spaceTemplate-charts-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `releaseName` string [​](#spec-template-spaceTemplate-charts-releaseName "Direct link to spec-template-spaceTemplate-charts-releaseName")ReleaseName is the preferred release name of the app ##### `releaseNamespace` string [​](#spec-template-spaceTemplate-charts-releaseNamespace "Direct link to spec-template-spaceTemplate-charts-releaseNamespace")ReleaseNamespace is the preferred release namespace of the app ##### `values` string [​](#spec-template-spaceTemplate-charts-values "Direct link to spec-template-spaceTemplate-charts-values")Values are the values that should get passed to the chart ##### `wait` boolean [​](#spec-template-spaceTemplate-charts-wait "Direct link to spec-template-spaceTemplate-charts-wait")Wait determines if Loft should wait during deploy for the app to become ready ##### `timeout` string [​](#spec-template-spaceTemplate-charts-timeout "Direct link to spec-template-spaceTemplate-charts-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) ##### `apps` object\[] [​](#spec-template-spaceTemplate-apps "Direct link to spec-template-spaceTemplate-apps")Apps specifies the apps that should get deployed by this template ##### `name` string [​](#spec-template-spaceTemplate-apps-name "Direct link to spec-template-spaceTemplate-apps-name")Name of the target app ##### `namespace` string [​](#spec-template-spaceTemplate-apps-namespace "Direct link to spec-template-spaceTemplate-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#spec-template-spaceTemplate-apps-releaseName "Direct link to spec-template-spaceTemplate-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#spec-template-spaceTemplate-apps-version "Direct link to spec-template-spaceTemplate-apps-version")Version of the app ##### `hash` string [​](#spec-template-spaceTemplate-apps-hash "Direct link to spec-template-spaceTemplate-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#spec-template-spaceTemplate-apps-parameters "Direct link to spec-template-spaceTemplate-apps-parameters")Parameters to use for the app ### `clusterRef` object [​](#spec-clusterRef "Direct link to spec-clusterRef")ClusterRef is the reference to the connected cluster holding this tenant cluster #### `cluster` string [​](#spec-clusterRef-cluster "Direct link to spec-clusterRef-cluster")Cluster is the connected cluster the space will be created in #### `namespace` string [​](#spec-clusterRef-namespace "Direct link to spec-clusterRef-namespace")Namespace is the namespace inside the connected cluster holding the space #### `virtualCluster` string [​](#spec-clusterRef-virtualCluster "Direct link to spec-clusterRef-virtualCluster")VirtualCluster is the name of the tenant cluster inside the namespace ### `parameters` string [​](#spec-parameters "Direct link to spec-parameters")Parameters are values to pass to the template. The values should be encoded as YAML string where each parameter is represented as a top-level field key. ### `extraAccessRules` object\[] [​](#spec-extraAccessRules "Direct link to spec-extraAccessRules")ExtraAccessRules defines extra rules which users and teams should have which access to the tenant cluster. #### `users` string\[] [​](#spec-extraAccessRules-users "Direct link to spec-extraAccessRules-users")Users this rule matches. \* means all users. #### `teams` string\[] [​](#spec-extraAccessRules-teams "Direct link to spec-extraAccessRules-teams")Teams that this rule matches. #### `clusterRole` string [​](#spec-extraAccessRules-clusterRole "Direct link to spec-extraAccessRules-clusterRole")ClusterRole is the cluster role that should be assigned to the ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access to the tenant cluster object itself #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ### `networkPeer` boolean [​](#spec-networkPeer "Direct link to spec-networkPeer")NetworkPeer specifies if the cluster is connected via tailscale. ### `external` boolean [​](#spec-external "Direct link to spec-external")External specifies if the tenant cluster is managed by the platform agent or externally. ### `standalone` boolean [​](#spec-standalone "Direct link to spec-standalone")Standalone specifies if the tenant cluster is standalone and not hosted in another Kubernetes cluster. ## `status` object [​](#status "Direct link to status") ### `phase` string [​](#status-phase "Direct link to status-phase")Phase describes the current phase the tenant cluster instance is in ### `reason` string [​](#status-reason "Direct link to status-reason")Reason describes the reason in machine-readable form why the cluster is in the current phase ### `message` string [​](#status-message "Direct link to status-message")Message describes the reason in human-readable form why the cluster is in the current phase ### `serviceUID` string [​](#status-serviceUID "Direct link to status-serviceUID")ServiceUID is the service uid of the tenant cluster to uniquely identify it. ### `kubernetesVersion` string [​](#status-kubernetesVersion "Direct link to status-kubernetesVersion")KubernetesVersion is the Kubernetes version of the tenant cluster. ### `deployHash` string [​](#status-deployHash "Direct link to status-deployHash")DeployHash is the hash of the last deployed values. ### `conditions` object [​](#status-conditions "Direct link to status-conditions")Conditions holds several conditions the tenant cluster might be in ### `virtualClusterObjects` object [​](#status-virtualClusterObjects "Direct link to status-virtualClusterObjects")VirtualClusterObjects are the objects that were applied within the tenant cluster itself #### `lastAppliedObjects` string [​](#status-virtualClusterObjects-lastAppliedObjects "Direct link to status-virtualClusterObjects-lastAppliedObjects")LastAppliedObjects holds the status for the objects that were applied #### `charts` object\[] [​](#status-virtualClusterObjects-charts "Direct link to status-virtualClusterObjects-charts")Charts are the charts that were applied ##### `name` string [​](#status-virtualClusterObjects-charts-name "Direct link to status-virtualClusterObjects-charts-name")Name of the chart that was applied ##### `namespace` string [​](#status-virtualClusterObjects-charts-namespace "Direct link to status-virtualClusterObjects-charts-namespace")Namespace of the chart that was applied ##### `lastAppliedChartConfigHash` string [​](#status-virtualClusterObjects-charts-lastAppliedChartConfigHash "Direct link to status-virtualClusterObjects-charts-lastAppliedChartConfigHash")LastAppliedChartConfigHash is the last applied configuration #### `apps` object\[] [​](#status-virtualClusterObjects-apps "Direct link to status-virtualClusterObjects-apps")Apps are the apps that were applied ##### `name` string [​](#status-virtualClusterObjects-apps-name "Direct link to status-virtualClusterObjects-apps-name")Name of the target app ##### `namespace` string [​](#status-virtualClusterObjects-apps-namespace "Direct link to status-virtualClusterObjects-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#status-virtualClusterObjects-apps-releaseName "Direct link to status-virtualClusterObjects-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#status-virtualClusterObjects-apps-version "Direct link to status-virtualClusterObjects-apps-version")Version of the app ##### `hash` string [​](#status-virtualClusterObjects-apps-hash "Direct link to status-virtualClusterObjects-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#status-virtualClusterObjects-apps-parameters "Direct link to status-virtualClusterObjects-apps-parameters")Parameters to use for the app ### `spaceObjects` object [​](#status-spaceObjects "Direct link to status-spaceObjects")SpaceObjects are the objects that were applied within the tenant cluster space #### `lastAppliedObjects` string [​](#status-spaceObjects-lastAppliedObjects "Direct link to status-spaceObjects-lastAppliedObjects")LastAppliedObjects holds the status for the objects that were applied #### `charts` object\[] [​](#status-spaceObjects-charts "Direct link to status-spaceObjects-charts")Charts are the charts that were applied ##### `name` string [​](#status-spaceObjects-charts-name "Direct link to status-spaceObjects-charts-name")Name of the chart that was applied ##### `namespace` string [​](#status-spaceObjects-charts-namespace "Direct link to status-spaceObjects-charts-namespace")Namespace of the chart that was applied ##### `lastAppliedChartConfigHash` string [​](#status-spaceObjects-charts-lastAppliedChartConfigHash "Direct link to status-spaceObjects-charts-lastAppliedChartConfigHash")LastAppliedChartConfigHash is the last applied configuration #### `apps` object\[] [​](#status-spaceObjects-apps "Direct link to status-spaceObjects-apps")Apps are the apps that were applied ##### `name` string [​](#status-spaceObjects-apps-name "Direct link to status-spaceObjects-apps-name")Name of the target app ##### `namespace` string [​](#status-spaceObjects-apps-namespace "Direct link to status-spaceObjects-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#status-spaceObjects-apps-releaseName "Direct link to status-spaceObjects-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#status-spaceObjects-apps-version "Direct link to status-spaceObjects-apps-version")Version of the app ##### `hash` string [​](#status-spaceObjects-apps-hash "Direct link to status-spaceObjects-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#status-spaceObjects-apps-parameters "Direct link to status-spaceObjects-apps-parameters")Parameters to use for the app ### `virtualCluster` object [​](#status-virtualCluster "Direct link to status-virtualCluster")VirtualCluster is the template rendered with all the parameters #### `metadata` object [​](#status-virtualCluster-metadata "Direct link to status-virtualCluster-metadata") ##### `labels` object [​](#status-virtualCluster-metadata-labels "Direct link to status-virtualCluster-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#status-virtualCluster-metadata-annotations "Direct link to status-virtualCluster-metadata-annotations")Annotations are annotations on the object #### `instanceTemplate` object [​](#status-virtualCluster-instanceTemplate "Direct link to status-virtualCluster-instanceTemplate")InstanceTemplate holds the tenant cluster instance template ##### `metadata` object [​](#status-virtualCluster-instanceTemplate-metadata "Direct link to status-virtualCluster-instanceTemplate-metadata") ##### `labels` object [​](#status-virtualCluster-instanceTemplate-metadata-labels "Direct link to status-virtualCluster-instanceTemplate-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#status-virtualCluster-instanceTemplate-metadata-annotations "Direct link to status-virtualCluster-instanceTemplate-metadata-annotations")Annotations are annotations on the object #### `apps` object\[] [​](#status-virtualCluster-apps "Direct link to status-virtualCluster-apps")Apps specifies the apps that should get deployed by this template ##### `name` string [​](#status-virtualCluster-apps-name "Direct link to status-virtualCluster-apps-name")Name of the target app ##### `namespace` string [​](#status-virtualCluster-apps-namespace "Direct link to status-virtualCluster-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#status-virtualCluster-apps-releaseName "Direct link to status-virtualCluster-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#status-virtualCluster-apps-version "Direct link to status-virtualCluster-apps-version")Version of the app ##### `hash` string [​](#status-virtualCluster-apps-hash "Direct link to status-virtualCluster-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#status-virtualCluster-apps-parameters "Direct link to status-virtualCluster-apps-parameters")Parameters to use for the app #### `charts` object\[] [​](#status-virtualCluster-charts "Direct link to status-virtualCluster-charts")Charts are helm charts that should get deployed ##### `name` string [​](#status-virtualCluster-charts-name "Direct link to status-virtualCluster-charts-name")Name is the chart name in the repository ##### `version` string [​](#status-virtualCluster-charts-version "Direct link to status-virtualCluster-charts-version")Version is the chart version in the repository ##### `repoURL` string [​](#status-virtualCluster-charts-repoURL "Direct link to status-virtualCluster-charts-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#status-virtualCluster-charts-username "Direct link to status-virtualCluster-charts-username")The username that is required for this repository ##### `usernameRef` object [​](#status-virtualCluster-charts-usernameRef "Direct link to status-virtualCluster-charts-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#status-virtualCluster-charts-usernameRef-projectSecretRef "Direct link to status-virtualCluster-charts-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#status-virtualCluster-charts-usernameRef-projectSecretRef-project "Direct link to status-virtualCluster-charts-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#status-virtualCluster-charts-usernameRef-projectSecretRef-name "Direct link to status-virtualCluster-charts-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#status-virtualCluster-charts-usernameRef-projectSecretRef-key "Direct link to status-virtualCluster-charts-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#status-virtualCluster-charts-password "Direct link to status-virtualCluster-charts-password")The password that is required for this repository ##### `passwordRef` object [​](#status-virtualCluster-charts-passwordRef "Direct link to status-virtualCluster-charts-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#status-virtualCluster-charts-passwordRef-projectSecretRef "Direct link to status-virtualCluster-charts-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#status-virtualCluster-charts-passwordRef-projectSecretRef-project "Direct link to status-virtualCluster-charts-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#status-virtualCluster-charts-passwordRef-projectSecretRef-name "Direct link to status-virtualCluster-charts-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#status-virtualCluster-charts-passwordRef-projectSecretRef-key "Direct link to status-virtualCluster-charts-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#status-virtualCluster-charts-insecureSkipTlsVerify "Direct link to status-virtualCluster-charts-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `releaseName` string [​](#status-virtualCluster-charts-releaseName "Direct link to status-virtualCluster-charts-releaseName")ReleaseName is the preferred release name of the app ##### `releaseNamespace` string [​](#status-virtualCluster-charts-releaseNamespace "Direct link to status-virtualCluster-charts-releaseNamespace")ReleaseNamespace is the preferred release namespace of the app ##### `values` string [​](#status-virtualCluster-charts-values "Direct link to status-virtualCluster-charts-values")Values are the values that should get passed to the chart ##### `wait` boolean [​](#status-virtualCluster-charts-wait "Direct link to status-virtualCluster-charts-wait")Wait determines if Loft should wait during deploy for the app to become ready ##### `timeout` string [​](#status-virtualCluster-charts-timeout "Direct link to status-virtualCluster-charts-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) #### `objects` string [​](#status-virtualCluster-objects "Direct link to status-virtualCluster-objects")Objects are Kubernetes style YAMLs that should get deployed into the tenant cluster #### `access` object [​](#status-virtualCluster-access "Direct link to status-virtualCluster-access")Access defines the access of users and teams to the tenant cluster. ##### `defaultClusterRole` string [​](#status-virtualCluster-access-defaultClusterRole "Direct link to status-virtualCluster-access-defaultClusterRole")Specifies which cluster role should get applied to users or teams that do not match a rule below. ##### `rules` object\[] [​](#status-virtualCluster-access-rules "Direct link to status-virtualCluster-access-rules")Rules defines which users and teams should have which access to the tenant cluster. If no rule matches an authenticated incoming user, the user will get cluster admin access. ##### `users` string\[] [​](#status-virtualCluster-access-rules-users "Direct link to status-virtualCluster-access-rules-users")Users this rule matches. \* means all users. ##### `teams` string\[] [​](#status-virtualCluster-access-rules-teams "Direct link to status-virtualCluster-access-rules-teams")Teams that this rule matches. ##### `clusterRole` string [​](#status-virtualCluster-access-rules-clusterRole "Direct link to status-virtualCluster-access-rules-clusterRole")ClusterRole is the cluster role that should be assigned to the #### `pro` object [​](#status-virtualCluster-pro "Direct link to status-virtualCluster-pro")Pro defines the pro settings for the tenant cluster ##### `enabled` boolean [​](#status-virtualCluster-pro-enabled "Direct link to status-virtualCluster-pro-enabled")Enabled defines if the tenant cluster is a pro cluster or not #### `helmRelease` object [​](#status-virtualCluster-helmRelease "Direct link to status-virtualCluster-helmRelease")HelmRelease is the helm release configuration for the tenant cluster. ##### `chart` object [​](#status-virtualCluster-helmRelease-chart "Direct link to status-virtualCluster-helmRelease-chart")infos about what chart to deploy ##### `name` string [​](#status-virtualCluster-helmRelease-chart-name "Direct link to status-virtualCluster-helmRelease-chart-name")the name of the helm chart ##### `repo` string [​](#status-virtualCluster-helmRelease-chart-repo "Direct link to status-virtualCluster-helmRelease-chart-repo")the repo of the helm chart ##### `username` string [​](#status-virtualCluster-helmRelease-chart-username "Direct link to status-virtualCluster-helmRelease-chart-username")The username that is required for this repository ##### `password` string [​](#status-virtualCluster-helmRelease-chart-password "Direct link to status-virtualCluster-helmRelease-chart-password")The password that is required for this repository ##### `version` string [​](#status-virtualCluster-helmRelease-chart-version "Direct link to status-virtualCluster-helmRelease-chart-version")the version of the helm chart to use ##### `insecureSkipTlsVerify` boolean [​](#status-virtualCluster-helmRelease-chart-insecureSkipTlsVerify "Direct link to status-virtualCluster-helmRelease-chart-insecureSkipTlsVerify")InsecureSkipTlsVerify skips the TLS verification for the helm chart ##### `values` string [​](#status-virtualCluster-helmRelease-values "Direct link to status-virtualCluster-helmRelease-values")the values for the given chart #### `accessPoint` object [​](#status-virtualCluster-accessPoint "Direct link to status-virtualCluster-accessPoint")AccessPoint defines settings to expose the tenant cluster directly via an ingress rather than through the (default) Loft proxy ##### `ingress` object [​](#status-virtualCluster-accessPoint-ingress "Direct link to status-virtualCluster-accessPoint-ingress")Ingress defines tenant cluster access via ingress ##### `enabled` boolean [​](#status-virtualCluster-accessPoint-ingress-enabled "Direct link to status-virtualCluster-accessPoint-ingress-enabled")Enabled defines if the tenant cluster access point (via ingress) is enabled or not; requires the connected cluster to have the `loft.sh/ingress-suffix` annotation set to define the domain name suffix used for the ingress. #### `forwardToken` boolean [​](#status-virtualCluster-forwardToken "Direct link to status-virtualCluster-forwardToken")ForwardToken signals the proxy to pass through the used token to the virtual Kubernetes api server and do a TokenReview there. Deprecated: use ForwardTokenMode instead. When ForwardTokenMode is empty, ForwardToken==true is treated as ForwardTokenMode "TokenReview" and ForwardToken==false as "Off". #### `forwardTokenMode` string [​](#status-virtualCluster-forwardTokenMode "Direct link to status-virtualCluster-forwardTokenMode")ForwardTokenMode controls how the proxy forwards the caller's token to the virtual Kubernetes api server.- "Off": do not forward the token (default). - "TokenReview": run a TokenReview against the virtual cluster and proxy the request using impersonation headers for the resulting user (historical ForwardToken behavior). - "Passthrough": forward the raw bearer token directly to the virtual cluster api server without a TokenReview and without impersonation headers. The virtual cluster api server is responsible for authenticating and authorizing the request (e.g. via its own OIDC config). When empty, the deprecated ForwardToken bool is used to derive the mode. Both "TokenReview" and "Passthrough" skip the platform's own instance-access check and delegate authorization to the virtual cluster api server, so only enable them on virtual clusters that enforce their own authentication and authorization. Downgrade note: older platform versions understand only the ForwardToken bool. "Off" and "TokenReview" are kept in sync with it automatically, so they downgrade cleanly. "Passthrough" has no bool equivalent and is unavailable on older versions; it downgrades to forwarding disabled. #### `spaceTemplate` object [​](#status-virtualCluster-spaceTemplate "Direct link to status-virtualCluster-spaceTemplate")SpaceTemplate holds the space template ##### `metadata` object [​](#status-virtualCluster-spaceTemplate-metadata "Direct link to status-virtualCluster-spaceTemplate-metadata") ##### `labels` object [​](#status-virtualCluster-spaceTemplate-metadata-labels "Direct link to status-virtualCluster-spaceTemplate-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#status-virtualCluster-spaceTemplate-metadata-annotations "Direct link to status-virtualCluster-spaceTemplate-metadata-annotations")Annotations are annotations on the object ##### `objects` string [​](#status-virtualCluster-spaceTemplate-objects "Direct link to status-virtualCluster-spaceTemplate-objects")Objects are Kubernetes style yamls that should get deployed into the tenant cluster namespace ##### `charts` object\[] [​](#status-virtualCluster-spaceTemplate-charts "Direct link to status-virtualCluster-spaceTemplate-charts")Charts are helm charts that should get deployed ##### `name` string [​](#status-virtualCluster-spaceTemplate-charts-name "Direct link to status-virtualCluster-spaceTemplate-charts-name")Name is the chart name in the repository ##### `version` string [​](#status-virtualCluster-spaceTemplate-charts-version "Direct link to status-virtualCluster-spaceTemplate-charts-version")Version is the chart version in the repository ##### `repoURL` string [​](#status-virtualCluster-spaceTemplate-charts-repoURL "Direct link to status-virtualCluster-spaceTemplate-charts-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#status-virtualCluster-spaceTemplate-charts-username "Direct link to status-virtualCluster-spaceTemplate-charts-username")The username that is required for this repository ##### `usernameRef` object [​](#status-virtualCluster-spaceTemplate-charts-usernameRef "Direct link to status-virtualCluster-spaceTemplate-charts-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#status-virtualCluster-spaceTemplate-charts-usernameRef-projectSecretRef "Direct link to status-virtualCluster-spaceTemplate-charts-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#status-virtualCluster-spaceTemplate-charts-usernameRef-projectSecretRef-project "Direct link to status-virtualCluster-spaceTemplate-charts-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#status-virtualCluster-spaceTemplate-charts-usernameRef-projectSecretRef-name "Direct link to status-virtualCluster-spaceTemplate-charts-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#status-virtualCluster-spaceTemplate-charts-usernameRef-projectSecretRef-key "Direct link to status-virtualCluster-spaceTemplate-charts-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#status-virtualCluster-spaceTemplate-charts-password "Direct link to status-virtualCluster-spaceTemplate-charts-password")The password that is required for this repository ##### `passwordRef` object [​](#status-virtualCluster-spaceTemplate-charts-passwordRef "Direct link to status-virtualCluster-spaceTemplate-charts-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#status-virtualCluster-spaceTemplate-charts-passwordRef-projectSecretRef "Direct link to status-virtualCluster-spaceTemplate-charts-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#status-virtualCluster-spaceTemplate-charts-passwordRef-projectSecretRef-project "Direct link to status-virtualCluster-spaceTemplate-charts-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#status-virtualCluster-spaceTemplate-charts-passwordRef-projectSecretRef-name "Direct link to status-virtualCluster-spaceTemplate-charts-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#status-virtualCluster-spaceTemplate-charts-passwordRef-projectSecretRef-key "Direct link to status-virtualCluster-spaceTemplate-charts-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#status-virtualCluster-spaceTemplate-charts-insecureSkipTlsVerify "Direct link to status-virtualCluster-spaceTemplate-charts-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `releaseName` string [​](#status-virtualCluster-spaceTemplate-charts-releaseName "Direct link to status-virtualCluster-spaceTemplate-charts-releaseName")ReleaseName is the preferred release name of the app ##### `releaseNamespace` string [​](#status-virtualCluster-spaceTemplate-charts-releaseNamespace "Direct link to status-virtualCluster-spaceTemplate-charts-releaseNamespace")ReleaseNamespace is the preferred release namespace of the app ##### `values` string [​](#status-virtualCluster-spaceTemplate-charts-values "Direct link to status-virtualCluster-spaceTemplate-charts-values")Values are the values that should get passed to the chart ##### `wait` boolean [​](#status-virtualCluster-spaceTemplate-charts-wait "Direct link to status-virtualCluster-spaceTemplate-charts-wait")Wait determines if Loft should wait during deploy for the app to become ready ##### `timeout` string [​](#status-virtualCluster-spaceTemplate-charts-timeout "Direct link to status-virtualCluster-spaceTemplate-charts-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) ##### `apps` object\[] [​](#status-virtualCluster-spaceTemplate-apps "Direct link to status-virtualCluster-spaceTemplate-apps")Apps specifies the apps that should get deployed by this template ##### `name` string [​](#status-virtualCluster-spaceTemplate-apps-name "Direct link to status-virtualCluster-spaceTemplate-apps-name")Name of the target app ##### `namespace` string [​](#status-virtualCluster-spaceTemplate-apps-namespace "Direct link to status-virtualCluster-spaceTemplate-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#status-virtualCluster-spaceTemplate-apps-releaseName "Direct link to status-virtualCluster-spaceTemplate-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#status-virtualCluster-spaceTemplate-apps-version "Direct link to status-virtualCluster-spaceTemplate-apps-version")Version of the app ##### `hash` string [​](#status-virtualCluster-spaceTemplate-apps-hash "Direct link to status-virtualCluster-spaceTemplate-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#status-virtualCluster-spaceTemplate-apps-parameters "Direct link to status-virtualCluster-spaceTemplate-apps-parameters")Parameters to use for the app ### `ignoreReconciliation` boolean [​](#status-ignoreReconciliation "Direct link to status-ignoreReconciliation")IgnoreReconciliation tells the controller to ignore reconciliation for this instance -- this is primarily used when migrating tenant cluster instances from project to project; this prevents a situation where there are two tenant cluster instances representing the same tenant cluster which could cause issues with concurrent reconciliations of the same object. Once the tenant cluster instance has been cloned and placed into the new project, this (the "old") tenant cluster instance can safely be deleted. ## Retrieve: Virtual Clusters[​](#retrieve-virtual-clusters "Direct link to Retrieve: Virtual Clusters") You can either use curl or kubectl to retrieve Virtual Clusters. * kubectl * curl ### Retrieve a list of Virtual Clusters[​](#retrieve-a-list-of-virtual-clusters "Direct link to Retrieve a list of Virtual Clusters") Run the following command to list all Virtual Clusters in project `my-project`: ``` kubectl get virtualclusterinstances.management.loft.sh -n p-my-project -o yaml ``` ### Retrieve a single Virtual Cluster by name[​](#retrieve-a-single-virtual-cluster-by-name "Direct link to Retrieve a single Virtual Cluster by name") Run the following kubectl command to get Virtual Cluster `my-virtual-cluster` in project `my-project`: ``` kubectl get virtualclusterinstances.management.loft.sh my-virtual-cluster -n p-my-project -o yaml ``` ### Retrieve a list of Virtual Clusters[​](#retrieve-a-list-of-virtual-clusters-1 "Direct link to Retrieve a list of Virtual Clusters") Run the following curl command to list all Virtual Clusters in project `my-project`: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/virtualclusterinstances" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single Virtual Cluster by name[​](#get-a-single-virtual-cluster-by-name "Direct link to Get a single Virtual Cluster by name") Run the following curl command to get Virtual Cluster `my-virtual-cluster` in project `my-project`: ``` # Exchange my-virtual-cluster in the url below with the name of the Virtual Cluster curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/virtualclusterinstances/my-virtual-cluster" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: Virtual Cluster[​](#create-virtual-cluster "Direct link to Create: Virtual Cluster") You can either use curl or kubectl to create a new Virtual Cluster. Make sure to set the project in the `metadata.namespace` field you want to create the Virtual Cluster in. If your project has the id `my-project`, the corresponding namespace would be `p-my-project`. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterInstance metadata: name: my-virtual-cluster namespace: loft-p-my-project spec: clusterRef: {} displayName: my-display-name owner: user: my-user parameters: 'my-parameter: my-value' templateRef: name: my-virtual-cluster-template status: {} ``` Then create the Virtual Cluster `my-virtual-cluster` in project `my-project` with: ``` kubectl create -f object.yaml -n p-my-project ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterInstance metadata: name: my-virtual-cluster namespace: loft-p-my-project spec: clusterRef: {} displayName: my-display-name owner: user: my-user parameters: 'my-parameter: my-value' templateRef: name: my-virtual-cluster-template status: {} ``` Run the following curl command to create a new Virtual Cluster `my-virtual-cluster` in project `my-project`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/virtualclusterinstances" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: Virtual Cluster[​](#update-virtual-cluster "Direct link to Update: Virtual Cluster") You can either use curl or kubectl to update Virtual Clusters. * kubectl * curl ### Update Virtual Cluster[​](#update-virtual-cluster "Direct link to Update Virtual Cluster") Run the following command to update Virtual Cluster `my-virtual-cluster` in project `my-project`: ``` kubectl edit virtualclusterinstances.management.loft.sh my-virtual-cluster -n p-my-project ``` Then edit the object and upon save, kubectl will update the resource. ### Patch Virtual Cluster[​](#patch-virtual-cluster "Direct link to Patch Virtual Cluster") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the Virtual Cluster `my-virtual-cluster` in project `my-project` via a patch: ``` kubectl patch virtualclusterinstances.management.loft.sh my-virtual-cluster -n p-my-project \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update Virtual Cluster[​](#update-virtual-cluster-1 "Direct link to Update Virtual Cluster") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterInstance metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-virtual-cluster namespace: loft-p-my-project resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: clusterRef: {} displayName: my-display-name owner: user: my-user parameters: 'my-parameter: my-value' templateRef: name: my-virtual-cluster-template status: {} ``` Run the following curl command to update a single Virtual Cluster `my-virtual-cluster` in project `my-project`: ``` # Replace the my-virtual-cluster in the url below with the name of the Virtual Cluster you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/virtualclusterinstances/my-virtual-cluster" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch Virtual Cluster[​](#patch-virtual-cluster-1 "Direct link to Patch Virtual Cluster") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the Virtual Cluster `my-virtual-cluster` in project `my-project` via a patch: ``` # Replace the my-virtual-cluster in the url below with the name of the Virtual Cluster you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/virtualclusterinstances/my-virtual-cluster" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: Virtual Cluster[​](#delete-virtual-cluster "Direct link to Delete: Virtual Cluster") You can either use curl or kubectl to delete Virtual Clusters. * kubectl * curl Run the following command to delete Virtual Cluster `my-virtual-cluster` in project `my-project`: ``` kubectl delete virtualclusterinstances.management.loft.sh my-virtual-cluster -n p-my-project ``` Run the following curl command to delete Virtual Cluster `my-virtual-cluster` in project `my-project`: ``` # Replace the my-virtual-cluster in the url below with the name of the Virtual Cluster you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/p-my-project/virtualclusterinstances/my-virtual-cluster" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Retrieve Kube Config (Ingress) If ingress endpoint is configured for the virtual cluster, you can retrieve the kube config for the virtual cluster through this endpoint ## Virtual Cluster Kube Config example[​](#virtual-cluster-kube-config-example "Direct link to Virtual Cluster Kube Config example") An example Virtual Cluster Kube Config: ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterInstanceKubeConfig metadata: namespace: your-namespace spec: {} status: kubeConfig: |- apiVersion: v1 kind: Config clusters: - cluster: ... ``` ## Virtual Cluster Kube Config reference[​](#virtual-cluster-kube-config-reference "Direct link to Virtual Cluster Kube Config reference") ## `spec` object [​](#spec "Direct link to spec") ### `certificateTTL` integer [​](#spec-certificateTTL "Direct link to spec-certificateTTL")CertificateTTL holds the ttl (in seconds) to set for the certificate associated with the returned kubeconfig. This field is optional, if no value is provided, the certificate TTL will be set to one day. If set to zero, this will cause loft to pass nil to the certificate signing request, which will result in the certificate being valid for the clusters `cluster-signing-duration` value which is typically one year. ### `server` string [​](#spec-server "Direct link to spec-server")Server allows user to override server in the kubeconfig. ### `clientCert` boolean [​](#spec-clientCert "Direct link to spec-clientCert")ClientCert, if set to true, will return kube config with generated client certs instead of platform token ## `status` object [​](#status "Direct link to status") ### `kubeConfig` string [​](#status-kubeConfig "Direct link to status-kubeConfig")KubeConfig holds the final kubeconfig output ## Virtual Cluster Kube Config (Create)[​](#virtual-cluster-kube-config-create "Direct link to Virtual Cluster Kube Config (Create)") * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterInstanceKubeConfig metadata: namespace: your-namespace spec: {} status: kubeConfig: |- apiVersion: v1 kind: Config clusters: - cluster: ... ``` Then run the following curl command: ``` # Exchange my-object in the url below with the name of the Virtual Cluster Kube Config curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/your-namespace/virtualclusterinstances/my-object/kubeconfig" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Retrieve Resource Usage Retrieve aggregated node and GPU resource usage for the tenant cluster's attached nodes, including a per-vendor GPU breakdown. ## Virtual cluster resource usage example[​](#virtual-cluster-resource-usage-example "Direct link to Virtual cluster resource usage example") An example Virtual cluster resource usage: ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterResourceUsage metadata: namespace: your-namespace status: resourceUsage: capacity: cpu: 48 nvidia.com/gpu: 4 gpus: - allocatable: 4 physical: 4 vendor: amd - allocatable: 8 physical: 8 vendor: intel - allocatable: 4 physical: 4 vendor: nvidia nodes: 3 ``` ## Virtual cluster resource usage reference[​](#virtual-cluster-resource-usage-reference "Direct link to Virtual cluster resource usage reference") ## `status` object [​](#status "Direct link to status") ### `resourceUsage` object [​](#status-resourceUsage "Direct link to status-resourceUsage")ResourceUsage contains the aggregated result of the queries to the tenant cluster's nodes. #### `nodes` required integer [​](#status-resourceUsage-nodes "Direct link to status-resourceUsage-nodes")Nodes is the total count of nodes attached to the tenant cluster. #### `capacity` object [​](#status-resourceUsage-capacity "Direct link to status-resourceUsage-capacity")Capacity is a map of raw node resource names to their total amounts across all attached nodes (e.g. "cpu", "nvidia.com/gpu"). It is per-resource-name, not per-vendor: for GPUs it only includes "nvidia.com/gpu" and not other accelerator vendors, so use GPUs (not Capacity) for a cross-vendor GPU total. #### `gpus` object\[] [​](#status-resourceUsage-gpus "Direct link to status-resourceUsage-gpus")GPUs is the accelerator usage across all attached nodes, broken down by GPU type (vendor and model). ##### `vendor` required string [​](#status-resourceUsage-gpus-vendor "Direct link to status-resourceUsage-gpus-vendor")Vendor is a normalized accelerator vendor id, e.g. "nvidia", "amd", "intel". ##### `model` string [​](#status-resourceUsage-gpus-model "Direct link to status-resourceUsage-gpus-model")Model is the hardware model when known, empty otherwise. ##### `allocatable` required integer [​](#status-resourceUsage-gpus-allocatable "Direct link to status-resourceUsage-gpus-allocatable")Allocatable is the total number of schedulable units of this type across all nodes. ##### `physical` required integer [​](#status-resourceUsage-gpus-physical "Direct link to status-resourceUsage-gpus-physical")Physical is the total number of physical accelerators of this type across all nodes. ## Virtual cluster resource usage (Get)[​](#virtual-cluster-resource-usage-get "Direct link to Virtual cluster resource usage (Get)") You can retrieve the aggregated node and GPU resource usage for a tenant cluster's attached nodes through this API. `capacity` only ever reflects `nvidia.com/gpu`; use `gpus` for the cross-vendor breakdown. * kubectl * curl Run the following command: ``` # Exchange my-object in the url below with the name of the VirtualClusterInstance kubectl get --raw "/kubernetes/management/apis/management.loft.sh/v1/namespaces/your-namespace/virtualclusterinstances/my-object/resourceusage" ``` Run the following curl command: ``` # Exchange my-object in the url below with the name of the VirtualClusterInstance curl -s -X GET --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/namespaces/your-namespace/virtualclusterinstances/my-object/resourceusage" \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Virtual Cluster Template Virtual Cluster templates can be used to create virtual clusters that have already certain applications deployed inside them or other predefined configuration applied. They are a powerful tool to create new predefined environments on demand. ## Virtual Cluster Template example[​](#virtual-cluster-template-example "Direct link to Virtual Cluster Template example") An example Virtual Cluster Template: ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterTemplate metadata: name: my-virtual-cluster-template spec: access: - users: - '*' verbs: - get description: This virtual cluster template deploys an isolated virtual cluster displayName: Isolated Virtual Cluster Template template: accessPoint: ingress: {} helmRelease: chart: {} values: |- # Below you can configure the virtual cluster isolation: enabled: true # Checkout https://vcluster.com/docs for more config options instanceTemplate: metadata: {} metadata: {} pro: {} spaceTemplate: metadata: {} status: {} ``` ## Virtual Cluster Template reference[​](#virtual-cluster-template-reference "Direct link to Virtual Cluster Template reference") ## `kind` string [​](#kind "Direct link to kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ## `apiVersion` string [​](#apiVersion "Direct link to apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: ## `metadata` object [​](#metadata "Direct link to metadata") ### `name` string [​](#metadata-name "Direct link to metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: ### `generateName` string [​](#metadata-generateName "Direct link to metadata-generateName")GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.If this field is specified and the generated name exists, the server will return a 409.Applied only if Name is not specified. More info: ### `namespace` string [​](#metadata-namespace "Direct link to metadata-namespace")Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.Must be a DNS\_LABEL. Cannot be updated. More info: ### `selfLink` string [​](#metadata-selfLink "Direct link to metadata-selfLink")Deprecated: selfLink is a legacy read-only field that is no longer populated by the system. ### `uid` string [​](#metadata-uid "Direct link to metadata-uid")UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.Populated by the system. Read-only. More info: ### `resourceVersion` string [​](#metadata-resourceVersion "Direct link to metadata-resourceVersion")An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: ### `generation` integer [​](#metadata-generation "Direct link to metadata-generation")A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. ### `creationTimestamp` object [​](#metadata-creationTimestamp "Direct link to metadata-creationTimestamp")CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.Populated by the system. Read-only. Null for lists. More info: ### `deletionTimestamp` object [​](#metadata-deletionTimestamp "Direct link to metadata-deletionTimestamp")DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.Populated by the system when a graceful deletion is requested. Read-only. More info: ### `deletionGracePeriodSeconds` integer [​](#metadata-deletionGracePeriodSeconds "Direct link to metadata-deletionGracePeriodSeconds")Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. ### `labels` object [​](#metadata-labels "Direct link to metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: ### `annotations` object [​](#metadata-annotations "Direct link to metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: ### `ownerReferences` object\[] [​](#metadata-ownerReferences "Direct link to metadata-ownerReferences")List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. #### `apiVersion` required string [​](#metadata-ownerReferences-apiVersion "Direct link to metadata-ownerReferences-apiVersion")API version of the referent. #### `kind` required string [​](#metadata-ownerReferences-kind "Direct link to metadata-ownerReferences-kind")Kind of the referent. More info: #### `name` required string [​](#metadata-ownerReferences-name "Direct link to metadata-ownerReferences-name")Name of the referent. More info: #### `uid` required string [​](#metadata-ownerReferences-uid "Direct link to metadata-ownerReferences-uid")UID of the referent. More info: #### `controller` boolean [​](#metadata-ownerReferences-controller "Direct link to metadata-ownerReferences-controller")If true, this reference points to the managing controller. #### `blockOwnerDeletion` boolean [​](#metadata-ownerReferences-blockOwnerDeletion "Direct link to metadata-ownerReferences-blockOwnerDeletion")If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. ### `finalizers` string\[] [​](#metadata-finalizers "Direct link to metadata-finalizers")Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. ### `managedFields` object\[] [​](#metadata-managedFields "Direct link to metadata-managedFields")ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. #### `manager` string [​](#metadata-managedFields-manager "Direct link to metadata-managedFields-manager")Manager is an identifier of the workflow managing these fields. #### `operation` string [​](#metadata-managedFields-operation "Direct link to metadata-managedFields-operation")Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. #### `apiVersion` string [​](#metadata-managedFields-apiVersion "Direct link to metadata-managedFields-apiVersion")APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. #### `time` object [​](#metadata-managedFields-time "Direct link to metadata-managedFields-time")Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. #### `fieldsType` string [​](#metadata-managedFields-fieldsType "Direct link to metadata-managedFields-fieldsType")FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1" #### `fieldsV1` object [​](#metadata-managedFields-fieldsV1 "Direct link to metadata-managedFields-fieldsV1")FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. #### `subresource` string [​](#metadata-managedFields-subresource "Direct link to metadata-managedFields-subresource")Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. ## `spec` object [​](#spec "Direct link to spec") ### `displayName` string [​](#spec-displayName "Direct link to spec-displayName")DisplayName is the name that is shown in the UI ### `description` string [​](#spec-description "Direct link to spec-description")Description describes the tenant cluster template ### `owner` object [​](#spec-owner "Direct link to spec-owner")Owner holds the owner of this object #### `user` string [​](#spec-owner-user "Direct link to spec-owner-user")User specifies a Loft user. #### `team` string [​](#spec-owner-team "Direct link to spec-owner-team")Team specifies a Loft team. ### `template` object [​](#spec-template "Direct link to spec-template")Template holds the tenant cluster template #### `metadata` object [​](#spec-template-metadata "Direct link to spec-template-metadata") ##### `labels` object [​](#spec-template-metadata-labels "Direct link to spec-template-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-template-metadata-annotations "Direct link to spec-template-metadata-annotations")Annotations are annotations on the object #### `instanceTemplate` object [​](#spec-template-instanceTemplate "Direct link to spec-template-instanceTemplate")InstanceTemplate holds the tenant cluster instance template ##### `metadata` object [​](#spec-template-instanceTemplate-metadata "Direct link to spec-template-instanceTemplate-metadata") ##### `labels` object [​](#spec-template-instanceTemplate-metadata-labels "Direct link to spec-template-instanceTemplate-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-template-instanceTemplate-metadata-annotations "Direct link to spec-template-instanceTemplate-metadata-annotations")Annotations are annotations on the object #### `apps` object\[] [​](#spec-template-apps "Direct link to spec-template-apps")Apps specifies the apps that should get deployed by this template ##### `name` string [​](#spec-template-apps-name "Direct link to spec-template-apps-name")Name of the target app ##### `namespace` string [​](#spec-template-apps-namespace "Direct link to spec-template-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#spec-template-apps-releaseName "Direct link to spec-template-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#spec-template-apps-version "Direct link to spec-template-apps-version")Version of the app ##### `hash` string [​](#spec-template-apps-hash "Direct link to spec-template-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#spec-template-apps-parameters "Direct link to spec-template-apps-parameters")Parameters to use for the app #### `charts` object\[] [​](#spec-template-charts "Direct link to spec-template-charts")Charts are helm charts that should get deployed ##### `name` string [​](#spec-template-charts-name "Direct link to spec-template-charts-name")Name is the chart name in the repository ##### `version` string [​](#spec-template-charts-version "Direct link to spec-template-charts-version")Version is the chart version in the repository ##### `repoURL` string [​](#spec-template-charts-repoURL "Direct link to spec-template-charts-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#spec-template-charts-username "Direct link to spec-template-charts-username")The username that is required for this repository ##### `usernameRef` object [​](#spec-template-charts-usernameRef "Direct link to spec-template-charts-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#spec-template-charts-usernameRef-projectSecretRef "Direct link to spec-template-charts-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-charts-usernameRef-projectSecretRef-project "Direct link to spec-template-charts-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-charts-usernameRef-projectSecretRef-name "Direct link to spec-template-charts-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-charts-usernameRef-projectSecretRef-key "Direct link to spec-template-charts-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#spec-template-charts-password "Direct link to spec-template-charts-password")The password that is required for this repository ##### `passwordRef` object [​](#spec-template-charts-passwordRef "Direct link to spec-template-charts-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#spec-template-charts-passwordRef-projectSecretRef "Direct link to spec-template-charts-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-charts-passwordRef-projectSecretRef-project "Direct link to spec-template-charts-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-charts-passwordRef-projectSecretRef-name "Direct link to spec-template-charts-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-charts-passwordRef-projectSecretRef-key "Direct link to spec-template-charts-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#spec-template-charts-insecureSkipTlsVerify "Direct link to spec-template-charts-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `releaseName` string [​](#spec-template-charts-releaseName "Direct link to spec-template-charts-releaseName")ReleaseName is the preferred release name of the app ##### `releaseNamespace` string [​](#spec-template-charts-releaseNamespace "Direct link to spec-template-charts-releaseNamespace")ReleaseNamespace is the preferred release namespace of the app ##### `values` string [​](#spec-template-charts-values "Direct link to spec-template-charts-values")Values are the values that should get passed to the chart ##### `wait` boolean [​](#spec-template-charts-wait "Direct link to spec-template-charts-wait")Wait determines if Loft should wait during deploy for the app to become ready ##### `timeout` string [​](#spec-template-charts-timeout "Direct link to spec-template-charts-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) #### `objects` string [​](#spec-template-objects "Direct link to spec-template-objects")Objects are Kubernetes style YAMLs that should get deployed into the tenant cluster #### `access` object [​](#spec-template-access "Direct link to spec-template-access")Access defines the access of users and teams to the tenant cluster. ##### `defaultClusterRole` string [​](#spec-template-access-defaultClusterRole "Direct link to spec-template-access-defaultClusterRole")Specifies which cluster role should get applied to users or teams that do not match a rule below. ##### `rules` object\[] [​](#spec-template-access-rules "Direct link to spec-template-access-rules")Rules defines which users and teams should have which access to the tenant cluster. If no rule matches an authenticated incoming user, the user will get cluster admin access. ##### `users` string\[] [​](#spec-template-access-rules-users "Direct link to spec-template-access-rules-users")Users this rule matches. \* means all users. ##### `teams` string\[] [​](#spec-template-access-rules-teams "Direct link to spec-template-access-rules-teams")Teams that this rule matches. ##### `clusterRole` string [​](#spec-template-access-rules-clusterRole "Direct link to spec-template-access-rules-clusterRole")ClusterRole is the cluster role that should be assigned to the #### `pro` object [​](#spec-template-pro "Direct link to spec-template-pro")Pro defines the pro settings for the tenant cluster ##### `enabled` boolean [​](#spec-template-pro-enabled "Direct link to spec-template-pro-enabled")Enabled defines if the tenant cluster is a pro cluster or not #### `helmRelease` object [​](#spec-template-helmRelease "Direct link to spec-template-helmRelease")HelmRelease is the helm release configuration for the tenant cluster. ##### `chart` object [​](#spec-template-helmRelease-chart "Direct link to spec-template-helmRelease-chart")infos about what chart to deploy ##### `name` string [​](#spec-template-helmRelease-chart-name "Direct link to spec-template-helmRelease-chart-name")the name of the helm chart ##### `repo` string [​](#spec-template-helmRelease-chart-repo "Direct link to spec-template-helmRelease-chart-repo")the repo of the helm chart ##### `username` string [​](#spec-template-helmRelease-chart-username "Direct link to spec-template-helmRelease-chart-username")The username that is required for this repository ##### `password` string [​](#spec-template-helmRelease-chart-password "Direct link to spec-template-helmRelease-chart-password")The password that is required for this repository ##### `version` string [​](#spec-template-helmRelease-chart-version "Direct link to spec-template-helmRelease-chart-version")the version of the helm chart to use ##### `insecureSkipTlsVerify` boolean [​](#spec-template-helmRelease-chart-insecureSkipTlsVerify "Direct link to spec-template-helmRelease-chart-insecureSkipTlsVerify")InsecureSkipTlsVerify skips the TLS verification for the helm chart ##### `values` string [​](#spec-template-helmRelease-values "Direct link to spec-template-helmRelease-values")the values for the given chart #### `accessPoint` object [​](#spec-template-accessPoint "Direct link to spec-template-accessPoint")AccessPoint defines settings to expose the tenant cluster directly via an ingress rather than through the (default) Loft proxy ##### `ingress` object [​](#spec-template-accessPoint-ingress "Direct link to spec-template-accessPoint-ingress")Ingress defines tenant cluster access via ingress ##### `enabled` boolean [​](#spec-template-accessPoint-ingress-enabled "Direct link to spec-template-accessPoint-ingress-enabled")Enabled defines if the tenant cluster access point (via ingress) is enabled or not; requires the connected cluster to have the `loft.sh/ingress-suffix` annotation set to define the domain name suffix used for the ingress. #### `forwardToken` boolean [​](#spec-template-forwardToken "Direct link to spec-template-forwardToken")ForwardToken signals the proxy to pass through the used token to the virtual Kubernetes api server and do a TokenReview there. Deprecated: use ForwardTokenMode instead. When ForwardTokenMode is empty, ForwardToken==true is treated as ForwardTokenMode "TokenReview" and ForwardToken==false as "Off". #### `forwardTokenMode` string [​](#spec-template-forwardTokenMode "Direct link to spec-template-forwardTokenMode")ForwardTokenMode controls how the proxy forwards the caller's token to the virtual Kubernetes api server.- "Off": do not forward the token (default). - "TokenReview": run a TokenReview against the virtual cluster and proxy the request using impersonation headers for the resulting user (historical ForwardToken behavior). - "Passthrough": forward the raw bearer token directly to the virtual cluster api server without a TokenReview and without impersonation headers. The virtual cluster api server is responsible for authenticating and authorizing the request (e.g. via its own OIDC config). When empty, the deprecated ForwardToken bool is used to derive the mode. Both "TokenReview" and "Passthrough" skip the platform's own instance-access check and delegate authorization to the virtual cluster api server, so only enable them on virtual clusters that enforce their own authentication and authorization. Downgrade note: older platform versions understand only the ForwardToken bool. "Off" and "TokenReview" are kept in sync with it automatically, so they downgrade cleanly. "Passthrough" has no bool equivalent and is unavailable on older versions; it downgrades to forwarding disabled. #### `spaceTemplate` object [​](#spec-template-spaceTemplate "Direct link to spec-template-spaceTemplate")SpaceTemplate holds the space template ##### `metadata` object [​](#spec-template-spaceTemplate-metadata "Direct link to spec-template-spaceTemplate-metadata") ##### `labels` object [​](#spec-template-spaceTemplate-metadata-labels "Direct link to spec-template-spaceTemplate-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-template-spaceTemplate-metadata-annotations "Direct link to spec-template-spaceTemplate-metadata-annotations")Annotations are annotations on the object ##### `objects` string [​](#spec-template-spaceTemplate-objects "Direct link to spec-template-spaceTemplate-objects")Objects are Kubernetes style yamls that should get deployed into the tenant cluster namespace ##### `charts` object\[] [​](#spec-template-spaceTemplate-charts "Direct link to spec-template-spaceTemplate-charts")Charts are helm charts that should get deployed ##### `name` string [​](#spec-template-spaceTemplate-charts-name "Direct link to spec-template-spaceTemplate-charts-name")Name is the chart name in the repository ##### `version` string [​](#spec-template-spaceTemplate-charts-version "Direct link to spec-template-spaceTemplate-charts-version")Version is the chart version in the repository ##### `repoURL` string [​](#spec-template-spaceTemplate-charts-repoURL "Direct link to spec-template-spaceTemplate-charts-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#spec-template-spaceTemplate-charts-username "Direct link to spec-template-spaceTemplate-charts-username")The username that is required for this repository ##### `usernameRef` object [​](#spec-template-spaceTemplate-charts-usernameRef "Direct link to spec-template-spaceTemplate-charts-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#spec-template-spaceTemplate-charts-usernameRef-projectSecretRef "Direct link to spec-template-spaceTemplate-charts-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-spaceTemplate-charts-usernameRef-projectSecretRef-project "Direct link to spec-template-spaceTemplate-charts-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-spaceTemplate-charts-usernameRef-projectSecretRef-name "Direct link to spec-template-spaceTemplate-charts-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-spaceTemplate-charts-usernameRef-projectSecretRef-key "Direct link to spec-template-spaceTemplate-charts-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#spec-template-spaceTemplate-charts-password "Direct link to spec-template-spaceTemplate-charts-password")The password that is required for this repository ##### `passwordRef` object [​](#spec-template-spaceTemplate-charts-passwordRef "Direct link to spec-template-spaceTemplate-charts-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#spec-template-spaceTemplate-charts-passwordRef-projectSecretRef "Direct link to spec-template-spaceTemplate-charts-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-template-spaceTemplate-charts-passwordRef-projectSecretRef-project "Direct link to spec-template-spaceTemplate-charts-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-template-spaceTemplate-charts-passwordRef-projectSecretRef-name "Direct link to spec-template-spaceTemplate-charts-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-template-spaceTemplate-charts-passwordRef-projectSecretRef-key "Direct link to spec-template-spaceTemplate-charts-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#spec-template-spaceTemplate-charts-insecureSkipTlsVerify "Direct link to spec-template-spaceTemplate-charts-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `releaseName` string [​](#spec-template-spaceTemplate-charts-releaseName "Direct link to spec-template-spaceTemplate-charts-releaseName")ReleaseName is the preferred release name of the app ##### `releaseNamespace` string [​](#spec-template-spaceTemplate-charts-releaseNamespace "Direct link to spec-template-spaceTemplate-charts-releaseNamespace")ReleaseNamespace is the preferred release namespace of the app ##### `values` string [​](#spec-template-spaceTemplate-charts-values "Direct link to spec-template-spaceTemplate-charts-values")Values are the values that should get passed to the chart ##### `wait` boolean [​](#spec-template-spaceTemplate-charts-wait "Direct link to spec-template-spaceTemplate-charts-wait")Wait determines if Loft should wait during deploy for the app to become ready ##### `timeout` string [​](#spec-template-spaceTemplate-charts-timeout "Direct link to spec-template-spaceTemplate-charts-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) ##### `apps` object\[] [​](#spec-template-spaceTemplate-apps "Direct link to spec-template-spaceTemplate-apps")Apps specifies the apps that should get deployed by this template ##### `name` string [​](#spec-template-spaceTemplate-apps-name "Direct link to spec-template-spaceTemplate-apps-name")Name of the target app ##### `namespace` string [​](#spec-template-spaceTemplate-apps-namespace "Direct link to spec-template-spaceTemplate-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#spec-template-spaceTemplate-apps-releaseName "Direct link to spec-template-spaceTemplate-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#spec-template-spaceTemplate-apps-version "Direct link to spec-template-spaceTemplate-apps-version")Version of the app ##### `hash` string [​](#spec-template-spaceTemplate-apps-hash "Direct link to spec-template-spaceTemplate-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#spec-template-spaceTemplate-apps-parameters "Direct link to spec-template-spaceTemplate-apps-parameters")Parameters to use for the app ### `parameters` object\[] [​](#spec-parameters "Direct link to spec-parameters")Parameters define additional app parameters that will set helm values #### `variable` string [​](#spec-parameters-variable "Direct link to spec-parameters-variable")Variable is the path of the variable. Can be foo or foo.bar for nested objects. #### `label` string [​](#spec-parameters-label "Direct link to spec-parameters-label")Label is the label to show for this parameter #### `description` string [​](#spec-parameters-description "Direct link to spec-parameters-description")Description is the description to show for this parameter #### `type` string [​](#spec-parameters-type "Direct link to spec-parameters-type")Type of the parameter. Can be one of: string, multiline, boolean, number and password #### `options` string\[] [​](#spec-parameters-options "Direct link to spec-parameters-options")Options is a slice of strings, where each string represents a mutually exclusive choice. #### `min` integer [​](#spec-parameters-min "Direct link to spec-parameters-min")Min is the minimum number if type is number #### `max` integer [​](#spec-parameters-max "Direct link to spec-parameters-max")Max is the maximum number if type is number #### `required` boolean [​](#spec-parameters-required "Direct link to spec-parameters-required")Required specifies if this parameter is required #### `defaultValue` string [​](#spec-parameters-defaultValue "Direct link to spec-parameters-defaultValue")DefaultValue is the default value if none is specified #### `placeholder` string [​](#spec-parameters-placeholder "Direct link to spec-parameters-placeholder")Placeholder shown in the UI #### `invalidation` string [​](#spec-parameters-invalidation "Direct link to spec-parameters-invalidation")Invalidation regex that if matched will reject the input #### `validation` string [​](#spec-parameters-validation "Direct link to spec-parameters-validation")Validation regex that if matched will allow the input #### `section` string [​](#spec-parameters-section "Direct link to spec-parameters-section")Section where this app should be displayed. Apps with the same section name will be grouped together ### `versions` object\[] [​](#spec-versions "Direct link to spec-versions")Versions are different versions of the template that can be referenced as well #### `template` object [​](#spec-versions-template "Direct link to spec-versions-template")Template holds the space template ##### `metadata` object [​](#spec-versions-template-metadata "Direct link to spec-versions-template-metadata") ##### `labels` object [​](#spec-versions-template-metadata-labels "Direct link to spec-versions-template-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-versions-template-metadata-annotations "Direct link to spec-versions-template-metadata-annotations")Annotations are annotations on the object ##### `instanceTemplate` object [​](#spec-versions-template-instanceTemplate "Direct link to spec-versions-template-instanceTemplate")InstanceTemplate holds the tenant cluster instance template ##### `metadata` object [​](#spec-versions-template-instanceTemplate-metadata "Direct link to spec-versions-template-instanceTemplate-metadata") ##### `labels` object [​](#spec-versions-template-instanceTemplate-metadata-labels "Direct link to spec-versions-template-instanceTemplate-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-versions-template-instanceTemplate-metadata-annotations "Direct link to spec-versions-template-instanceTemplate-metadata-annotations")Annotations are annotations on the object ##### `apps` object\[] [​](#spec-versions-template-apps "Direct link to spec-versions-template-apps")Apps specifies the apps that should get deployed by this template ##### `name` string [​](#spec-versions-template-apps-name "Direct link to spec-versions-template-apps-name")Name of the target app ##### `namespace` string [​](#spec-versions-template-apps-namespace "Direct link to spec-versions-template-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#spec-versions-template-apps-releaseName "Direct link to spec-versions-template-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#spec-versions-template-apps-version "Direct link to spec-versions-template-apps-version")Version of the app ##### `hash` string [​](#spec-versions-template-apps-hash "Direct link to spec-versions-template-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#spec-versions-template-apps-parameters "Direct link to spec-versions-template-apps-parameters")Parameters to use for the app ##### `charts` object\[] [​](#spec-versions-template-charts "Direct link to spec-versions-template-charts")Charts are helm charts that should get deployed ##### `name` string [​](#spec-versions-template-charts-name "Direct link to spec-versions-template-charts-name")Name is the chart name in the repository ##### `version` string [​](#spec-versions-template-charts-version "Direct link to spec-versions-template-charts-version")Version is the chart version in the repository ##### `repoURL` string [​](#spec-versions-template-charts-repoURL "Direct link to spec-versions-template-charts-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#spec-versions-template-charts-username "Direct link to spec-versions-template-charts-username")The username that is required for this repository ##### `usernameRef` object [​](#spec-versions-template-charts-usernameRef "Direct link to spec-versions-template-charts-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#spec-versions-template-charts-usernameRef-projectSecretRef "Direct link to spec-versions-template-charts-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-versions-template-charts-usernameRef-projectSecretRef-project "Direct link to spec-versions-template-charts-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-versions-template-charts-usernameRef-projectSecretRef-name "Direct link to spec-versions-template-charts-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-versions-template-charts-usernameRef-projectSecretRef-key "Direct link to spec-versions-template-charts-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#spec-versions-template-charts-password "Direct link to spec-versions-template-charts-password")The password that is required for this repository ##### `passwordRef` object [​](#spec-versions-template-charts-passwordRef "Direct link to spec-versions-template-charts-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#spec-versions-template-charts-passwordRef-projectSecretRef "Direct link to spec-versions-template-charts-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-versions-template-charts-passwordRef-projectSecretRef-project "Direct link to spec-versions-template-charts-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-versions-template-charts-passwordRef-projectSecretRef-name "Direct link to spec-versions-template-charts-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-versions-template-charts-passwordRef-projectSecretRef-key "Direct link to spec-versions-template-charts-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#spec-versions-template-charts-insecureSkipTlsVerify "Direct link to spec-versions-template-charts-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `releaseName` string [​](#spec-versions-template-charts-releaseName "Direct link to spec-versions-template-charts-releaseName")ReleaseName is the preferred release name of the app ##### `releaseNamespace` string [​](#spec-versions-template-charts-releaseNamespace "Direct link to spec-versions-template-charts-releaseNamespace")ReleaseNamespace is the preferred release namespace of the app ##### `values` string [​](#spec-versions-template-charts-values "Direct link to spec-versions-template-charts-values")Values are the values that should get passed to the chart ##### `wait` boolean [​](#spec-versions-template-charts-wait "Direct link to spec-versions-template-charts-wait")Wait determines if Loft should wait during deploy for the app to become ready ##### `timeout` string [​](#spec-versions-template-charts-timeout "Direct link to spec-versions-template-charts-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) ##### `objects` string [​](#spec-versions-template-objects "Direct link to spec-versions-template-objects")Objects are Kubernetes style YAMLs that should get deployed into the tenant cluster ##### `access` object [​](#spec-versions-template-access "Direct link to spec-versions-template-access")Access defines the access of users and teams to the tenant cluster. ##### `defaultClusterRole` string [​](#spec-versions-template-access-defaultClusterRole "Direct link to spec-versions-template-access-defaultClusterRole")Specifies which cluster role should get applied to users or teams that do not match a rule below. ##### `rules` object\[] [​](#spec-versions-template-access-rules "Direct link to spec-versions-template-access-rules")Rules defines which users and teams should have which access to the tenant cluster. If no rule matches an authenticated incoming user, the user will get cluster admin access. ##### `users` string\[] [​](#spec-versions-template-access-rules-users "Direct link to spec-versions-template-access-rules-users")Users this rule matches. \* means all users. ##### `teams` string\[] [​](#spec-versions-template-access-rules-teams "Direct link to spec-versions-template-access-rules-teams")Teams that this rule matches. ##### `clusterRole` string [​](#spec-versions-template-access-rules-clusterRole "Direct link to spec-versions-template-access-rules-clusterRole")ClusterRole is the cluster role that should be assigned to the ##### `pro` object [​](#spec-versions-template-pro "Direct link to spec-versions-template-pro")Pro defines the pro settings for the tenant cluster ##### `enabled` boolean [​](#spec-versions-template-pro-enabled "Direct link to spec-versions-template-pro-enabled")Enabled defines if the tenant cluster is a pro cluster or not ##### `helmRelease` object [​](#spec-versions-template-helmRelease "Direct link to spec-versions-template-helmRelease")HelmRelease is the helm release configuration for the tenant cluster. ##### `chart` object [​](#spec-versions-template-helmRelease-chart "Direct link to spec-versions-template-helmRelease-chart")infos about what chart to deploy ##### `name` string [​](#spec-versions-template-helmRelease-chart-name "Direct link to spec-versions-template-helmRelease-chart-name")the name of the helm chart ##### `repo` string [​](#spec-versions-template-helmRelease-chart-repo "Direct link to spec-versions-template-helmRelease-chart-repo")the repo of the helm chart ##### `username` string [​](#spec-versions-template-helmRelease-chart-username "Direct link to spec-versions-template-helmRelease-chart-username")The username that is required for this repository ##### `password` string [​](#spec-versions-template-helmRelease-chart-password "Direct link to spec-versions-template-helmRelease-chart-password")The password that is required for this repository ##### `version` string [​](#spec-versions-template-helmRelease-chart-version "Direct link to spec-versions-template-helmRelease-chart-version")the version of the helm chart to use ##### `insecureSkipTlsVerify` boolean [​](#spec-versions-template-helmRelease-chart-insecureSkipTlsVerify "Direct link to spec-versions-template-helmRelease-chart-insecureSkipTlsVerify")InsecureSkipTlsVerify skips the TLS verification for the helm chart ##### `values` string [​](#spec-versions-template-helmRelease-values "Direct link to spec-versions-template-helmRelease-values")the values for the given chart ##### `accessPoint` object [​](#spec-versions-template-accessPoint "Direct link to spec-versions-template-accessPoint")AccessPoint defines settings to expose the tenant cluster directly via an ingress rather than through the (default) Loft proxy ##### `ingress` object [​](#spec-versions-template-accessPoint-ingress "Direct link to spec-versions-template-accessPoint-ingress")Ingress defines tenant cluster access via ingress ##### `enabled` boolean [​](#spec-versions-template-accessPoint-ingress-enabled "Direct link to spec-versions-template-accessPoint-ingress-enabled")Enabled defines if the tenant cluster access point (via ingress) is enabled or not; requires the connected cluster to have the `loft.sh/ingress-suffix` annotation set to define the domain name suffix used for the ingress. ##### `forwardToken` boolean [​](#spec-versions-template-forwardToken "Direct link to spec-versions-template-forwardToken")ForwardToken signals the proxy to pass through the used token to the virtual Kubernetes api server and do a TokenReview there. Deprecated: use ForwardTokenMode instead. When ForwardTokenMode is empty, ForwardToken==true is treated as ForwardTokenMode "TokenReview" and ForwardToken==false as "Off". ##### `forwardTokenMode` string [​](#spec-versions-template-forwardTokenMode "Direct link to spec-versions-template-forwardTokenMode")ForwardTokenMode controls how the proxy forwards the caller's token to the virtual Kubernetes api server.- "Off": do not forward the token (default). - "TokenReview": run a TokenReview against the virtual cluster and proxy the request using impersonation headers for the resulting user (historical ForwardToken behavior). - "Passthrough": forward the raw bearer token directly to the virtual cluster api server without a TokenReview and without impersonation headers. The virtual cluster api server is responsible for authenticating and authorizing the request (e.g. via its own OIDC config). When empty, the deprecated ForwardToken bool is used to derive the mode. Both "TokenReview" and "Passthrough" skip the platform's own instance-access check and delegate authorization to the virtual cluster api server, so only enable them on virtual clusters that enforce their own authentication and authorization. Downgrade note: older platform versions understand only the ForwardToken bool. "Off" and "TokenReview" are kept in sync with it automatically, so they downgrade cleanly. "Passthrough" has no bool equivalent and is unavailable on older versions; it downgrades to forwarding disabled. ##### `spaceTemplate` object [​](#spec-versions-template-spaceTemplate "Direct link to spec-versions-template-spaceTemplate")SpaceTemplate holds the space template ##### `metadata` object [​](#spec-versions-template-spaceTemplate-metadata "Direct link to spec-versions-template-spaceTemplate-metadata") ##### `labels` object [​](#spec-versions-template-spaceTemplate-metadata-labels "Direct link to spec-versions-template-spaceTemplate-metadata-labels")Labels are labels on the object ##### `annotations` object [​](#spec-versions-template-spaceTemplate-metadata-annotations "Direct link to spec-versions-template-spaceTemplate-metadata-annotations")Annotations are annotations on the object ##### `objects` string [​](#spec-versions-template-spaceTemplate-objects "Direct link to spec-versions-template-spaceTemplate-objects")Objects are Kubernetes style yamls that should get deployed into the tenant cluster namespace ##### `charts` object\[] [​](#spec-versions-template-spaceTemplate-charts "Direct link to spec-versions-template-spaceTemplate-charts")Charts are helm charts that should get deployed ##### `name` string [​](#spec-versions-template-spaceTemplate-charts-name "Direct link to spec-versions-template-spaceTemplate-charts-name")Name is the chart name in the repository ##### `version` string [​](#spec-versions-template-spaceTemplate-charts-version "Direct link to spec-versions-template-spaceTemplate-charts-version")Version is the chart version in the repository ##### `repoURL` string [​](#spec-versions-template-spaceTemplate-charts-repoURL "Direct link to spec-versions-template-spaceTemplate-charts-repoURL")RepoURL is the repo url where the chart can be found ##### `username` string [​](#spec-versions-template-spaceTemplate-charts-username "Direct link to spec-versions-template-spaceTemplate-charts-username")The username that is required for this repository ##### `usernameRef` object [​](#spec-versions-template-spaceTemplate-charts-usernameRef "Direct link to spec-versions-template-spaceTemplate-charts-usernameRef")The username that is required for this repository ##### `projectSecretRef` object [​](#spec-versions-template-spaceTemplate-charts-usernameRef-projectSecretRef "Direct link to spec-versions-template-spaceTemplate-charts-usernameRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-versions-template-spaceTemplate-charts-usernameRef-projectSecretRef-project "Direct link to spec-versions-template-spaceTemplate-charts-usernameRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-versions-template-spaceTemplate-charts-usernameRef-projectSecretRef-name "Direct link to spec-versions-template-spaceTemplate-charts-usernameRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-versions-template-spaceTemplate-charts-usernameRef-projectSecretRef-key "Direct link to spec-versions-template-spaceTemplate-charts-usernameRef-projectSecretRef-key")Key of the project secret to use. ##### `password` string [​](#spec-versions-template-spaceTemplate-charts-password "Direct link to spec-versions-template-spaceTemplate-charts-password")The password that is required for this repository ##### `passwordRef` object [​](#spec-versions-template-spaceTemplate-charts-passwordRef "Direct link to spec-versions-template-spaceTemplate-charts-passwordRef")The password that is required for this repository ##### `projectSecretRef` object [​](#spec-versions-template-spaceTemplate-charts-passwordRef-projectSecretRef "Direct link to spec-versions-template-spaceTemplate-charts-passwordRef-projectSecretRef")ProjectSecretRef holds the reference to a project secret ##### `project` string [​](#spec-versions-template-spaceTemplate-charts-passwordRef-projectSecretRef-project "Direct link to spec-versions-template-spaceTemplate-charts-passwordRef-projectSecretRef-project")Project is the project name where the secret is located in. ##### `name` string [​](#spec-versions-template-spaceTemplate-charts-passwordRef-projectSecretRef-name "Direct link to spec-versions-template-spaceTemplate-charts-passwordRef-projectSecretRef-name")Name of the project secret to use. ##### `key` string [​](#spec-versions-template-spaceTemplate-charts-passwordRef-projectSecretRef-key "Direct link to spec-versions-template-spaceTemplate-charts-passwordRef-projectSecretRef-key")Key of the project secret to use. ##### `insecureSkipTlsVerify` boolean [​](#spec-versions-template-spaceTemplate-charts-insecureSkipTlsVerify "Direct link to spec-versions-template-spaceTemplate-charts-insecureSkipTlsVerify")If tls certificate checks for the chart download should be skipped ##### `releaseName` string [​](#spec-versions-template-spaceTemplate-charts-releaseName "Direct link to spec-versions-template-spaceTemplate-charts-releaseName")ReleaseName is the preferred release name of the app ##### `releaseNamespace` string [​](#spec-versions-template-spaceTemplate-charts-releaseNamespace "Direct link to spec-versions-template-spaceTemplate-charts-releaseNamespace")ReleaseNamespace is the preferred release namespace of the app ##### `values` string [​](#spec-versions-template-spaceTemplate-charts-values "Direct link to spec-versions-template-spaceTemplate-charts-values")Values are the values that should get passed to the chart ##### `wait` boolean [​](#spec-versions-template-spaceTemplate-charts-wait "Direct link to spec-versions-template-spaceTemplate-charts-wait")Wait determines if Loft should wait during deploy for the app to become ready ##### `timeout` string [​](#spec-versions-template-spaceTemplate-charts-timeout "Direct link to spec-versions-template-spaceTemplate-charts-timeout")Timeout is the time to wait for any individual Kubernetes operation (like Jobs for hooks) (default 5m0s) ##### `apps` object\[] [​](#spec-versions-template-spaceTemplate-apps "Direct link to spec-versions-template-spaceTemplate-apps")Apps specifies the apps that should get deployed by this template ##### `name` string [​](#spec-versions-template-spaceTemplate-apps-name "Direct link to spec-versions-template-spaceTemplate-apps-name")Name of the target app ##### `namespace` string [​](#spec-versions-template-spaceTemplate-apps-namespace "Direct link to spec-versions-template-spaceTemplate-apps-namespace")Namespace specifies in which target namespace the app should get deployed in ##### `releaseName` string [​](#spec-versions-template-spaceTemplate-apps-releaseName "Direct link to spec-versions-template-spaceTemplate-apps-releaseName")ReleaseName is the name of the app release ##### `version` string [​](#spec-versions-template-spaceTemplate-apps-version "Direct link to spec-versions-template-spaceTemplate-apps-version")Version of the app ##### `hash` string [​](#spec-versions-template-spaceTemplate-apps-hash "Direct link to spec-versions-template-spaceTemplate-apps-hash")Hash is the hash of the app configuration ##### `parameters` string [​](#spec-versions-template-spaceTemplate-apps-parameters "Direct link to spec-versions-template-spaceTemplate-apps-parameters")Parameters to use for the app #### `parameters` object\[] [​](#spec-versions-parameters "Direct link to spec-versions-parameters")Parameters define additional app parameters that will set helm values ##### `variable` string [​](#spec-versions-parameters-variable "Direct link to spec-versions-parameters-variable")Variable is the path of the variable. Can be foo or foo.bar for nested objects. ##### `label` string [​](#spec-versions-parameters-label "Direct link to spec-versions-parameters-label")Label is the label to show for this parameter ##### `description` string [​](#spec-versions-parameters-description "Direct link to spec-versions-parameters-description")Description is the description to show for this parameter ##### `type` string [​](#spec-versions-parameters-type "Direct link to spec-versions-parameters-type")Type of the parameter. Can be one of: string, multiline, boolean, number and password ##### `options` string\[] [​](#spec-versions-parameters-options "Direct link to spec-versions-parameters-options")Options is a slice of strings, where each string represents a mutually exclusive choice. ##### `min` integer [​](#spec-versions-parameters-min "Direct link to spec-versions-parameters-min")Min is the minimum number if type is number ##### `max` integer [​](#spec-versions-parameters-max "Direct link to spec-versions-parameters-max")Max is the maximum number if type is number ##### `required` boolean [​](#spec-versions-parameters-required "Direct link to spec-versions-parameters-required")Required specifies if this parameter is required ##### `defaultValue` string [​](#spec-versions-parameters-defaultValue "Direct link to spec-versions-parameters-defaultValue")DefaultValue is the default value if none is specified ##### `placeholder` string [​](#spec-versions-parameters-placeholder "Direct link to spec-versions-parameters-placeholder")Placeholder shown in the UI ##### `invalidation` string [​](#spec-versions-parameters-invalidation "Direct link to spec-versions-parameters-invalidation")Invalidation regex that if matched will reject the input ##### `validation` string [​](#spec-versions-parameters-validation "Direct link to spec-versions-parameters-validation")Validation regex that if matched will allow the input ##### `section` string [​](#spec-versions-parameters-section "Direct link to spec-versions-parameters-section")Section where this app should be displayed. Apps with the same section name will be grouped together #### `version` string [​](#spec-versions-version "Direct link to spec-versions-version")Version is the version. Needs to be in X.X.X format. ### `access` object\[] [​](#spec-access "Direct link to spec-access")Access holds the access rights for users and teams #### `name` string [​](#spec-access-name "Direct link to spec-access-name")Name is an optional name that is used for this access rule #### `verbs` required string\[] [​](#spec-access-verbs "Direct link to spec-access-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule. VerbAll represents all kinds. #### `subresources` string\[] [​](#spec-access-subresources "Direct link to spec-access-subresources")Subresources defines the sub resources that are allowed by this access rule #### `users` string\[] [​](#spec-access-users "Direct link to spec-access-users")Users specifies which users should be able to access this secret with the aforementioned verbs #### `teams` string\[] [​](#spec-access-teams "Direct link to spec-access-teams")Teams specifies which teams should be able to access this secret with the aforementioned verbs ### `spaceTemplateRef` object [​](#spec-spaceTemplateRef "Direct link to spec-spaceTemplateRef")DEPRECATED: SpaceTemplate to use to create the tenant cluster space if it does not exist #### `name` string [​](#spec-spaceTemplateRef-name "Direct link to spec-spaceTemplateRef-name")Name of the space template ## `status` object [​](#status "Direct link to status") ## Retrieve: Virtual Cluster Templates[​](#retrieve-virtual-cluster-templates "Direct link to Retrieve: Virtual Cluster Templates") You can either use curl or kubectl to retrieve Virtual Cluster Templates. * kubectl * curl ### Retrieve a list of Virtual Cluster Templates[​](#retrieve-a-list-of-virtual-cluster-templates "Direct link to Retrieve a list of Virtual Cluster Templates") Run the following command to list all Virtual Cluster Templates: ``` kubectl get virtualclustertemplates.management.loft.sh -o yaml ``` ### Retrieve a single Virtual Cluster Template by name[​](#retrieve-a-single-virtual-cluster-template-by-name "Direct link to Retrieve a single Virtual Cluster Template by name") Run the following kubectl command to get Virtual Cluster Template `my-virtual-cluster-template`: ``` kubectl get virtualclustertemplates.management.loft.sh my-virtual-cluster-template -o yaml ``` ### Retrieve a list of Virtual Cluster Templates[​](#retrieve-a-list-of-virtual-cluster-templates-1 "Direct link to Retrieve a list of Virtual Cluster Templates") Run the following curl command to list all Virtual Cluster Templates: ``` curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/virtualclustertemplates" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ### Get a single Virtual Cluster Template by name[​](#get-a-single-virtual-cluster-template-by-name "Direct link to Get a single Virtual Cluster Template by name") Run the following curl command to get Virtual Cluster Template `my-virtual-cluster-template`: ``` # Exchange my-virtual-cluster-template in the url below with the name of the Virtual Cluster Template curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/virtualclustertemplates/my-virtual-cluster-template" \ -X GET --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Create: Virtual Cluster Template[​](#create-virtual-cluster-template "Direct link to Create: Virtual Cluster Template") You can either use curl or kubectl to create a new Virtual Cluster Template. * kubectl * curl Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterTemplate metadata: name: my-virtual-cluster-template spec: access: - users: - '*' verbs: - get description: This virtual cluster template deploys an isolated virtual cluster displayName: Isolated Virtual Cluster Template template: accessPoint: ingress: {} helmRelease: chart: {} values: |- # Below you can configure the virtual cluster isolation: enabled: true # Checkout https://vcluster.com/docs for more config options instanceTemplate: metadata: {} metadata: {} pro: {} spaceTemplate: metadata: {} status: {} ``` Then create the Virtual Cluster Template `my-virtual-cluster-template` with: ``` kubectl create -f object.yaml ``` Create a file `object.yaml` with the following contents: ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterTemplate metadata: name: my-virtual-cluster-template spec: access: - users: - '*' verbs: - get description: This virtual cluster template deploys an isolated virtual cluster displayName: Isolated Virtual Cluster Template template: accessPoint: ingress: {} helmRelease: chart: {} values: |- # Below you can configure the virtual cluster isolation: enabled: true # Checkout https://vcluster.com/docs for more config options instanceTemplate: metadata: {} metadata: {} pro: {} spaceTemplate: metadata: {} status: {} ``` Run the following curl command to create a new Virtual Cluster Template `my-virtual-cluster-template`: ``` curl -s -X POST --insecure \ "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/virtualclustertemplates" \ --data-binary "$(cat object.yaml)" \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" ``` ## Update: Virtual Cluster Template[​](#update-virtual-cluster-template "Direct link to Update: Virtual Cluster Template") You can either use curl or kubectl to update Virtual Cluster Templates. * kubectl * curl ### Update Virtual Cluster Template[​](#update-virtual-cluster-template "Direct link to Update Virtual Cluster Template") Run the following command to update Virtual Cluster Template `my-virtual-cluster-template`: ``` kubectl edit virtualclustertemplates.management.loft.sh my-virtual-cluster-template ``` Then edit the object and upon save, kubectl will update the resource. ### Patch Virtual Cluster Template[​](#patch-virtual-cluster-template "Direct link to Patch Virtual Cluster Template") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following kubectl command to add a new annotation `my-annotation: my-value` to the Virtual Cluster Template `my-virtual-cluster-template` via a patch: ``` kubectl patch virtualclustertemplates.management.loft.sh my-virtual-cluster-template \ --type json \ -p '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ### Update Virtual Cluster Template[​](#update-virtual-cluster-template-1 "Direct link to Update Virtual Cluster Template") First retrieve the current object into a file `object.yaml`. This could look like: ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterTemplate metadata: creationTimestamp: "2023-04-03T00:00:00Z" generation: 12 name: my-virtual-cluster-template resourceVersion: "66325905" uid: af5f9f0f-8ab9-4b4b-a595-a95a5921f3c2 spec: access: - users: - '*' verbs: - get description: This virtual cluster template deploys an isolated virtual cluster displayName: Isolated Virtual Cluster Template template: accessPoint: ingress: {} helmRelease: chart: {} values: |- # Below you can configure the virtual cluster isolation: enabled: true # Checkout https://vcluster.com/docs for more config options instanceTemplate: metadata: {} metadata: {} pro: {} spaceTemplate: metadata: {} status: {} ``` Run the following curl command to update a single Virtual Cluster Template `my-virtual-cluster-template`: ``` # Replace the my-virtual-cluster-template in the url below with the name of the Virtual Cluster Template you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/virtualclustertemplates/my-virtual-cluster-template" \ -X PUT --insecure \ -H "Content-Type: application/yaml" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data-binary "$(cat object.yaml)" ``` ### Patch Virtual Cluster Template[​](#patch-virtual-cluster-template-1 "Direct link to Patch Virtual Cluster Template") Patching a resource is useful if you want to generically exchange only a small portion of the object instead of retrieving the whole object first and then modifying it. To learn more about patches in Kubernetes, please take a look at the [official docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#use-a-json-merge-patch-to-update-a-deployment). Run the following curl command to add a new annotation `my-annotation: my-value` to the Virtual Cluster Template `my-virtual-cluster-template` via a patch: ``` # Replace the my-virtual-cluster-template in the url below with the name of the Virtual Cluster Template you want to update curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/virtualclustertemplates/my-virtual-cluster-template" \ -X PATCH --insecure \ -H "Content-Type: application/json-patch+json" \ -H "Authorization: Bearer $ACCESS_KEY" \ --data '[{"op": "add", "path": "/metadata/annotations/my-annotation", "value": "my-value"}]' ``` ## Delete: Virtual Cluster Template[​](#delete-virtual-cluster-template "Direct link to Delete: Virtual Cluster Template") You can either use curl or kubectl to delete Virtual Cluster Templates. * kubectl * curl Run the following command to delete Virtual Cluster Template `my-virtual-cluster-template`: ``` kubectl delete virtualclustertemplates.management.loft.sh my-virtual-cluster-template ``` Run the following curl command to delete Virtual Cluster Template `my-virtual-cluster-template`: ``` # Replace the my-virtual-cluster-template in the url below with the name of the Virtual Cluster Template you want to delete curl -s "https://$LOFT_DOMAIN/kubernetes/management/apis/management.loft.sh/v1/virtualclustertemplates/my-virtual-cluster-template" \ -X DELETE --insecure \ -H "Authorization: Bearer $ACCESS_KEY" ``` --- # Use API The vCluster platform management API is a Kubernetes API server that only serves virtual resources (not persisted in etcd) that facilitate interaction with vCluster platform. These resources are primarily used by the UI and CLI to change state in the Kubernetes cluster where vCluster platform is installed in, without the need to have cluster wide access. In order to access the vCluster Platform management API, you'll need a vCluster Platform access key. See [access keys](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) for more information how to create a new access key. After you have obtained an access key, login into vCluster platform and connect to the management API: ``` vcluster platform login [domain] --access-key=[ACCESS_KEY] # Retrieve management API kube-context afterwards: vcluster platform connect management ``` warning The vCluster platform management API does only support resources in the API group `management.loft.sh` and you won't be able to query any pods, namespaces or other kubernetes resources from it. To view what resources are available in the management API Kubernetes server, you can take a look at the API reference. ## Optional: Connect to the Kubernetes Cluster vCluster platform is installed in[​](#optional-connect-to-the-kubernetes-cluster-vcluster-platform-is-installed-in "Direct link to Optional: Connect to the Kubernetes Cluster vCluster platform is installed in") Since the vCluster Platform API server doesn't contain resources such as `pods`, `services`, and so on, some tools get confused and won't work with the vCluster Platform API server directly (such as ArgoCD). Instead of using `vcluster platform connect management` to access the resources, you can run the following command instead: ``` # Get a context to the cluster where loft is installed in instead vcluster platform connect cluster loft-cluster # You can also query the management.loft.sh resources here through kubectl kubectl get users.management.loft.sh # Other resources also exist here kubectl get pods ``` The main difference between this approach and `vcluster platform connect management` is that all requests are routed through an actual Kubernetes cluster, which can be wanted if other resources are needed, but also expose all APIs to the requester, which is why we recommend to use `vcluster platform connect management` if possible. --- # Configure CORS policy Configure Cross-Origin Resource Sharing (CORS) headers at the ingress level so external browser-based clients can make API requests to vCluster Platform. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * A deployed vCluster Platform instance with [external access configured](https://vcluster.com/docs/platform/configure/installation-options/domain.md). * An ingress controller deployed on the control plane cluster. * `helm` v3.10+ and `kubectl` with admin access to the cluster. ## Overview[​](#overview "Direct link to Overview") If you build browser-based tools, such as a custom operations dashboard, a monitoring UI, or a CI interface, that call the Platform API from a different domain, browsers block those requests by default. This is the [same-origin policy](https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin_policy). CORS headers tell the browser which external origins are allowed. For example, a dashboard at `https://dashboard.example.com` that fetches data from the Platform API at `https://platform.example.com` fails unless the Platform ingress returns the correct CORS headers. When CORS configuration is needed The Platform UI and API share the same origin. Normal Platform access, `kubectl` commands, and in-cluster service-to-service traffic don't require CORS. Configure CORS only when a browser-based application at a different origin calls the Platform API. The Platform doesn't set CORS headers by default. Add them through your ingress controller. ## Configure ingress-level CORS[​](#ingress-cors "Direct link to Configure ingress-level CORS") ### ingress-nginx[​](#ingress-nginx "Direct link to ingress-nginx") ingress-nginx is deprecated The [ingress-nginx project](https://kubernetes.github.io/ingress-nginx/) is deprecated upstream. For new deployments, use the [Gateway API CORS filter](#other-ingress-controllers) or another actively maintained ingress controller. 1. Create or update your `vcluster-platform.yaml` with CORS annotations on the Platform ingress: Modify the following with your specific values to generate a copyable command: PLATFORM\_HOSTplatform.example.com ALLOWED\_ORIGINhttps\://dashboard.example.com TLS\_SECRETtls-vcluster-platform vcluster-platform.yaml ``` ingress: enabled: true host: "platform.example.com" annotations: # CORS nginx.ingress.kubernetes.io/enable-cors: "true" nginx.ingress.kubernetes.io/cors-allow-origin: "https://dashboard.example.com" nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, PUT, DELETE, PATCH, OPTIONS" nginx.ingress.kubernetes.io/cors-allow-headers: "Authorization, Content-Type" nginx.ingress.kubernetes.io/cors-allow-credentials: "true" nginx.ingress.kubernetes.io/cors-max-age: "3600" tls: enabled: true secret: tls-vcluster-platform ``` 2. Apply the configuration: Modify the following with your specific values to generate a copyable command: RELEASE\_NAMEvcluster-platform RELEASE\_NAMESPACEvcluster-platform Upgrade the Platform with CORS annotations ``` helm upgrade vcluster-platform vcluster-platform \ --repo https://charts.loft.sh/ \ --namespace vcluster-platform \ --reuse-values \ -f vcluster-platform.yaml ``` 3. Verify the ingress returns CORS headers by sending a preflight request: Modify the following with your specific values to generate a copyable command: ALLOWED\_ORIGINhttps\://dashboard.example.com PLATFORM\_HOSTplatform.example.com Test CORS preflight ``` curl -v -X OPTIONS \ -H "Origin: https://dashboard.example.com" \ -H "Access-Control-Request-Method: GET" \ https://platform.example.com/kubernetes/management/apis ``` A correct response includes these headers: ``` Access-Control-Allow-Origin: https://dashboard.example.com Access-Control-Allow-Methods: GET, POST, PUT, DELETE, PATCH, OPTIONS Access-Control-Allow-Headers: Authorization, Content-Type ``` If these headers are missing, check the ingress annotations and confirm the ingress controller processes them. ### Annotation reference[​](#annotation-reference "Direct link to Annotation reference") All annotations use the `nginx.ingress.kubernetes.io/` prefix. | Annotation | Description | Default | | ------------------------ | ----------------------------------- | -------------------------------------------- | | `enable-cors` | Enable CORS headers | `"false"` | | `cors-allow-origin` | Allowed origins (comma-separated) | `"*"` | | `cors-allow-methods` | Allowed HTTP methods | `"GET, PUT, POST, DELETE, PATCH, OPTIONS"` | | `cors-allow-headers` | Allowed request headers | Standard browser headers and `Authorization` | | `cors-allow-credentials` | Allow credentials (cookies, auth) | `"false"` | | `cors-expose-headers` | Response headers exposed to browser | (none) | | `cors-max-age` | Preflight cache duration in seconds | `"1728000"` | warning Don't set `cors-allow-origin` to `"*"` when `cors-allow-credentials` is `"true"`. Browsers reject this combination per the CORS specification. List explicit origins instead. ### Other ingress controllers[​](#other-ingress-controllers "Direct link to Other ingress controllers") Traefik, Istio, and Gateway API support CORS through their own configuration: * **Traefik**: Use the [headers middleware](https://doc.traefik.io/traefik/middlewares/http/headers/) with `accessControlAllowOriginList` on an IngressRoute. * **Istio**: Use the [`corsPolicy` field](https://istio.io/latest/docs/reference/config/networking/virtual-service/) on a VirtualService. See [Expose vCluster Platform with Istio](https://vcluster.com/docs/platform/how-to/exposing-vCluster-platform-with-Istio.md) for a related setup guide. * **Gateway API**: The experimental CORS filter (Gateway API v1.3+) supports CORS on HTTPRoute resources. See the [Gateway API CORS guide](https://gateway-api.sigs.k8s.io/guides/http-cors/) for details. ## Security considerations[​](#security-considerations "Direct link to Security considerations") ### Restrict allowed origins[​](#restrict-allowed-origins "Direct link to Restrict allowed origins") Setting `cors-allow-origin` to `"*"` disables CORS protection. Any website a user visits can make cross-origin requests to the Platform API. danger Never use `cors-allow-origin: "*"` in production. List only the specific domains that need access. * Prefer explicit origins (`https://dashboard.example.com`) over wildcards. * Separate development and production configurations. Don't include localhost origins in production values. ### CORS doesn't replace authentication[​](#cors-doesnt-replace-authentication "Direct link to CORS doesn't replace authentication") CORS headers control which origins a browser permits for cross-origin requests. They don't authenticate or authorize API calls. The Platform's authentication, RBAC, and network policies remain the primary access controls. ## Troubleshoot CORS issues[​](#troubleshoot-cors-issues "Direct link to Troubleshoot CORS issues") ### Browser shows "blocked by CORS policy"[​](#browser-shows-blocked-by-cors-policy "Direct link to Browser shows \"blocked by CORS policy\"") The ingress didn't return an `Access-Control-Allow-Origin` header for the requesting origin. Verify the following: 1. The `enable-cors` annotation is `"true"` on the Platform ingress. 2. The `cors-allow-origin` value matches the exact origin the browser sends. Scheme, host, and port must all match. 3. The ingress controller terminates TLS. Check the Platform's [TLS configuration](https://vcluster.com/docs/platform/configure/installation-options/domain.md). ## Limitations[​](#limitations "Direct link to Limitations") * Changes to ingress annotations require a `helm upgrade` of the Platform to update the ingress resource. ## Related topics[​](#related-topics "Direct link to Related topics") * [Configure external access and TLS](https://vcluster.com/docs/platform/configure/installation-options/domain.md) * [Expose vCluster Platform with Istio](https://vcluster.com/docs/platform/how-to/exposing-vCluster-platform-with-Istio.md) --- # Agent upgrades ## Platform-managed agents[​](#platform-managed-agents "Direct link to Platform-managed agents") Platform-managed agents automatically receive upgrades controlled by the control plane. These automatic upgrades also handle more significant agent changes and migrations. All agents connect using an egress-only connection to the platform. ### Upgrade process[​](#upgrade-process "Direct link to Upgrade process") When the platform initiates an upgrade on an egress-only agent, it will first deploy a proxy-only agent. As the name suggests, this proxy-only agent solely aims to proxy the communication between the platform and the connected cluster's Kubernetes API server. The platform deploys the proxy-only agent by creating a deployment in the connected cluster's agent's namespace. It additionally reuses the agent's service account and associated RBAC permissions. Once the proxy-only agent is healthy, the platform upgrades the agent deployment. In case of a failure during the upgrade, the platform can still connect using the proxy-only agent and perform a rollback to a previous agent version. When the platform starts the agent upgrade, the agent will be unavailable for a brief period until it restarts and reconnects. This proxy-only agent maintains a connection to the connected cluster's Kubernetes apiserver during that upgrade period and serves as a backup connection. The proxy-only agent exists for the entire duration of the upgrade process, and the platform uses it to communicate with the connected cluster's Kubernetes API server. By doing so, the platform ensures that the connected cluster remains operational and can continue to receive updates and perform operations. Once the agent upgrade is complete, the platform will remove the proxy-only agent and the associated resources from the connected cluster. ## Self-managed agents[​](#self-managed-agents "Direct link to Self-managed agents") If the platform does not manage your agents, manually upgrade them by repeating the installation steps using the CLI or Helm command. Instructions on how to do can be found [here](https://vcluster.com/docs/platform/configure/agent-settings/overview.md#connect-to-platform). ## Disable agent upgrades[​](#disable-agent-upgrades "Direct link to Disable agent upgrades") There might be cases where you don't want vCluster Platform to automatically handle vCluster Platform agent updates. This can be achieved either by setting the environment variable `DISABLE_AGENT` to `true` in the vCluster Platform container or by setting the annotation `loft.sh/cluster-ignore-agent: 'true'` on a connected cluster. Or you can finish this in vCluster Platform UI: 1. Go to Infrastructure > Control Plane Clusters. 2. Click the Edit option for the cluster you want to modify. 3. In the configuration sheet that opens, click the Agent tab and enable Ignore Agent. 4. Click the Save Changes button. warning If you do not install vCluster Platform agent into a connected cluster at all, certain capability, such as Spaces, Auto Sleep, Apps, Accounts, Account Quotas & Security Templates will not be available in the cluster --- # Per-control plane cluster customization As described in [installation modes](https://vcluster.com/docs/platform/configure/installation-options/overview.md#installation-modes), the vCluster Platform Agent can be installed using the same `vcluster-platform` chart by setting `agentOnly: true`. The configuration of the agent by default will be decided by the configuration of the platform in the primary control plane cluster and the `agentValues` content. However, this means that all control plane clusters connecting to the platform will share the same configuration. If different connected control plane clusters require different agent configurations, there are two supported approaches. ## `loft.sh/agent-values` annotation[​](#loftsh-annotations "Direct link to loftsh-annotations") Add the `loft.sh/agent-values` annotation to a specific Cluster resource (through the UI or YAML). This annotation overrides the platform-level `agentValues`. The override applies only to the annotated Cluster. For example: ``` loft.sh/agent-values: | replicaCount: 2 resources: limits: memory: 2Gi ``` ## Override values[​](#override-values "Direct link to Override values") When installing the agent directly on a control plane cluster, you can override values by passing a custom values file to Helm. But first, you should tell vCluster Platform to ignore the agent of the specific cluster by adding annotation: ``` kubectl annotate cluster cluster-B loft.sh/cluster-ignore-agent="true" ``` Then you can set the values through the `--values` flag: ``` helm install vcluster-platform vcluster-platform \ --set agentOnly=true \ --values custom-values.yaml ``` This approach allows full control over agent configuration at installation time for that specific control plane cluster. You can also update the agent settings in vCluster Platform UI. 1. Go to Infrastructure > Control Plane Clusters. 2. Click the Edit option for the cluster you want to modify. 3. In the configuration sheet that opens, click the Agent tab. Provide the values in the Extra Agent Values field in YAML format. 4. Click the Save Changes button. --- # Least Privilege Mode | | | Enterprise | | | | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Least Privilege Mode](https://vcluster.com/docs/platform/configure/agent-settings/least-privilege-mode.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). By default, to minimize operational overhead, the **vCluster Platform Agent** requests broad permissions to support all platform features. If your organization follows strict RBAC policies, **Least Privilege Mode** can be used to limit the **vCluster Platform Agent** permissions only to your deployment needs. Scope **Least Privilege Mode** applies only to agents deployed on **external control plane clusters**. It does **not** apply to the agent running in the cluster where the platform is installed. Agent Upgrades To use this approach, you must disable automatic agent upgrades in the **vCluster Platform** because this process requires the agent to also have self-upgrade permissions. Instead, you must manage the upgrades yourself, so that the agent permissions can be limited to its runtime needs. See [**Self-managed agents / Disable agent upgrades**](https://vcluster.com/docs/platform/configure/agent-settings/agent-upgrade.md#disable-agent-upgrades) for more details. ## How it works[​](#how-it-works "Direct link to How it works") **Least Privilege Mode** is configured under `agentValues.leastPrivilegeMode`. The following configuration options are available: * **Feature toggles** - Enable/disable toggles for the features that should be supported by the agent (ClusterAccess, ProjectQuotas, Secrets, SleepMode). * **Cluster scope permissions** – Controls the cluster scope permissions granted to the vCluster Platform Agent. Permissions can be extended to accommodate permissions for custom resources. * **Managed namespaces scope permissions** – Controls the permissions granted to the vCluster Platform Agent into the namespaces where tenant cluster instances are installed. When a feature toggle is disabled, the corresponding permissions will not be requested and the internal Kubernetes controllers will not be started. Requests for the disabled features will not be fulfilled. For example, if the ProjectQuotas feature is disabled, the project quotas will not be enforced for the tenant cluster instances deployed on the connected cluster. See the [Configuration reference](#configuration-reference) section for feature specific disabled state handling. ## Example[​](#example "Direct link to Example") The following example enables **Least Privilege Mode** and disables all optional features for the vCluster Platform agent: platform.yaml ``` agentValues: leastPrivilegeMode: enabled: true clusterAccess: enabled: false projectQuotas: enabled: false secrets: enabled: false sleepMode: enabled: false ``` ## Configuration reference[​](#configuration-reference "Direct link to Configuration reference") ### `leastPrivilegeMode`[​](#leastprivilegemode "Direct link to leastprivilegemode")**Least Privilege Mode** configuration #### `enabled` boolean false[​](#enabled-boolean-false "Direct link to enabled-boolean-false")Enables **Least Privilege Mode** for the agent If set to `false`, the agent uses the default permission model configured via `serviceAccount.clusterRole` configuration. #### `clusterAccess` object[​](#clusteraccess-object "Direct link to clusteraccess-object")Cluster Access feature configuration ##### `enabled` boolean true [​](#enabled-boolean-true- "Direct link to enabled-boolean-true-")Controls whether the agent supports the vCluster Platform **Cluster Access** feature. If set to `false`, vCluster Platform users will not be able to access the connected cluster through the platform infrastructure. #### `projectQuotas` object[​](#projectquotas-object "Direct link to projectquotas-object")Project Quotas feature configuration ##### `enabled` boolean true [​](#enabled-boolean-true--1 "Direct link to enabled-boolean-true--1")Controls whether the agent supports the vCluster Platform **Project Quotas** feature. If set to `false`, project quotas will not be enforced for the tenant cluster instances deployed on the connected cluster. #### `secrets` object[​](#secrets-object "Direct link to secrets-object")Global/Project Secrets feature configuration ##### `enabled` boolean true [​](#enabled-boolean-true--2 "Direct link to enabled-boolean-true--2")Controls whether the agent supports the vCluster Platform **Global/Project Secrets** feature. If set to `false`, Global/Project secrets will not be synchronized with the connected cluster. #### `sleepMode` object[​](#sleepmode-object "Direct link to sleepmode-object")SleepMode feature configuration ##### `enabled` boolean true [​](#enabled-boolean-true--3 "Direct link to enabled-boolean-true--3")Controls whether the agent supports the vCluster Platform **SleepMode** feature. If set to `false`, sleep/auto-sleep/wake actions will be ignored for the tenant cluster instances deployed on the connected cluster. #### `clusterRole` object[​](#clusterrole-object "Direct link to clusterrole-object")ClusterRole configuration ##### `enabled` boolean true [​](#enabled-boolean-true--4 "Direct link to enabled-boolean-true--4")Controls whether the vCluster Platform Agent ClusterRole and ClusterRoleBinding resources should be created. Disable this if vCluster Platform Agent RBAC configuration is managed through different means. ##### `extraRules` object\[] [​](#extrarules-object-- "Direct link to extrarules-object--")Allows granting additional permissions to the vCluster Platform Agent at the cluster scope. platform.yaml ``` agentValues: leastPrivilegeMode: enabled: true clusterRole: extraRules: - apiGroups: ["my.custom.group/v1"] resources: ["my-custom-resources"] verbs: ["get", "list", "watch"] ``` ##### `overwriteRules` object\[] [​](#overwriterules-object-- "Direct link to overwriterules-object--")Allows overwriting the vCluster Platform Agent cluster scope permissions. Advanced option. To be used only if you need to remove the default permissions granted to the agent. #### `namespaceAdminRole` object[​](#namespaceadminrole-object "Direct link to namespaceadminrole-object")Managed namespace admin configuration The vCluster Platform Agent namespace-admin Role provides permissions to allow the agent to manage tenant cluster instances within a managed namespace, without cluster-admin level permissions. As part of the tenant cluster instances reconciliation loop, the namespace-admin role is created in the managed namespace and assigned to the vCluster Platform Agent service account. ##### `enabled` boolean true [​](#enabled-boolean-true--5 "Direct link to enabled-boolean-true--5")Controls whether the vCluster Platform Agent namespace-admin Role and RoleBinding resources should be created within the managed namespaces. Disable this if vCluster Platform Agent RBAC configuration is managed through different means. ##### `extraRules` object\[] [​](#extrarules-object---1 "Direct link to extrarules-object---1")Allows granting additional permissions to the vCluster Platform Agent within the managed namespaces where tenant cluster instances are installed. platform.yaml ``` agentValues: leastPrivilegeMode: enabled: true namespaceAdminRole: extraRules: - apiGroups: ["my.custom.group/v1"] resources: ["my-custom-resources"] verbs: ["get", "list", "watch"] ``` ##### `overwriteRules` object\[] [​](#overwriterules-object---1 "Direct link to overwriterules-object---1")Allows overwriting the vCluster Platform Agent permissions within the managed namespaces. Advanced option. To be used only if you need to remove the default permissions granted to the agent. #### `role` object[​](#role-object "Direct link to role-object")Role configuration The vCluster Platform Agent Role provides common Kubernetes controllers permissions required to allow the agent to function. Permissions for configMaps read, secrets read and coordination API are included in this role. ##### `enabled` boolean true [​](#enabled-boolean-true--6 "Direct link to enabled-boolean-true--6")Controls whether the vCluster Platform Agent Role and RoleBinding resources should be created. Disable this if vCluster Platform Agent RBAC configuration is managed through different means. ## Recommended approach[​](#recommended-approach "Direct link to Recommended approach") Start with the absolute minimum configuration. Add the required features one by one. A typical rollout looks like this: 1. Confirm that the vCluster Platform managed [agent upgrades](https://vcluster.com/docs/platform/configure/agent-settings/agent-upgrade.md#disable-agent-upgrades) is disabled for the connected clusters. 2. Enable **Least Privilege Mode** and **disable** all optional features. 3. Verify deployment: * Confirm that tenant cluster instances can be deployed as expected. * Validate deployment compliance with your organization's policies. 4. Enable required features one by one and repeat the verification steps. 5. Test agent behavior in a non-production environment. 6. Roll out the same configuration to production. ## Limitations[​](#limitations "Direct link to Limitations") * **Cost Control** incompatibility - Enabling the observability stack expands the agent's RBAC permissions. To limit the agent permissions to its runtime needs, deploy the observability stack using a dedicated process. * **Snapshots support** - The cluster instance snapshot feature requires `pods/exec` (`create` verb) in the managed namespaces, which is not included in the default **Least Privilege Mode** configuration. Organizations with strict policies that prohibit `pods/exec` cannot use this feature with **Least Privilege Mode** enabled. ## Troubleshoot least privilege mode[​](#troubleshoot-least-privilege-mode "Direct link to Troubleshoot least privilege mode") If the agent stops working after enabling **Least Privilege Mode**: * Review agent logs for RBAC permission errors. `agentValues.env.KUBERNETES_VERBOSITY_LEVEL: "4"` option can be used to gain visibility into the vCluster Platform Agent Kubernetes API requests. * Confirm that you are applying this only to the agents running on external control plane clusters If you need more help troubleshooting agent behavior, see [Troubleshoot connected cluster connection failures](https://vcluster.com/docs/platform/troubleshoot/connected-cluster-connectivity.md). --- # Overview When a **vCluster Platform** is deployed on a control plane cluster (or, primary control plane cluster), it can act as a centralized control plane. Other control plane clusters can connect to the vCluster Platform running on the primary control plane cluster and be managed by it. In this architecture: * The primary control plane cluster runs the vCluster Platform. * Other control plane clusters connect to the primary control plane cluster and run the vCluster Platform Agent. * The Platform coordinates and manages all connected control plane clusters through their agents. * There are both vCluster Platform and vCluster Platform agent running on the primary control plane cluster. The vCluster Platform manages the primary control plane cluster through its vCluster Platform Agent as well. Agent settings are the content in the [`values.yaml`](https://vcluster.com/docs/platform/configure/introduction.md) under the `agentValues`. It controls the behavior of vCluster Platform Agents installed in the control plane clusters that are managed by the vCluster Platform. The `agentValues` behavior is as follows: * By default, `agentValues` is an empty object `{}`. * An empty `agentValues` object means that agents installed on connected control plane clusters will inherit the same configuration as the platform running on the primary control plane cluster. * You can populate `agentValues` to override the default agent configuration globally for all connected control plane clusters. ## Connect to platform[​](#connect-to-platform "Direct link to Connect to platform") A new cluster can be connected to the platform through the UI or CLI: * UI * CLI * Helm 1. Go to Infrastructure > Control Plane Clusters. 2. Click Connect Cluster. 3. Give your cluster a name in Display Name. Optionally, give a name for the underlying kubernetes resource in the Cluster ID field, or leave it empty to autogenerate one. Then click Next. 4. Copy & execute the displayed vCluster CLI or Helm command. 5. Wait until vCluster Platform installs the agent in your connected cluster. 6. Once successful, use the displayed CLI command to create a space / virtual cluster in the newly connected cluster. 7. Click Finish to go to the Control Plane Clusters view. Connect a cluster UI will generate a CLI command that can be used to connect the cluster. Make sure to check the `CLI` tab for working with the command. 1. When connecting a cluster from the vCluster CLI, you will need to provide a name for the cluster. This name will be used for the Cluster resource that gets created in Kubernetes. Optionally, you can use the `display-name` flag to specify a name for this cluster, when it gets displayed under Infrastructure > Control Plane Clusters in vCluster Platform. ``` vcluster platform add cluster cluster-name --display-name [display-name] ``` The above command automatically installs the [vCluster Platform Agent](https://vcluster.com/docs/platform/administer/clusters/advanced/agent-config.md) in the cluster that corresponds to your current Kubernetes context. The control plane cluster where vCluster Platform itself is deployed is automatically connected to the platform during installation. Therefore, this command should be executed in a different cluster from the one where vCluster Platform is running. By default, the `vcluster-platform` namespace is created automatically if it does not already exist on the cluster. The vCluster Platform Agent resources are deployed into this namespace. You can override this behavior by explicitly specifying a different namespace using the `--namespace` flag. If you want to connect a cluster that is not associated with your current Kubernetes context, use the `--context` flag to specify the context of the target cluster. info For more information about available CLI flags, refer to the [CLI documentation](https://vcluster.com/docs/vcluster/cli/vcluster_platform_add_cluster). ``` vcluster platform add cluster cluster-name --display-name [display-name] --context [context] ``` 2. You can verify that the cluster is connected to the platform by running: ``` vcluster platform list clusters ``` and the expected output is: ``` CLUSTER | AGE --------------------+-------- cluster-name | 7s host-cluster | 10m4s ``` The `host-cluster` is the cluster name where the vCluster Platform is deployed. platform login **Connect to the Platform**: to use any `vcluster platform` command, you need to be connected to the platform. See [authentication documentation](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) for more details. When you started the platform, you would have automatically connected to the platform when you ran `vcluster platform start`. If you aren't sure if you are connected and already have a running platform, then you can login. Before running this command, be sure that your kubecontext is set to the cluster running the platform. Login to the platform. ``` vcluster platform login ``` If your kubectl context is not set to the cluster with running platform, it's possible to login to the platform using the CLI and access key. If you haven't already, you need to [create access key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md#create-an-access-key) to be able to login to the platform. Login to the platform using access key ``` vcluster platform login [platform-url] --access-key [access-key] ``` 1. If you prefer to use Helm, you need to make sure that you have a valid Access Key. You can either reuse the CLI Access Key or create a new one by following the [Access Key guide](https://vcluster.com/docs/platform/administer/authentication/access-keys.md#create-an-access-key). * Reusing the CLI Access Key ``` export ACCESS_KEY=$(vcluster platform access-key | jq -r .status.token) ``` * Create new Access Key by following the [Access Key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md#create-an-access-key) guide. ``` export ACCESS_KEY="[Newly Created Access Key]" ``` 2. After that, you need to provide a name for the cluster. ``` export CLUSTER_NAME="cluster-name" ``` 3. Next, you must create the `Cluster` custom resource within the platform using the following code: ``` cat < --values=https:// ``` An [access key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md#how-access-keys-work) is required for authentication.
| Host | Cluster Name | Source | Request | Response | | ----------------------- | ------------ | -------------------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | | `nimbus.vcluster.cloud` | `staging` | none | `https://nimbus.vcluster.cloud/clusters/staging?token=access-key` | Current agent values | | `nimbus.vcluster.cloud` | `staging` | `platform-config` | `https://nimbus.vcluster.cloud/clusters/staging?token=access-key&source=platform-config` | Content of `agentValues` section in platform config | | `nimbus.vcluster.cloud` | `staging` | `cluster-annotation` | `https://nimbus.vcluster.cloud/clusters/staging?token=access-key&source=cluster-annotation` | Value of the `loft.sh/agent-values` annotation on the `Cluster Object` | --- # Configure security context Security context is one of the common agent configurations. Agent pods support multiple levels of security context configuration. Security Context Precedence When multiple security contexts are configured, the agent uses this precedence order (highest to lowest): 1. Cluster-specific `agentValues` (via `loft.sh/agent-values` annotation) 2. Platform-wide `agentValues` (in platform values.yaml) 3. Platform default `securityContext` and `podSecurityContext` See [Platform configuration](https://vcluster.com/docs/platform/configure/platform-configs/overview.md) for platform-wide settings. Inspect the security context applied to agent pods: Modify the following with your specific values to generate a copyable command: MANAGEMENT\_NAMESPACEvcluster-platform View security contexts ``` kubectl get pod -n vcluster-platform -l app=loft -o yaml | grep -A 10 "securityContext:" ``` Example output: ``` securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL privileged: false runAsNonRoot: true ``` Container vs Pod Security Context * **Container Security Context** (`securityContext`): Applied to individual containers within a pod. Controls settings like user ID, privilege escalation, and capabilities. * **Pod Security Context** (`podSecurityContext`): Applied at the pod level. Controls settings like filesystem group, supplemental groups, and sysctls that affect all containers in the pod. Both Agent Pod and Agent Proxy Pod (deployed during upgrades) follow the same precedence order. #### Platform-wide security context[​](#platform-wide-security-context "Direct link to Platform-wide security context") As mentioned [here](https://vcluster.com/docs/platform/configure/agent-settings/overview.md), the platform agents by default inherit values from the platform configuration but you can use `agentValues` section of [`values.yaml`](https://vcluster.com/docs/platform/configure/introduction.md) to override the configuration values. Configure security contexts in your `vcluster-platform` chart values file [`values.yaml`](https://vcluster.com/docs/platform/configure/introduction.md) during platform installation: Example of vcluster-platform values.yaml ``` # Platform-wide default security contexts securityContext: allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 1000 capabilities: drop: - ALL podSecurityContext: fsGroup: 2000 runAsGroup: 3000 # Default agent values that apply to all connected clusters agentValues: securityContext: allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 1001 capabilities: drop: - ALL podSecurityContext: fsGroup: 2001 runAsGroup: 3001 ``` #### Cluster-specific security context[​](#cluster-specific-security-context "Direct link to Cluster-specific security context") As mentioned [here](https://vcluster.com/docs/platform/configure/agent-settings/customization.md), you can customize the agent in each connected control plane cluster independently. To achieve cluster-specific security context, you can override security contexts for specific clusters using the [`loft.sh/agent-values` annotation](https://vcluster.com/docs/platform/configure/agent-settings/overview.md#loftsh-annotations): Example of cluster-specific-agent-values.yaml ``` # Applied via loft.sh/agent-values annotation on the Cluster resource securityContext: allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 1002 capabilities: drop: - ALL add: - NET_BIND_SERVICE podSecurityContext: fsGroup: 2002 runAsGroup: 3002 supplementalGroups: - 4000 - 5000 ``` Or you can apply cluster-specific security context through the UI by following the steps in [Setting Agent Values via UI](https://vcluster.com/docs/platform/configure/agent-settings/overview.md#connect-to-platform) and add your security context configuration in the Extra Agent Values text area. --- # Configure External Access & TLS After deploying the platform, you may want to access the deployment through a routable IP address or a resolvable domain name. You may also wish to secure the deployment with a TLS certificate. This is particularly useful for sharing platform access with other team members who may or may not have direct access to Kubernetes. Requirements Setting up the platform for access through a routable IP or domain name, and securing it with TLS requires the following: * The cluster where the platform is deployed must have an available load balancer or ingress controller * Appropriate DNS settings must be in place * A mechanism for assigning TLS certificates (such as cert-manager) must be available ## External access[​](#external-access "Direct link to External access") The platform, like any other service in Kubernetes, can be exposed in multiple ways. **The LoadBalancer method is recommended.** The NGINX ingress controller examples below are preserved for reference but the ingress-nginx project is deprecated upstream. Use a LoadBalancer or a Gateway API-compatible controller for new deployments. For tenant cluster endpoint planning, see [Gateway API](https://vcluster.com/docs/vcluster/key-features/gateway-api.md) and [Migrate from ingress-nginx to Gateway API](https://vcluster.com/docs/vcluster/manage/ingress-nginx-to-gateway-api.md). ### \[Deprecated]: External access through NGINX ingress controller[​](#deprecated-external-access-through-nginx-ingress-controller "Direct link to \[Deprecated]: External access through NGINX ingress controller") * Automatic or existing installation * Manual ingress controller installation Deprecated The ingress-nginx project has been deprecated. We recommend using a different ingress controller. Heads up. This section assumes you already have the nginx ingress controller installed, if you don't, check out the 'Manual Ingress Controller Installation' tab. 1. Run the command: Start the platform ``` vcluster platform start --host=vcluster-platform.mydomain.tld ``` Update the Domain. Make sure you update the hostname in the command. 2. Set the `VERSION` variable to the platform version you want to upgrade to or set it to the current version using: Set VERSION variable ``` CHART=$(kubectl get service loft -n vcluster-platform -o jsonpath={.metadata.labels.chart}) VERSION=${CHART:5} ``` 3. To upgrade the platform via vCluster CLI, update `$PLATFORM_VERSION` with a valid platform version and run: Upgrade the platform ``` vcluster platform start --upgrade --version=$PLATFORM_VERSION --values=vcluster-platform.yaml ``` Versions and Values The `$PLATFORM_VERSION` variable is an environment variable set to your desired platform version to deploy. The `vcluster-platform.yaml` file is an optional YAML file that contains Helm values you would like to use when upgrading your platform deployment. 4. Determine the External-IP address: Get the loft-ingress external IP address ``` kubectl get ingress -n vcluster-platform NAME CLASS HOSTS ADDRESS PORTS AGE loft-ingress vcluster-platform.mydomain.tld x.x.x.x 80, 443 10m ``` Address If you never get a valid IP Address in the ADDRESS column shown in the command, you might need to work with your Kubernetes administrator to ensure your ingress is properly configured. 5. [Set up a DNS A record](https://www.google.com/search?q=set+up+a+record+dns) to the ingress address. Make sure the platform is reachable at the address, you can check that with curl as shown below, or simply browse to the address in your browser of choice: Check the platform version ``` curl https://vcluster-platform.mydomain.tld/version --insecure | jq { "kind": "Version", "apiVersion": "version.loft.sh", "metadata": { "creationTimestamp": null }, "version": "v3.0.0", "major": "3", "minor": "0", "instance": "", "kubeVersion": "v1.24.2", "newerVersion": "", "shouldUpgrade": false } ``` Deprecated The ingress-nginx project has been deprecated. We recommend using a different ingress controller. 1. \[Deprecated]: Deploy `nginx-ingress` controller to your cluster. Update `$PLATFORM_VERSION` with a valid platform version. \[Deprecated- the ingress-nginx project has been deprecated]: Deploy ingress-nginx controller ``` helm upgrade --install ingress-nginx ingress-nginx \ --repository-config='' \ -n ingress-nginx \ --create-namespace \ --repo https://kubernetes.github.io/ingress-nginx \ --set-string controller.config.hsts=false \ --wait \ --version $PLATFORM_VERSION ``` 2. \[Deprecated]: To update the platform deployment to use the NGINX ingress controller, create a simple values file that tells the platform to do just that. If you created a values file during your initial deployment, you can simply edit that file, if you did not, you can create a new file with the following contents. \[Deprecated- the ingress-nginx project has been deprecated]: Update to use NGINX ingress controller ``` ingress: enabled: true host: "vcluster-platform.mydomain.tld" # Make sure to change this ingressClass: "nginx" # Optional, nginx is default value! ``` 3. To upgrade the platform via vCluster CLI, update `$PLATFORM_VERSION` with a valid platform version and run: Upgrade the platform ``` vcluster platform start --upgrade --version=$PLATFORM_VERSION --values=vcluster-platform.yaml ``` Versions and Values The `$PLATFORM_VERSION` variable is an environment variable set to your desired platform version to deploy. The `vcluster-platform.yaml` file is an optional YAML file that contains Helm values you would like to use when upgrading your platform deployment. 4. Determine the External-IP address: Get the loft-ingress external IP address ``` kubectl get ingress -n vcluster-platform NAME CLASS HOSTS ADDRESS PORTS AGE loft-ingress vcluster-platform.mydomain.tld x.x.x.x 80, 443 10m ``` Address If you never get a valid IP Address in the ADDRESS column shown in the command, you might need to work with your Kubernetes administrator to ensure your ingress is properly configured. 5. [Set up a DNS A record](https://www.google.com/search?q=set+up+a+record+dns) to the ingress address. Make sure the platform is reachable at the address, you can check that with curl as shown below, or simply browse to the address in your browser of choice: Check the platform version ``` curl https://vcluster-platform.mydomain.tld/version --insecure | jq { "kind": "Version", "apiVersion": "version.loft.sh", "metadata": { "creationTimestamp": null }, "version": "v3.0.0", "major": "3", "minor": "0", "instance": "", "kubeVersion": "v1.24.2", "newerVersion": "", "shouldUpgrade": false } ``` ### External access through LoadBalancer[​](#external-access-through-loadbalancer "Direct link to External access through LoadBalancer") 1. * AWS ELB + ACM * Other load balancers AWS Load Balancers If you are using AWS, make sure you are using a Network Load Balancer (NLB) to route traffic, since other load balancers do not support the SPDY protocol Kubernetes requires. Create a LoadBalancer Service ``` apiVersion: v1 kind: Service metadata: annotations: # Make sure to adjust the next line: service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:xxx:certificate/xxx" service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443" service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp" service.beta.kubernetes.io/aws-load-balancer-type: "nlb" name: vcluster-platform-loadbalancer namespace: vcluster-platform spec: type: LoadBalancer ports: - name: http port: 80 protocol: TCP targetPort: 8080 - name: https port: 443 protocol: TCP targetPort: 10443 selector: app: loft ``` Create a LoadBalancer Service ``` apiVersion: v1 kind: Service metadata: name: vcluster-platform-loadbalancer namespace: vcluster-platform spec: type: LoadBalancer ports: - name: https port: 443 targetPort: 10443 protocol: TCP selector: app: loft ``` 2. Create the load balancer with this command: Create the LoadBalancer Service ``` kubectl apply -f vcluster-platform-loadbalancer.yaml ``` 3. Wait until the load balancer receives an External-IP address: Get the vcluster-platform-loadbalancer external IP address ``` kubectl get svc vcluster-platform-loadbalancer -n vcluster-platform NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE vcluster-platform-loadbalancer LoadBalancer 10.112.2.142 x.x.x.x 443:30933/TCP 3m16s ``` 4. Make sure the platform is reachable at the load balancer's external address, you can check that with curl as shown below, or simply browse to the address in your browser of choice: Check the platform version ``` curl https://x.x.x.x/version --insecure | jq { "kind": "Version", "apiVersion": "version.loft.sh", "metadata": { "creationTimestamp": null }, "version": "v3.0.0", "major": "3", "minor": "0", "instance": "", "kubeVersion": "v1.24.2", "newerVersion": "", "shouldUpgrade": false } ``` ## Configure `TLS`[​](#configure-tls "Direct link to configure-tls") This section provides various options for configuring TLS for your platform deployment. Choose the method that best fits your infrastructure and requirements. ### Cert-Manager[​](#cert-manager "Direct link to Cert-Manager") 1. Install `cert-manager` to your cluster: Install cert-manager ``` helm upgrade --install cert-manager cert-manager --repository-config=''\ --namespace cert-manager --create-namespace \ --repo https://charts.jetstack.io \ --set installCRDs=true \ --wait ``` 2. Edit your existing `vcluster-platform.yaml` file, or create a new file named `vcluster-platform.yaml` with the following content: vcluster-platform.yaml ``` ingress: enabled: true annotations: # Make sure the following line matches the name of your issuer (or use the section below to create one) cert-manager.io/cluster-issuer: lets-encrypt-http-issuer tls: enabled: true secret: tls-vcluster-platform certIssuer: create: true # Change this if you already have your own cert-issuer name: lets-encrypt-http-issuer email: "YOUR_EMAIL" # REQUIRED secretName: vcluster-platform-letsencrypt-credentials httpResolver: enabled: true ingressClass: nginx # the ingress-nginx project has been deprecated resolvers: [] server: https://acme-v02.api.letsencrypt.org/directory ``` 3. Set the `VERSION` variable to the platform version you want to upgrade to or set it to the current version using: Set VERSION variable ``` CHART=$(kubectl get service loft -n vcluster-platform -o jsonpath={.metadata.labels.chart}) VERSION=${CHART:5} ``` 4. Upgrade the platform Upgrade considerations * Review the release notes for the target version to understand any breaking changes or new features. * Test the upgrade in a non-production environment before applying it to your production setup. Upgrade the platform using: * CLI * Helm To upgrade the platform using the `vcluster` CLI, update `$PLATFORM_VERSION` with a valid platform version and run: Upgrade the platform using vCluster CLI ``` RELEASE_NAMESPACE=vcluster-platform # Replace with the namespace of the platform deployment if different PLATFORM_VERSION='' # Set this to a specific version or leave empty for latest vcluster platform start --upgrade --version=$PLATFORM_VERSION --namespace=$RELEASE_NAMESPACE --values=vcluster-platform.yaml ``` To upgrade the platform using `helm`, update `$PLATFORM_VERSION` with a valid platform version and run: Upgrade the platform using helm ``` RELEASE_NAME=vcluster-platform # Replace with the release name if different RELEASE_NAMESPACE=vcluster-platform # Replace with the namespace of the platform deployment if different PLATFORM_VERSION='' # Set this to a specific version or leave empty for latest helm upgrade $RELEASE_NAME vcluster-platform -n $RELEASE_NAMESPACE --repository-config '' --repo https://charts.loft.sh \ --version $PLATFORM_VERSION \ --reuse-values \ -f vcluster-platform.yaml ``` Server-side apply with Helm 4 Helm 4 keeps the apply method a release was created with, so upgrades need no extra flags. If you switch a release to server-side apply with `--server-side=true`, the upgrade fails with a conflict on any field the platform has written itself. The most common one is `data.config` in the `loft-manager-config` secret after you edit the config in the UI. Add `--force-conflicts` to let Helm take ownership of those fields. Either way, keep the `config` section of your values file in sync with changes made in the UI. The upgrade replaces the secret with the values from the file. ### AWS Certificate Manager (ACM)[​](#aws-certificate-manager-acm "Direct link to AWS Certificate Manager (ACM)") * Domain through ingress * Domain through load balancer 1. Determine the External-IP address of your ingress: Get ingress external IP ``` kubectl get ingress -n vcluster-platform NAME CLASS HOSTS ADDRESS PORTS AGE loft-ingress vcluster-platform.mydomain.tld x.x.x.x 80, 443 10m ^^^^^^^ ``` 2. Find the AWS Elastic Load Balancer (ELB) for this IP address in the AWS console. 3. Switch to the "Listeners" tab. 4. In the "SSL Certificates" column, click on the "View/edit certificates" link. 5. Click on the "+" symbol next to the "Certificates" tab and add your Access Control Manager (ACM) managed certificate to the ingress controller's Load Balancer. Make sure to follow the [Load Balancer > AWS ELB + ACM](#external-access-through-loadbalancer) guide above. ### Manually provisioned certificate[​](#manually-provisioned-certificate "Direct link to Manually provisioned certificate") 1. Create a Kubernetes secret from your certificate: Create TLS secret ``` kubectl create secret generic tls-vcluster-platform -n vcluster-platform --type=kubernetes.io/tls \ --from-file=tls.crt=tls.crt \ --from-file=tls.key=tls.key ``` 2. Edit your existing `vcluster-platform.yaml` file, or create a new file named `vcluster-platform.yaml` with the following content: * Platform ingress handles TLS * Platform pod handles TLS * Load balancer handles TLS vcluster-platform.yaml for ingress TLS ``` ingress: tls: enabled: true secret: tls-vcluster-platform # Make sure this matches the name of your cert from the previous step ``` If you want to configure the platform with additionalCA, the Certificate Authority used for signing this certificate has to be added as a base64 encoded value: Encode CA certificate ``` cat ca.crt | base64 ``` Use this output in the `.additionalCA` helm value. vcluster-platform.yaml for pod TLS ``` tls: enabled: true secret: tls-vcluster-platform # Make sure this matches the name of your cert from the previous step ``` This must be configured outside of the platform deployment. 3. Set the `VERSION` variable to the platform version you want to upgrade to or set it to the current version using: Set VERSION variable ``` CHART=$(kubectl get service loft -n vcluster-platform -o jsonpath={.metadata.labels.chart}) VERSION=${CHART:5} ``` 4. Upgrade the platform Upgrade considerations * Review the release notes for the target version to understand any breaking changes or new features. * Test the upgrade in a non-production environment before applying it to your production setup. Upgrade the platform using: * CLI * Helm To upgrade the platform using the `vcluster` CLI, update `$PLATFORM_VERSION` with a valid platform version and run: Upgrade the platform using vCluster CLI ``` RELEASE_NAMESPACE=vcluster-platform # Replace with the namespace of the platform deployment if different PLATFORM_VERSION='' # Set this to a specific version or leave empty for latest vcluster platform start --upgrade --version=$PLATFORM_VERSION --namespace=$RELEASE_NAMESPACE --values=vcluster-platform.yaml ``` To upgrade the platform using `helm`, update `$PLATFORM_VERSION` with a valid platform version and run: Upgrade the platform using helm ``` RELEASE_NAME=vcluster-platform # Replace with the release name if different RELEASE_NAMESPACE=vcluster-platform # Replace with the namespace of the platform deployment if different PLATFORM_VERSION='' # Set this to a specific version or leave empty for latest helm upgrade $RELEASE_NAME vcluster-platform -n $RELEASE_NAMESPACE --repository-config '' --repo https://charts.loft.sh \ --version $PLATFORM_VERSION \ --reuse-values \ -f vcluster-platform.yaml ``` Server-side apply with Helm 4 Helm 4 keeps the apply method a release was created with, so upgrades need no extra flags. If you switch a release to server-side apply with `--server-side=true`, the upgrade fails with a conflict on any field the platform has written itself. The most common one is `data.config` in the `loft-manager-config` secret after you edit the config in the UI. Add `--force-conflicts` to let Helm take ownership of those fields. Either way, keep the `config` section of your values file in sync with changes made in the UI. The upgrade replaces the secret with the values from the file. ### Self-signed certificate[​](#self-signed-certificate "Direct link to Self-signed certificate") For testing or internal use, you can create and use a self-signed certificate. 1. Create a new private key for Certificate Authority: Generate CA private key ``` openssl genrsa 2048 > ca-key.pem ``` 2. Create a new certificate authority: Generate CA certificate ``` openssl req -new -x509 -nodes -days 365000 -key ca-key.pem -out ca.crt ``` 3. Create a new private key: Generate private key ``` openssl genrsa -out tls.key 4096 ``` 4. Create a file named `ssl.conf` with the following content: ssl.conf ``` [ req ] default_bits = 4096 distinguished_name = req_distinguished_name x509_extensions = v3_ca req_extensions = v3_req x509_extensions = usr_cert [ req_distinguished_name ] organizationName = Organization Name (for example company) organizationName_default = vcluster-platform commonName = Common Name (for example server FQDN or YOUR name) commonName_default = vcluster-platform.mydomain.tld [ usr_cert ] basicConstraints = CA:FALSE nsCertType = client, server keyUsage = digitalSignature extendedKeyUsage = serverAuth, clientAuth [ v3_req ] subjectAltName = @alt_names extendedKeyUsage = serverAuth, clientAuth basicConstraints = CA:FALSE keyUsage = digitalSignature [ alt_names ] DNS.1 = localhost ``` Set `commonName_default` and `DNS.1` to the domain you want to use. 5. Create a certificate signing request: Generate certificate signing request ``` openssl req -new -sha256 \ -out tls.csr \ -key tls.key \ -config ssl.conf ``` 6. Generate the certificate: Generate certificate ``` openssl x509 -req \ -sha256 \ -days 3650 \ -in tls.csr \ -out tls.crt \ -extensions v3_req \ -extfile ssl.conf \ -CA ca.crt \ -CAkey ca-key.pem ``` 7. Create a Kubernetes secret from your certificate: Create TLS secret ``` kubectl create secret generic tls-vcluster-platform -n vcluster-platform --type=kubernetes.io/tls \ --from-file=tls.crt=tls.crt \ --from-file=tls.key=tls.key ``` If you want to configure the platform with additionalCA, use: Encode CA certificate ``` cat ca.crt | base64 ``` as an input for the `.additionalCA` helm value. 8. Edit your existing `vcluster-platform.yaml` file, or create a new file named `vcluster-platform.yaml` with the following content: * Platform ingress handles TLS * Platform pod handles TLS * Load balancer handles TLS vcluster-platform.yaml for ingress TLS ``` ingress: tls: enabled: true secret: tls-vcluster-platform # Make sure this matches the name of your cert from the previous step tls: enabled: true # Virtual clusters running on the same host as the platform directly use the platform service. secret: tls-vcluster-platform # Ensure this matches the name of your cert from the previous step ``` vcluster-platform.yaml for pod TLS ``` tls: enabled: true secret: tls-vcluster-platform # Make sure this matches the name of your cert from the previous step ``` This must be configured outside of the platform deployment. 9. Set the `VERSION` variable to the platform version you want to upgrade to or set it to the current version using: Set VERSION variable ``` CHART=$(kubectl get service loft -n vcluster-platform -o jsonpath={.metadata.labels.chart}) VERSION=${CHART:5} ``` 10. Upgrade the platform Upgrade considerations * Review the release notes for the target version to understand any breaking changes or new features. * Test the upgrade in a non-production environment before applying it to your production setup. Upgrade the platform using: * CLI * Helm To upgrade the platform using the `vcluster` CLI, update `$PLATFORM_VERSION` with a valid platform version and run: Upgrade the platform using vCluster CLI ``` RELEASE_NAMESPACE=vcluster-platform # Replace with the namespace of the platform deployment if different PLATFORM_VERSION='' # Set this to a specific version or leave empty for latest vcluster platform start --upgrade --version=$PLATFORM_VERSION --namespace=$RELEASE_NAMESPACE --values=vcluster-platform.yaml ``` To upgrade the platform using `helm`, update `$PLATFORM_VERSION` with a valid platform version and run: Upgrade the platform using helm ``` RELEASE_NAME=vcluster-platform # Replace with the release name if different RELEASE_NAMESPACE=vcluster-platform # Replace with the namespace of the platform deployment if different PLATFORM_VERSION='' # Set this to a specific version or leave empty for latest helm upgrade $RELEASE_NAME vcluster-platform -n $RELEASE_NAMESPACE --repository-config '' --repo https://charts.loft.sh \ --version $PLATFORM_VERSION \ --reuse-values \ -f vcluster-platform.yaml ``` Server-side apply with Helm 4 Helm 4 keeps the apply method a release was created with, so upgrades need no extra flags. If you switch a release to server-side apply with `--server-side=true`, the upgrade fails with a conflict on any field the platform has written itself. The most common one is `data.config` in the `loft-manager-config` secret after you edit the config in the UI. Add `--force-conflicts` to let Helm take ownership of those fields. Either way, keep the `config` section of your values file in sync with changes made in the UI. The upgrade replaces the secret with the values from the file. ## Import tenant clusters with certificate validation[​](#importing-existing-tenant-clusters "Direct link to Import tenant clusters with certificate validation") When importing externally managed tenant clusters, you can enable secure communication by using the same certificate authority configured for the platform. 1. Get the current certificate authority value from your platform: Get the certificate authority value ``` helm get values --namespace $RELEASE_NAMESPACE $RELEASE_NAME --all | grep additionalCA ``` 2. Import the tenant cluster with certificate validation: Import a tenant cluster with certificate validation ``` vcluster platform add vcluster \ --namespace= \ --project= \ --ca-data ``` The imported tenant cluster restarts and reports `Ready` once it establishes a secure connection to the platform. --- # Overview Installation options are the content in the [`values.yaml`](https://vcluster.com/docs/platform/configure/introduction.md) except the content under the `config` section and the `agentValues` section. It contains fields like `resources`, `replicas`, `ingress` and also custom fields like `admin`, `product`, `agentOnly`. You set values of installation options to customize the deployment of vCluster Platform. These installation options are not available in the vCluster Platform UI after the vCluster Platform is installed in the control plane cluster but can only be [applied using `helm`](https://vcluster.com/docs/platform/configure/introduction.md#applying-configuration) before the deployment. ## Installation modes[​](#installation-modes "Direct link to Installation modes") Both **vCluster Platform** and **vCluster Platform Agent** are installed using the same Helm chart: `vcluster-platform`. The installation mode is controlled by the `agentOnly` value in the `values.yaml`: ``` agentOnly: false # Installs the vCluster Platform (control plane). agentOnly: true # Installs the vCluster Platform Agent. ``` --- # Introduction **vCluster Platform** is deployed on a Kubernetes cluster that acts as the primary Control Plane Cluster. Additional clusters can connect to this Platform instance and be managed by it. In this architecture: * The primary cluster runs vCluster Platform. * Additional Control Plane Clusters connect to the primary cluster and run the vCluster Platform agent. * Platform coordinates and manages all connected clusters through their agents. * The primary cluster also runs the vCluster Platform agent, so Platform manages it the same way as any other connected cluster. The `values.yaml` of `vcluster-platform` Helm chart in `loft-sh` repository contains the configuration values when installing or upgrading vCluster Platform or vCluster Platform Agent. Both vCluster Platform and vCluster Platform Agent are using the `vcluster-platform` Helm chart. By setting values in `values.yaml`, you can choose to install or upgrade vCluster Platform or vCluster Platform Agent. You can also control the post-deployment platform behavior and the behavior of agents on other connected clusters. ## Values sections[​](#values-sections "Direct link to Values sections") The platform's `values.yaml` file contains multiple configuration sections: 1. **Installation options** - Control how the platform deploys * Top-level fields in the `vcluster-platform` chart's `values.yaml`. These configure Kubernetes deployment aspects like resources and replicas. * Located in the `vcluster-platform` Helm chart ([loft-sh/loft](https://github.com/loft-sh/loft/tree/master/chart)). * Examples: `replicaCount`, `resources`, `ingress`, `persistence`, etc. 2) **Platform configs** - Control how the platform functions after deployment. You can also manage these through the UI in **Admin > Config**. * These are grouped under the `config:` section of the `vcluster-platform` chart's `values.yaml`. * Located in the `vcluster-platform` Helm chart ([loft-sh/loft](https://github.com/loft-sh/loft/tree/master/chart)). * Examples: `config.auth`, `config.audit`, `config.loftHost`. 3. **Agent settings** - Control how the platform agent operates on connected clusters * Set through the top-level `agentValues:` section of the `vcluster-platform` chart. You can override these per-cluster with annotations. * Located in the `vcluster-platform` Helm chart ([loft-sh/loft](https://github.com/loft-sh/loft/tree/master/chart)). * Examples: `replicaCount`, `resources`, `ingress`, `persistence`, etc. The following is an example of `vcluster-platform` Helm chart `values.yaml` file showing the configuration sections: Example platform values.yaml with all configuration sections ``` # Installation options (top-level) replicaCount: 2 resources: requests: memory: 512Mi cpu: 400m admin: create: true username: my-admin-user password: my-secure-password ingress: enabled: true host: platform.example.com # Platform configs (inside config section) config: loftHost: platform.example.com audit: enabled: true auth: github: clientId: $CLIENT_ID # Agent settings (top-level) agentValues: resources: requests: memory: 256Mi # Security contexts for agent pods securityContext: runAsNonRoot: true runAsUser: 1001 podSecurityContext: fsGroup: 2001 ``` ## Common configuration fields[​](#common-configuration-tasks "Direct link to Common configuration fields") For common tasks, configure the following: | Task | Configuration location | Example | | --------------------------- | ------------------------------------ | ---------------------------------- | | Setting admin credentials | `admin:` section (root level) | `admin.username`, `admin.password` | | Configuring replicas for HA | `replicaCount:` setting (root level) | `replicaCount: 2` | | Setting up ingress | `ingress:` section (root level) | `ingress.enabled: true` | | Setting resource limits | `resources:` section (root level) | `resources.limits.memory` | | Configuring auth providers | `config.auth:` section | `config.auth.github` | | Setting custom domain | `config:` section | `config.loftHost` | ## Apply configuration[​](#applying-configuration "Direct link to Apply configuration") There are two ways to apply the `values.yaml` content: * Through `helm` values when installing or upgrading the platform or agent. * Through the vCluster Platform UI in **Admin > Config** after installation. This is only for platform configs. Using `helm` values allows you to manage configuration declaratively and enables deployment through [GitOps solutions](https://vcluster.com/docs/platform/install/gitops.md) such as ArgoCD. Here's an example of applying platform configuration with `helm`: Modify the following with your specific values to generate a copyable command: RELEASE\_NAMEvcluster-platform RELEASE\_NAMESPACEvcluster-platform PLATFORM\_VERSION4.12.0 Apply platform configuration using Helm ``` helm upgrade vcluster-platform vcluster-platform \ --install \ --repo https://charts.loft.sh/ \ --namespace vcluster-platform \ --create-namespace \ --values values.yaml \ --version 4.12.0 ``` Reconnect clusters after key value changes After changing `config.loftHost`, `additionalCA`, or `insecureSkipVerify`, you must reconnect all clusters to the platform. Run the same connection commands in each Kubernetes context. See [connecting clusters](https://vcluster.com/docs/platform/administer/clusters/connect-cluster.md) for instructions. --- # Audit Logging | | | Enterprise | | | | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Audit Logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). ## Overview[​](#overview "Direct link to Overview") Audit logging in the platform provides a security-relevant, chronological set of records documenting the sequence of actions. It audits activities generated by users and applications that use the platform API. The platform can log activities related to: * Management instance changes, such as creation of new tenant clusters, spaces, and so on. * Changes within a tenant cluster or space * Changes within a connected cluster Auditing in the platform is similar to [auditing Kubernetes clusters](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/) in general. Each request on each stage of its execution generates an audit event, which is then pre-processed according to a certain policy and written to a backend (currently only log backends are supported). The policy determines what's recorded and the backends persist the records. Each request can be recorded with an associated stage. The defined stages are: * `RequestReceived`: The stage for events generated as soon as the audit handler receives the request, and before it is delegated down the handler chain. * `ResponseComplete`: The response body has been completed and no more bytes are sent. * `Panic`: Events generated when a panic occurred. The audit logging feature increases the memory consumption of the platform because some context required for auditing is stored for each request. Memory consumption depends on the audit logging configuration. ## Enable auditing[​](#enable-auditing "Direct link to Enable auditing") Two separate audit fields `values.yaml` has two distinct `audit` fields with different purposes: * **`audit`** (top-level Helm value) — configures the audit sidecar and log persistence (PVC) for the platform pod. This is a Helm-only setting. Changes require a Helm upgrade and a platform restart. * **`config.audit`** (under the `config` section) — controls audit recording and policy, and enables the Audit Log UI. This can be set in **Platform > Platform Config > Config** in the UI or under `config` in `values.yaml`. `config.audit.enabled` defaults to `true`, so audit events are recorded without any additional configuration. Changes to `config.audit` made through the UI take effect automatically within about 10-15 seconds, no restart required. Changing `config.audit` through a Helm upgrade also restarts the platform pod, since the chart stamps the pod template with a config hash. The setting itself still takes effect through the same hot-reload mechanism either way. Configure auditing through the platform config in the platform UI (**Platform > Platform Config > Config**). Content added in the UI goes automatically under the `config` section of `values.yaml`. An example configuration could look like: Example audit configuration in platform UI ``` audit: enabled: true level: 1 ``` See the [complete list of fields](https://vcluster.com/docs/platform/api/resources/config.md#status-audit) under the `config.audit` section of `values.yaml`. ## Audit levels[​](#audit-levels "Direct link to Audit levels") The platform provides audit levels, which are preconfigured audit policies for the most common use cases. These levels range from 1 to 4 where 1 logs the fewest requests, while 4 logs the most: * **Level 1**: Logs modifying requests such as creation, modification, or deletion of any objects. Creation requests also log the request payload. * **Level 2**: Like Level 1 but also logs the metadata of reading requests, such as listing pods inside a tenant cluster or space. Reading requests won't log the response or request payload and instead only the metadata, such as request origin and target. * **Level 3**: Like Level 2 but instead of only logging the request metadata, also logs the complete request payload sent to the platform. * **Level 4**: Like Level 3 but instead of only logging metadata and request payload, also logs the response the platform has sent to the requester. Captured payloads are normalized to JSON and size-limited before storage. Unconvertible or oversized payloads are omitted with an annotation. See [Request and response payload storage](#request-and-response-payload-storage). Every level also applies a fixed set of exclusions and downgrades for internal, high-volume, or credential-carrying requests, on top of the level's own rules. See [Default policy exclusions](#default-policy-exclusions) for what's affected and why. A [custom audit policy](#audit-policy) is the only way to change this behavior, and it replaces these defaults entirely rather than layering on top of them. Configure the audit level through the platform config, which can be modified either through the platform UI or Helm: * UI * Helm 1. Go to the Platform > Platform Config > Config view using the menu on the left. 2. In the input field that appears, enter the following config: ``` audit: enabled: true level: 1 ``` 3. Click on the Apply button. The change takes effect automatically within about 10-15 seconds, with no platform restart required. Create a new file `values.yaml`: values.yaml ``` config: audit: enabled: true level: 1 ``` Then apply these values with Helm (make sure to specify the correct platform version): Apply helm values ``` helm upgrade loft vcluster-platform --namespace vcluster-platform \ --repo https://charts.loft.sh \ --version $LOFT_VERSION \ --reuse-values \ --values values.yaml ``` ## Request and response payload storage[​](#request-and-response-payload-storage "Direct link to Request and response payload storage") Audit levels and custom policies both determine payload storage through one of four granularity settings. `Request` stores the request payload. `RequestResponse` also stores the response payload. See [Audit policy](#audit-policy) for the full list. The platform stores payloads for resource requests only. Long-running requests, such as log streaming, never store payloads. Creates of sensitive resources, such as shared secrets and access keys, keep metadata-only events on every level. See [Default policy exclusions](#default-policy-exclusions) for the full list of affected resources and why. The platform always stores payloads as JSON: * Payloads that arrive as plain JSON are stored unchanged. * Gzip-compressed request and response bodies are stored in their decompressed form. * Payloads sent in the Kubernetes Protobuf format are converted to their JSON equivalent for known Kubernetes API types. The stored payload matches what a JSON client would have sent. ### Payload size limit[​](#payload-size-limit "Direct link to Payload size limit") The platform captures at most 2 MiB per request or response payload. It doesn't store payloads above this limit. ### Omitted payloads[​](#omitted-payloads "Direct link to Omitted payloads") When the platform can't store a payload, it keeps the audit event with all of its metadata but leaves out the payload. The event then carries an annotation that names the omission reason: * `audit.platform.vcluster.com/request-object-omitted` for the request payload * `audit.platform.vcluster.com/response-object-omitted` for the response payload The following reasons can appear as annotation values: | Reason | Description | | ---------------------- | ------------------------------------------------------------------------------------------ | | `truncated` | The payload exceeded the capture limit, so the platform captured only a partial copy. | | `too-large` | The payload exceeded the capture limit after decompression or conversion to JSON. | | `decode-failed` | The platform couldn't decompress or decode the payload. | | `unsupported-encoding` | The payload used a content encoding or media type that the platform can't convert to JSON. | | `invalid-json` | The payload wasn't valid JSON and not in any recognized format. | ## Audit policy[​](#audit-policy "Direct link to Audit policy") Audit policy is a more complex alternative to audit levels, letting you define exact rules about what events should be recorded and what data they should include. When an event is processed, it's compared against the list of rules in order. The first matching rule sets the audit granularity of the event. The defined audit granularity options are: * `None`: Don't log events that match this rule. * `Metadata`: Log request metadata (requesting user, timestamp, resource, verb, and so on) but not request or response body. * `Request`: Log event metadata and request body but not response body. This doesn't apply for non-resource requests. * `RequestResponse`: Log event metadata, and request and response bodies. This doesn't apply for non-resource requests. Custom rules replace the defaults entirely Setting a non-empty `policy.rules` list under `config.audit` replaces the resolved default policy outright, and the platform then ignores the `level` field. None of the [default policy exclusions](#default-policy-exclusions) apply automatically. Start from the [default rules for a level](#default-policy-rules-by-level) and add to them, rather than writing a custom policy from scratch. An empty `rules: []` list doesn't count as a custom policy: the platform still resolves and applies the configured level's defaults. An example policy that catches all requests: Example audit policy in values.yaml ``` config: audit: enabled: true policy: rules: - level: Metadata omitStages: - RequestReceived ``` See the [complete policy reference](https://vcluster.com/docs/platform/api/resources/config.md#status-audit-policy). ## Default policy exclusions[​](#default-policy-exclusions "Direct link to Default policy exclusions") Before a request reaches an audit level's own rules, three built-in rules run first, in this order. Because the platform uses the first matching rule, these built-ins take priority over anything the level would otherwise log. They apply to all four audit levels. ### Never logged[​](#never-logged "Direct link to Never logged") The platform never logs these requests, at any level, including level 4: | Resource | API group | Why | | ------------------------------------------------------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `selfsubjectaccessreviews`, `subjectaccessreviews` | any | Permission checks the UI and CLI issue continuously to decide what to show. They aren't an action on a resource, and logging them would flood the audit log. | | `agentauditevents` | `management.loft.sh` | The endpoint agents use to send their own audit events back to the platform (see [Audit logging for Regional Cluster Endpoints](#audit-logging-for-regional-cluster-endpoints)). Logging it would create an audit event about receiving an audit event. | | `licensetokens`, `directclusterendpointtokens`, `ingressauthtokens` | `management.loft.sh` | Internal tokens validated on nearly every request. | | `selves` | `management.loft.sh` | The endpoint the UI and CLI call to resolve the current user, requested on nearly every page load. | | `virtualclusterinstances/kubeconfig` | `management.loft.sh` | Retrieving the kubeconfig for a tenant cluster. | No audit record of kubeconfig retrieval Because `virtualclusterinstances/kubeconfig` is on the never-logged list, there's no audit record of who retrieved a tenant cluster's kubeconfig, or when, at any level. To get a `Metadata` record of these requests instead (who, when, which tenant cluster, without the returned credentials), remove `virtualclusterinstances/kubeconfig` from the never-logged resources in a [custom policy built from the defaults](#default-policy-rules-by-level). The [sensitive data rule](#metadata-only-for-sensitive-data) then takes over and audits it at `Metadata`, since it already lists this resource. ### Metadata-only for sensitive data[​](#metadata-only-for-sensitive-data "Direct link to Metadata-only for sensitive data") For `create` requests only, the platform always logs these resources at `Metadata`, never `Request` or `RequestResponse`, regardless of the configured level: | Resource | API group | Why | | --------------------------------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ownedaccesskeys`, `resetaccesskeys` | `management.loft.sh` | The response contains the access key value. | | `sharedsecrets` | `management.loft.sh` | The request can contain the secret data. | | `clusters` | `management.loft.sh` | Cluster registration includes a kubeconfig or connection token in the request. | | `users/profile` | `management.loft.sh` | The request can contain a new password. | | `virtualclusterinstances/kubeconfig` | `management.loft.sh` | The response contains a working kubeconfig. This rule only applies if you remove the resource from the never-logged list above, since that rule matches first. | | `virtualclusterinstances/snapshotcredentials` | `management.loft.sh` | The response contains sensitive storage credentials. | This keeps the event auditable (who did it, when, on which object) without writing the credential itself into the audit log, which is typically retained longer and read more widely than the underlying secret store. ### Metadata-only for long-running requests[​](#metadata-only-for-long-running-requests "Direct link to Metadata-only for long-running requests") | Resource | API group | Why | | ------------------------------------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `pods/log`, `pods/exec`, `pods/portforward` | core (`""`) | Streaming requests inside a tenant cluster or connected cluster. There's no discrete response body to capture, and exec or port-forward would mean logging the interactive session traffic. | | `virtualclusterinstances/log`, `appinstances/log` | `management.loft.sh` | Same reasoning, for log streaming through the platform API. | ## Default policy rules by level[​](#default-policy-rules-by-level "Direct link to Default policy rules by level") Each audit level resolves to a fixed list of policy rules, evaluated in order, where the first match sets an event's granularity. Copy the rules for a level below as a starting point for a custom policy, then add, remove, or reorder rules to fit your needs. All four levels start with the same three rules, covered in [Default policy exclusions](#default-policy-exclusions), followed by level-specific rules for everything else. * Level 1 * Level 2 * Level 3 * Level 4 Level 1 default policy ``` config: audit: policy: rules: - level: None resources: - group: "*" resources: - selfsubjectaccessreviews - subjectaccessreviews - group: management.loft.sh resources: - agentauditevents - licensetokens - directclusterendpointtokens - ingressauthtokens - selves - virtualclusterinstances/kubeconfig - level: Metadata omitStages: - RequestReceived verbs: - create resources: - group: management.loft.sh resources: - ownedaccesskeys - resetaccesskeys - sharedsecrets - clusters - users/profile - virtualclusterinstances/kubeconfig - virtualclusterinstances/snapshotcredentials - level: Metadata omitStages: - RequestReceived resources: - group: "" resources: - pods/log - pods/exec - pods/portforward - group: management.loft.sh resources: - virtualclusterinstances/log - appinstances/log - level: None verbs: - get - list - watch - level: Request omitStages: - RequestReceived verbs: - create - level: Metadata omitStages: - RequestReceived ``` Level 2 default policy ``` config: audit: policy: rules: - level: None resources: - group: "*" resources: - selfsubjectaccessreviews - subjectaccessreviews - group: management.loft.sh resources: - agentauditevents - licensetokens - directclusterendpointtokens - ingressauthtokens - selves - virtualclusterinstances/kubeconfig - level: Metadata omitStages: - RequestReceived verbs: - create resources: - group: management.loft.sh resources: - ownedaccesskeys - resetaccesskeys - sharedsecrets - clusters - users/profile - virtualclusterinstances/kubeconfig - virtualclusterinstances/snapshotcredentials - level: Metadata omitStages: - RequestReceived resources: - group: "" resources: - pods/log - pods/exec - pods/portforward - group: management.loft.sh resources: - virtualclusterinstances/log - appinstances/log - level: Request omitStages: - RequestReceived verbs: - create - level: Metadata omitStages: - RequestReceived resources: - group: "*" resources: - "*" ``` Level 3 default policy ``` config: audit: policy: rules: - level: None resources: - group: "*" resources: - selfsubjectaccessreviews - subjectaccessreviews - group: management.loft.sh resources: - agentauditevents - licensetokens - directclusterendpointtokens - ingressauthtokens - selves - virtualclusterinstances/kubeconfig - level: Metadata omitStages: - RequestReceived verbs: - create resources: - group: management.loft.sh resources: - ownedaccesskeys - resetaccesskeys - sharedsecrets - clusters - users/profile - virtualclusterinstances/kubeconfig - virtualclusterinstances/snapshotcredentials - level: Metadata omitStages: - RequestReceived resources: - group: "" resources: - pods/log - pods/exec - pods/portforward - group: management.loft.sh resources: - virtualclusterinstances/log - appinstances/log - level: Request omitStages: - RequestReceived resources: - group: "*" resources: - "*" ``` Level 4 default policy ``` config: audit: policy: rules: - level: None resources: - group: "*" resources: - selfsubjectaccessreviews - subjectaccessreviews - group: management.loft.sh resources: - agentauditevents - licensetokens - directclusterendpointtokens - ingressauthtokens - selves - virtualclusterinstances/kubeconfig - level: Metadata omitStages: - RequestReceived verbs: - create resources: - group: management.loft.sh resources: - ownedaccesskeys - resetaccesskeys - sharedsecrets - clusters - users/profile - virtualclusterinstances/kubeconfig - virtualclusterinstances/snapshotcredentials - level: Metadata omitStages: - RequestReceived resources: - group: "" resources: - pods/log - pods/exec - pods/portforward - group: management.loft.sh resources: - virtualclusterinstances/log - appinstances/log - level: RequestResponse omitStages: - RequestReceived resources: - group: "*" resources: - "*" ``` ## Persist audit logs[​](#persist-audit-logs "Direct link to Persist audit logs") Persist platform audit logs in one of two ways: deploy the platform with a persistent volume claim (PVC), or connect the platform to a persistent database. HA only supported with persistent database If you are running the platform in high availability (HA mode), the only way to persist audit logs is to use a persistent database. The PVC approach doesn't work. ### Deploy the platform with a PVC to save audit logs[​](#deploy-the-platform-with-a-pvc-to-save-audit-logs "Direct link to Deploy the platform with a PVC to save audit logs") Create a new `values.yaml` with the following values: values.yaml for PVC ``` audit: persistence: enabled: true # size: 30Gi ``` This is the top-level `audit` field in `values.yaml`, not `config.audit` (see [Enable auditing](#enable-auditing) for the distinction). Apply the values using Helm: Apply Helm values ``` helm upgrade vcluster-platform vcluster-platform \ --namespace vcluster-platform \ --repo https://charts.loft.sh \ --reuse-values \ -f values.yaml ``` ### Use a persistent database as platform audit backend[​](#use-a-persistent-database-as-platform-audit-backend "Direct link to Use a persistent database as platform audit backend") Instead of the internal sqlite storage (see [View and export audit logs](#view-and-export-audit-logs)), the platform can write audit events to a persistent MySQL or PostgreSQL database. This is a `config.audit` setting, so you can configure it through the platform UI or Helm. Through the UI, go to **Platform > Platform Config > Config** and specify the following platform config setting: Platform config for persistent database ``` audit: dataStoreEndpoint: mysql://username:password@tcp(hostname:3306)/database-name ``` Through Helm, add the following to `values.yaml`: values.yaml for persistent database ``` config: audit: enabled: true dataStoreEndpoint: mysql://username:password@tcp(hostname:3306)/database-name ``` For PostgreSQL, use a `postgres://` endpoint instead: values.yaml for a PostgreSQL audit datastore ``` config: audit: enabled: true dataStoreEndpoint: postgres://username:password@hostname:5432/database-name ``` Apply the changes. Through the UI, they take effect automatically within about 10-15 seconds, with no platform restart required. Through Helm, `helm upgrade` restarts the platform pod as a side effect of the upgrade, though the setting itself takes effect through the same hot-reload mechanism either way. #### IAM authentication instead of a static password[​](#iam-authentication-instead-of-a-static-password "Direct link to IAM authentication instead of a static password") Set `config.audit.dataStoreIdentityProvider: aws` to authenticate with a short-lived IAM token instead of a static password. This works against a datastore with IAM database authentication enabled, such as RDS or Aurora. `aws` is the only supported value today, and it only applies to MySQL and PostgreSQL, not sqlite. values.yaml for IAM-authenticated audit datastore ``` config: audit: enabled: true dataStoreEndpoint: mysql://iam-user@tcp(hostname:3306)/database-name dataStoreIdentityProvider: aws ``` Omit the password from `dataStoreEndpoint` when using IAM authentication. The platform mints a new token periodically to stay ahead of expiry. #### Verify the datastore with a CA file[​](#verify-the-datastore-with-a-ca-file "Direct link to Verify the datastore with a CA file") Set `config.audit.dataStoreCAFile` to the path of a PEM-encoded CA bundle to verify the datastore's TLS certificate. The platform reads this file directly, so mount the Secret or ConfigMap containing it into the platform pod through the chart's top-level `volumes` and `volumeMounts`. This is the same approach as [`config.database`'s `caFile`](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/.md): values.yaml for a CA-verified audit datastore ``` volumes: - name: audit-datastore-ca secret: secretName: audit-datastore-ca volumeMounts: - name: audit-datastore-ca mountPath: /etc/vcluster/audit-datastore-ca readOnly: true config: audit: enabled: true dataStoreEndpoint: postgres://username:password@hostname:5432/database-name dataStoreCAFile: /etc/vcluster/audit-datastore-ca/ca.pem ``` The platform reads `dataStoreCAFile` once at startup. Rotating the mounted CA requires a platform pod restart to take effect. TLS precedence An explicit `sslmode=` (PostgreSQL) or `tls=` (MySQL) query parameter already present on `dataStoreEndpoint` always wins over `dataStoreCAFile`, silently. Remove the query parameter if you want `dataStoreCAFile` to take effect. Without an explicit override: * PostgreSQL defaults to `sslmode=require` (encrypted, but the server certificate isn't verified). Setting `dataStoreCAFile` upgrades this to `sslmode=verify-full`. * MySQL has no TLS by default, unless `dataStoreIdentityProvider` is set, in which case it defaults to `tls=skip-verify`. Setting `dataStoreCAFile` upgrades either case to full certificate verification. `dataStoreCAFile` applies regardless of `dataStoreIdentityProvider`, and has no effect on the internal sqlite backend. ## View and export audit logs[​](#view-and-export-audit-logs "Direct link to View and export audit logs") By default, the platform logs audit events to the following locations: * To a log file in JSON format located at `/var/lib/loft/audit.log` inside the platform container. Each line inside the log represents a single audit event. * To an internal sqlite storage located at `/var/lib/loft/audit.db` inside the platform container. This sqlite database is used to display audit log events in the platform UI. By default audit events in the sqlite are not persisted, so restarting the platform clears the database. See [Use a persistent database as platform audit backend](#use-a-persistent-database-as-platform-audit-backend) to configure a persistent MySQL backend instead. ### Enable audit sidecar[​](#enable-audit-sidecar "Direct link to Enable audit sidecar") To export audit events to third-party systems, enable the audit log sidecar. It prints all audit events to `stdout` in a separate container, so you can watch and export them from there. You can only enable the sidecar through Helm values. Create a `values.yaml` with the following contents: values.yaml for audit sidecar ``` audit: enableSideCar: true ``` This is the top-level `audit` field in `values.yaml`, not `config.audit`, so it cannot be set in **Platform > Platform Config > Config** (see [Enable auditing](#enable-auditing) for the distinction). Update the Helm release: Update Helm release ``` helm upgrade vcluster-platform vcluster-platform \ --namespace vcluster-platform \ --repo https://charts.loft.sh \ --reuse-values \ --values values.yaml ``` Wait until the platform has restarted, then view the audit logs: View audit logs ``` kubectl logs -n vcluster-platform -l app=loft -c audit -f ``` ## Audit logging for Regional Cluster Endpoints[​](#audit-logging-for-regional-cluster-endpoints "Direct link to Audit logging for Regional Cluster Endpoints") [Regional Cluster Endpoints](https://vcluster.com/docs/platform/administer/clusters/advanced/regional-cluster-endpoints.md) allow direct communication between users and clusters, bypassing the central platform instance. When this feature is enabled, the platform audit configuration is synced to each agent. Each platform agent propagates audit events it receives back to the central platform instance, which then logs it as a regular audit event. These "propagated" events can be identified by the `annotations.audit.loft.sh/sent-by-agent` identifier in an audit event. To disable this behavior, set the audit config option `disableAgentSyncBack`. --- # Cost Control Dashboard The platform comes with the cost control dashboard enabled by default, offering insights into potential [cost savings](https://www.vcluster.com/cost-savings) through tenant clusters. info This feature is available from the **Platform** version v4.2.0 and was introduced in **vCluster** version v0.22.0 To track allocations and calculate savings for workloads running inside tenant clusters the platform deploys and manages [Prometheus](https://prometheus.io/) and [OpenCost](https://www.opencost.io/) on each connected cluster. Prometheus uses a time series database that requires persistent volume storage to retain metrics over time. The platform-managed Prometheus is configured to collect the minimum metrics required for the cost control dashboard. OpenCost provides CPU and Memory allocation metrics for the deployed workloads. In addition to the Prometheus instance on the connected cluster, the cluster hosting the platform has a global Prometheus installed that is configured to aggregate metrics from all connected clusters. This utilizes the connected cluster private network so no additional networking configuration is required. Storage requirements By default, the Cost Control Dashboard requests 60Gi persistent volumes for each connected cluster's metrics, and an additional 60Gi persistent volume for global metric aggregation. See the [Configuration](#configuration) section for details on enabling/disabling the dashboard or configuring its resource needs. Network requirements In air-gapped or private GKE clusters it may be necessary to add a firewall rule allowing incoming traffic to port `9090`. This port is used when aggregating metrics from connected clusters. ## Configuration[​](#configuration "Direct link to Configuration") Although the platform manages Prometheus and OpenCost automatically, certain settings can be configured to better suit specific environments or needs. The primary configuration is part of the [platform `constControl` config](https://vcluster.com/docs/platform/configure/platform-configs/cost-control.md). To turn off the cost control dashboard, which is enabled by default, set `costControl.enabled` to `false` in platform configs: Turning off the Cost Control Dashboard in values.yaml ``` config: costControl: enabled: false ``` The remaining configuration customizes the cost settings, global Prometheus, and connected cluster OpenCost and Prometheus. These are detailed in the following sections. ### Cost settings[​](#cost-settings "Direct link to Cost settings") The platform provides initial cost defaults as shown below. These can be customized for your environment. Default Cost Settings in values.yaml ``` config: costControl: settings: # Average CPU cost averageCPUPricePerNode: price: 31 timePeriod: Monthly # Average Memory cost averageRAMPricePerNode: price: 31 timePeriod: Monthly # Monthly or Yearly cost for Hosted Control Planes controlPlanePricePerCluster: price: 900 timePeriod: Yearly ``` ### Global observability[​](#global-observability "Direct link to Global observability") These settings control, the global Prometheus's availability, retention, storage, CPU, and memory configuration. The default settings are provided below for reference, and may be customized. Default Global Prometheus Settings in values.yaml ``` config: costControl: global: metrics: # Number of replicas for the Prometheus Deployment replicas: 1 # Prometheus TSDB retention settings: https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects retention: 1y # CPU and Memory Requests and Limits for the Prometheus Deployment resources: requests: cpu: 500m memory: 6Gi limits: cpu: '2' memory: 6Gi # Storage Settings for Persistent Volumes storage: size: 60Gi # Valid values: # - null: use the default storage class # - "": disable automatic storage provisioning # - name of the preferred configured storage class storageClass: null ``` `retention` refers to Prometheus's TSDB retention settings. Details regarding valid units can be found in the [prometheus documentation](https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects) `storageClass` configures the storage class that should be used for provisioning of the Prometheus deployment's StatefulSet. Valid values are: * `null`: use the cluster's default storage class * `""` (empty string): turn off automatic provisioning * The name of a pre-configured storage class on the cluster Overriding Resources Overridden `resources` are not merged with the defaults to avoid unexpected changes from Helm charts used by the platform to manage Prometheus. Therefore, all requests and limits that you wish to apply should be provided. ### Cluster observability[​](#cluster-observability "Direct link to Cluster observability") These settings control the Prometheus's configuration for each connected cluster. These are applied uniformly across multiple clusters, but can be customized using [Cluster Specific Configuration](#other-cluster-settings). Default Cluster Prometheus Settings in values.yaml ``` config: costControl: cluster: metrics: # Number of replicas for the Prometheus Deployment replicas: 1 # Prometheus TSDB retention settings: https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects retention: 1y # CPU and Memory Requests and Limits for the Prometheus Deployment resources: requests: cpu: 500m memory: 3Gi limits: cpu: '1' memory: 3Gi # Storage Settings for Persistent Volumes storage: size: 60Gi # Valid values: # - null: use the default storage class # - "": disable automatic storage provisioning # - name of the preferred configured storage class storageClass: null ``` Overriding Resources Overridden `resources` are not merged with the defaults to avoid unexpected changes from Helm charts used by the platform to manage Prometheus. Therefore, all requests and limits that you wish to apply should be provided. ### Cluster cost control[​](#cluster-cost-control "Direct link to Cluster cost control") These settings control the OpenCost deployment on each connected cluster. While there are many possible ways to configure OpenCost, the Platform configures it with the minimum settings required for the dashboard. As such only availability, CPU, and memory settings may be configured. Default OpenCost Settings in values.yaml ``` config: costControl: cluster: opencost: # Number of replicas for the OpenCost Deployment replicas: 1 # CPU and Memory Requests and Limits for the OpenCost Deployment resources: requests: cpu: 500m memory: 3Gi limits: cpu: '1' memory: 3Gi ``` Overriding Resources Overridden `resources` are not merged with the defaults to avoid unexpected changes from Helm charts used by the platform to manage OpenCost. Therefore, all requests and limits that you wish to apply should be provided. ### Other cluster settings[​](#other-cluster-settings "Direct link to Other cluster settings") For individual cluster configuration, the [Cluster](https://vcluster.com/docs/platform/api/resources/clusters/.md) resource allows for further configuration specific to each cluster. --- # Custom branding | | | Enterprise | | | | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Advanced UI Customizations](https://vcluster.com/docs/platform/configure/platform-configs/custom-branding.md#advanced-custom-branding) | | | | | | [Custom Branding](https://vcluster.com/docs/platform/configure/platform-configs/custom-branding.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). The platform UI can be customized to match your corporate design, enhancing brand consistency and user experience. Custom branding can be configured using the platform config. Add the following YAML configuration to your platform config (the `config` section of the [`values.yaml`](https://vcluster.com/docs/platform/configure/introduction.md)) or through vCluster Platform UI **Admin > Config**: Basic custom branding configuration ``` uiSettings: # Terms of Service, Privacy Policy etc. # Can be an html string. legalTemplate: You agree to the Terms of Service # An **accessible** url. Can be in your cluster as long as you make sure the UI can find it logoURL: https://acme.com/static/our-logo.svg # Primary color # Can be `hex`, `rgb` or `rgba` string primaryColor: "#1890ff" # Sidebar color # Can be `hex`, `rgb` or `rgba` string sidebarColor: "#001529" # Accent color # Can be `hex`, `rgb` or `rgba` string accentColor: "#00BDFF" ``` ## Advanced custom branding[​](#advanced-custom-branding "Direct link to Advanced custom branding") For more extensive customization, you can add custom navigation items and apply additional CSS and JavaScript. ### Custom navigation items[​](#custom-navigation-items "Direct link to Custom navigation items") Extend the primary sidebar with your own navigation items: Adding custom navigation items ``` navBarButtons: - text: "My Link" icon: "https://acme.com/static/my-icon.svg" link: "https://acme.com/my-link" position: TopEnd # Options: TopStart, TopEnd, BottomStart, BottomEnd (default: BottomEnd) ``` ### Custom CSS and JavaScript[​](#custom-css-and-javascript "Direct link to Custom CSS and JavaScript") For more control over the UI appearance and behavior, you can include custom CSS and JavaScript files: Including custom CSS and JavaScript ``` customCss: - https://your-server.tld/overrides.css customJavaScript: - https://your-server.tld/script.js ``` info Ensure that all custom files (CSS, JavaScript, and images) are accessible to the platform UI. The CSS classes and HTML IDs in the platform UI DOM are stable across versions, allowing for consistent styling and scripting. --- # Overview Platform configs are the content in the [`values.yaml`](https://vcluster.com/docs/platform/configure/introduction.md) under the `config` section. It controls the behavior of the vCluster Platform after it is installed on the host cluster. You can also configure the content in vCluster Platform UI via **Admin > Config**. ## UI vs `config` section[​](#ui-vs-config-section "Direct link to ui-vs-config-section") After the deployment, the behavior of the platform can be configured by the `config` section of the `values.yaml` or **Admin > Config** in vCluster Platform UI. When using `values.yaml`, you should add the configuration under the `config` section of the `vcluster-platform` chart's `values.yaml`. For example: ``` config: vault: enabled: true ``` However, when using vCluster Platform UI, you can directly only add the configuration content. ``` vault: enabled: true ``` ## Manage sensitive information[​](#managing-sensitive-information "Direct link to Manage sensitive information") Many configuration options require sensitive information such as API keys or tokens. Rather than storing these directly in your `values.yaml` file, use environment variable placeholders. ### Use secret references[​](#using-secret-references "Direct link to Use secret references") Store sensitive values in a Kubernetes secret and reference it in your configuration using `envValueFrom`: Store sensitive values using Kubernetes secrets ``` # Secret references envValueFrom: CLIENT_ID: secretKeyRef: name: github-auth-secret key: client_id CLIENT_SECRET: secretKeyRef: name: github-auth-secret key: client_secret # Platform configuration config: auth: github: clientId: $CLIENT_ID clientSecret: $CLIENT_SECRET redirectURI: https://my-platform.example.com/auth/github/callback ``` ## Configure custom HTTP headers[​](#custom-http-headers "Direct link to Configure custom HTTP headers") You can configure the platform to add custom HTTP headers to all API responses. This is useful for security-related headers or when integrating with specific environments. Adding custom HTTP headers ``` config: auth: customHttpHeaders: X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block Content-Security-Policy: default-src 'self' ``` ## Reference[​](#reference "Direct link to Reference") The complete list of platform configuration fields is as follows: ## `auth` object [​](#auth "Direct link to auth")Authentication holds the information for authentication ### `oidc` object [​](#auth-oidc "Direct link to auth-oidc")OIDC holds oidc authentication configuration #### `issuerUrl` string [​](#auth-oidc-issuerUrl "Direct link to auth-oidc-issuerUrl")IssuerURL is the URL the provider signs ID Tokens as. This will be the "iss" field of all tokens produced by the provider and is used for configuration discovery.The URL is usually the provider's URL without a path, for example "" or "".The provider must implement configuration discovery. See: #### `clientId` string [​](#auth-oidc-clientId "Direct link to auth-oidc-clientId")ClientID the JWT must be issued for, the "sub" field. This plugin only trusts a single client to ensure the plugin can be used with public providers.The plugin supports the "authorized party" OpenID Connect claim, which allows specialized providers to issue tokens to a client for a different client. See: #### `clientSecret` string [​](#auth-oidc-clientSecret "Direct link to auth-oidc-clientSecret")ClientSecret to issue tokens from the OIDC provider #### `redirectURI` string [​](#auth-oidc-redirectURI "Direct link to auth-oidc-redirectURI")loft redirect uri. E.g. #### `postLogoutRedirectURI` string [​](#auth-oidc-postLogoutRedirectURI "Direct link to auth-oidc-postLogoutRedirectURI")Loft URI to be redirected to after successful logout by OIDC Provider #### `caFile` string [​](#auth-oidc-caFile "Direct link to auth-oidc-caFile")Path to a PEM encoded root certificate of the provider. Optional #### `insecureCa` boolean [​](#auth-oidc-insecureCa "Direct link to auth-oidc-insecureCa")Specify whether to communicate without validating SSL certificates #### `preferredUsername` string [​](#auth-oidc-preferredUsername "Direct link to auth-oidc-preferredUsername")Configurable key which contains the preferred username claims #### `loftUsernameClaim` string [​](#auth-oidc-loftUsernameClaim "Direct link to auth-oidc-loftUsernameClaim")LoftUsernameClaim is the JWT field to use as the user's username. #### `usernameClaim` string [​](#auth-oidc-usernameClaim "Direct link to auth-oidc-usernameClaim")UsernameClaim is the JWT field to use as the user's id. #### `emailClaim` string [​](#auth-oidc-emailClaim "Direct link to auth-oidc-emailClaim")EmailClaim is the JWT field to use as the user's email. #### `allowedExtraClaims` string\[] [​](#auth-oidc-allowedExtraClaims "Direct link to auth-oidc-allowedExtraClaims")AllowedExtraClaims are claims of interest that are not part of User by default but may be provided by the OIDC provider. #### `usernamePrefix` string [​](#auth-oidc-usernamePrefix "Direct link to auth-oidc-usernamePrefix")UsernamePrefix, if specified, causes claims mapping to username to be prefix with the provided value. A value "oidc:" would result in usernames like "oidc:john". #### `groupsClaim` string [​](#auth-oidc-groupsClaim "Direct link to auth-oidc-groupsClaim")GroupsClaim, if specified, causes the OIDCAuthenticator to try to populate the user's groups with an ID Token field. If the GroupsClaim field is present in an ID Token the value must be a string or list of strings. #### `groups` string\[] [​](#auth-oidc-groups "Direct link to auth-oidc-groups")If required groups is non empty, access is denied if the user is not part of at least one of the specified groups. #### `scopes` string\[] [​](#auth-oidc-scopes "Direct link to auth-oidc-scopes")Scopes that should be sent to the server. If empty, defaults to "email" and "profile". #### `getUserInfo` boolean [​](#auth-oidc-getUserInfo "Direct link to auth-oidc-getUserInfo")GetUserInfo, if specified, tells the OIDCAuthenticator to try to populate the user's information from the UserInfo. #### `groupsPrefix` string [​](#auth-oidc-groupsPrefix "Direct link to auth-oidc-groupsPrefix")GroupsPrefix, if specified, causes claims mapping to group names to be prefixed with the value. A value "oidc:" would result in groups like "oidc:engineering" and "oidc:marketing". #### `type` string [​](#auth-oidc-type "Direct link to auth-oidc-type")Type of the OIDC to show in the UI. Only for displaying purposes #### `resource` string [​](#auth-oidc-resource "Direct link to auth-oidc-resource")Resource, if specified, is the value that is set for the "resource" URL parameter when making a request to the /token endpoint of the OIDC provider. ### `github` object [​](#auth-github "Direct link to auth-github")Github holds github authentication configuration #### `clientId` string [​](#auth-github-clientId "Direct link to auth-github-clientId")ClientID holds the github client id #### `clientSecret` required string [​](#auth-github-clientSecret "Direct link to auth-github-clientSecret")ClientID holds the github client secret #### `redirectURI` required string [​](#auth-github-redirectURI "Direct link to auth-github-redirectURI")RedirectURI holds the redirect URI. Should be #### `orgs` object\[] [​](#auth-github-orgs "Direct link to auth-github-orgs")Loft queries the following organizations for group information. Group claims are formatted as "(org):(team)". For example if a user is part of the "engineering" team of the "coreos" org, the group claim would include "coreos:engineering".If orgs are specified in the config then user MUST be a member of at least one of the specified orgs to authenticate with loft. ##### `name` string [​](#auth-github-orgs-name "Direct link to auth-github-orgs-name")Organization name in github (not slug, full name). Only users in this github organization can authenticate. ##### `teams` string\[] [​](#auth-github-orgs-teams "Direct link to auth-github-orgs-teams")Names of teams in a github organization. A user will be able to authenticate if they are members of at least one of these teams. Users in the organization can authenticate if this field is omitted from the config file. #### `hostName` string [​](#auth-github-hostName "Direct link to auth-github-hostName")Required ONLY for GitHub Enterprise. This is the Hostname of the GitHub Enterprise account listed on the management console. Ensure this domain is routable on your network. #### `rootCA` string [​](#auth-github-rootCA "Direct link to auth-github-rootCA")ONLY for GitHub Enterprise. Optional field. Used to support self-signed or untrusted CA root certificates. ### `gitlab` object [​](#auth-gitlab "Direct link to auth-gitlab")Gitlab holds gitlab authentication configuration #### `clientId` required string [​](#auth-gitlab-clientId "Direct link to auth-gitlab-clientId")Gitlab client id #### `clientSecret` required string [​](#auth-gitlab-clientSecret "Direct link to auth-gitlab-clientSecret")Gitlab client secret #### `redirectURI` required string [​](#auth-gitlab-redirectURI "Direct link to auth-gitlab-redirectURI")Redirect URI #### `baseURL` string [​](#auth-gitlab-baseURL "Direct link to auth-gitlab-baseURL")BaseURL is optional, default = #### `groups` string\[] [​](#auth-gitlab-groups "Direct link to auth-gitlab-groups")Optional groups whitelist, communicated through the "groups" scope. If `groups` is omitted, all of the user's GitLab groups are returned. If `groups` is provided, this acts as a whitelist - only the user's GitLab groups that are in the configured `groups` below will go into the groups claim. Conversely, if the user is not in any of the configured `groups`, the user will not be authenticated. ### `google` object [​](#auth-google "Direct link to auth-google")Google holds google authentication configuration #### `clientId` required string [​](#auth-google-clientId "Direct link to auth-google-clientId")Google client id #### `clientSecret` required string [​](#auth-google-clientSecret "Direct link to auth-google-clientSecret")Google client secret #### `redirectURI` required string [​](#auth-google-redirectURI "Direct link to auth-google-redirectURI")loft redirect uri. E.g. #### `scopes` string\[] [​](#auth-google-scopes "Direct link to auth-google-scopes")defaults to "profile" and "email" #### `hostedDomains` string\[] [​](#auth-google-hostedDomains "Direct link to auth-google-hostedDomains")Optional list of whitelisted domains If this field is nonempty, only users from a listed domain will be allowed to log in #### `groups` string\[] [​](#auth-google-groups "Direct link to auth-google-groups")Optional list of whitelisted groups If this field is nonempty, only users from a listed group will be allowed to log in #### `serviceAccountFilePath` string [​](#auth-google-serviceAccountFilePath "Direct link to auth-google-serviceAccountFilePath")Optional path to service account json If nonempty, and groups claim is made, will use authentication from file to check groups with the admin directory api #### `adminEmail` string [​](#auth-google-adminEmail "Direct link to auth-google-adminEmail")Required if ServiceAccountFilePath The email of a GSuite super user which the service account will impersonate when listing groups ### `microsoft` object [​](#auth-microsoft "Direct link to auth-microsoft")Microsoft holds microsoft authentication configuration #### `clientId` required string [​](#auth-microsoft-clientId "Direct link to auth-microsoft-clientId")Microsoft client id #### `clientSecret` required string [​](#auth-microsoft-clientSecret "Direct link to auth-microsoft-clientSecret")Microsoft client secret #### `redirectURI` required string [​](#auth-microsoft-redirectURI "Direct link to auth-microsoft-redirectURI")loft redirect uri. Usually #### `tenant` string [​](#auth-microsoft-tenant "Direct link to auth-microsoft-tenant")tenant configuration parameter controls what kinds of accounts may be authenticated in loft. By default, all types of Microsoft accounts (consumers and organizations) can authenticate in loft via Microsoft. To change this, set the tenant parameter to one of the following:common - both personal and business/school accounts can authenticate in loft via Microsoft (default) consumers - only personal accounts can authenticate in loft organizations - only business/school accounts can authenticate in loft tenant uuid or tenant name - only accounts belonging to specific tenant identified by either tenant uuid or tenant name can authenticate in loft #### `groups` string\[] [​](#auth-microsoft-groups "Direct link to auth-microsoft-groups")It is possible to require a user to be a member of a particular group in order to be successfully authenticated in loft. #### `onlySecurityGroups` boolean [​](#auth-microsoft-onlySecurityGroups "Direct link to auth-microsoft-onlySecurityGroups")configuration option restricts the list to include only security groups. By default all groups (security, Office 365, mailing lists) are included. #### `useGroupsAsWhitelist` boolean [​](#auth-microsoft-useGroupsAsWhitelist "Direct link to auth-microsoft-useGroupsAsWhitelist")Restrict the groups claims to include only the user’s groups that are in the configured groups ### `saml` object [​](#auth-saml "Direct link to auth-saml")SAML holds saml authentication configuration #### `redirectURI` string [​](#auth-saml-redirectURI "Direct link to auth-saml-redirectURI")If the response assertion status value contains a Destination element, it must match this value exactly. Usually looks like #### `ssoURL` string [​](#auth-saml-ssoURL "Direct link to auth-saml-ssoURL")SSO URL used for POST value. #### `caData` string [​](#auth-saml-caData "Direct link to auth-saml-caData")CAData is a base64 encoded string that holds the ca certificate for validating the signature of the SAML response. Either CAData, CA or InsecureSkipSignatureValidation needs to be defined. #### `usernameAttr` string [​](#auth-saml-usernameAttr "Direct link to auth-saml-usernameAttr")Name of attribute in the returned assertions to map to username #### `emailAttr` string [​](#auth-saml-emailAttr "Direct link to auth-saml-emailAttr")Name of attribute in the returned assertions to map to email #### `groupsAttr` string [​](#auth-saml-groupsAttr "Direct link to auth-saml-groupsAttr")Name of attribute in the returned assertions to map to groups #### `ca` string [​](#auth-saml-ca "Direct link to auth-saml-ca")CA to use when validating the signature of the SAML response. #### `insecureSkipSignatureValidation` boolean [​](#auth-saml-insecureSkipSignatureValidation "Direct link to auth-saml-insecureSkipSignatureValidation")Ignore the ca cert #### `entityIssuer` string [​](#auth-saml-entityIssuer "Direct link to auth-saml-entityIssuer")When provided Loft will include this as the Issuer value during AuthnRequest. It will also override the redirectURI as the required audience when evaluating AudienceRestriction elements in the response. #### `ssoIssuer` string [​](#auth-saml-ssoIssuer "Direct link to auth-saml-ssoIssuer")Issuer value expected in the SAML response. Optional. #### `groupsDelim` string [​](#auth-saml-groupsDelim "Direct link to auth-saml-groupsDelim")If GroupsDelim is supplied the connector assumes groups are returned as a single string instead of multiple attribute values. This delimiter will be used split the groups string. #### `allowedGroups` string\[] [​](#auth-saml-allowedGroups "Direct link to auth-saml-allowedGroups")List of groups to filter access based on membership #### `filterGroups` boolean [​](#auth-saml-filterGroups "Direct link to auth-saml-filterGroups")If used with allowed groups, only forwards the allowed groups and not all groups specified. #### `nameIDPolicyFormat` string [​](#auth-saml-nameIDPolicyFormat "Direct link to auth-saml-nameIDPolicyFormat")Requested format of the NameID. The NameID value is is mapped to the ID Token 'sub' claim.This can be an abbreviated form of the full URI with just the last component. For example, if this value is set to "emailAddress" the format will resolve to:urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddressIf no value is specified, this value defaults to:urn:oasis:names:tc:SAML:2.0:nameid-format:persistent ### `password` object [​](#auth-password "Direct link to auth-password")Password holds password authentication relevant information #### `disabled` boolean [​](#auth-password-disabled "Direct link to auth-password-disabled")If true login via password is disabled ### `connectors` object\[] [​](#auth-connectors "Direct link to auth-connectors")Connectors are optional additional connectors for Loft. #### `id` string [​](#auth-connectors-id "Direct link to auth-connectors-id")ID is the id that should show up in the url #### `displayName` string [​](#auth-connectors-displayName "Direct link to auth-connectors-displayName")DisplayName is the name that should show up in the ui #### `oidc` object [​](#auth-connectors-oidc "Direct link to auth-connectors-oidc")OIDC holds oidc authentication configuration ##### `issuerUrl` string [​](#auth-connectors-oidc-issuerUrl "Direct link to auth-connectors-oidc-issuerUrl")IssuerURL is the URL the provider signs ID Tokens as. This will be the "iss" field of all tokens produced by the provider and is used for configuration discovery.The URL is usually the provider's URL without a path, for example "" or "".The provider must implement configuration discovery. See: ##### `clientId` string [​](#auth-connectors-oidc-clientId "Direct link to auth-connectors-oidc-clientId")ClientID the JWT must be issued for, the "sub" field. This plugin only trusts a single client to ensure the plugin can be used with public providers.The plugin supports the "authorized party" OpenID Connect claim, which allows specialized providers to issue tokens to a client for a different client. See: ##### `clientSecret` string [​](#auth-connectors-oidc-clientSecret "Direct link to auth-connectors-oidc-clientSecret")ClientSecret to issue tokens from the OIDC provider ##### `redirectURI` string [​](#auth-connectors-oidc-redirectURI "Direct link to auth-connectors-oidc-redirectURI")loft redirect uri. E.g. ##### `postLogoutRedirectURI` string [​](#auth-connectors-oidc-postLogoutRedirectURI "Direct link to auth-connectors-oidc-postLogoutRedirectURI")Loft URI to be redirected to after successful logout by OIDC Provider ##### `caFile` string [​](#auth-connectors-oidc-caFile "Direct link to auth-connectors-oidc-caFile")Path to a PEM encoded root certificate of the provider. Optional ##### `insecureCa` boolean [​](#auth-connectors-oidc-insecureCa "Direct link to auth-connectors-oidc-insecureCa")Specify whether to communicate without validating SSL certificates ##### `preferredUsername` string [​](#auth-connectors-oidc-preferredUsername "Direct link to auth-connectors-oidc-preferredUsername")Configurable key which contains the preferred username claims ##### `loftUsernameClaim` string [​](#auth-connectors-oidc-loftUsernameClaim "Direct link to auth-connectors-oidc-loftUsernameClaim")LoftUsernameClaim is the JWT field to use as the user's username. ##### `usernameClaim` string [​](#auth-connectors-oidc-usernameClaim "Direct link to auth-connectors-oidc-usernameClaim")UsernameClaim is the JWT field to use as the user's id. ##### `emailClaim` string [​](#auth-connectors-oidc-emailClaim "Direct link to auth-connectors-oidc-emailClaim")EmailClaim is the JWT field to use as the user's email. ##### `allowedExtraClaims` string\[] [​](#auth-connectors-oidc-allowedExtraClaims "Direct link to auth-connectors-oidc-allowedExtraClaims")AllowedExtraClaims are claims of interest that are not part of User by default but may be provided by the OIDC provider. ##### `usernamePrefix` string [​](#auth-connectors-oidc-usernamePrefix "Direct link to auth-connectors-oidc-usernamePrefix")UsernamePrefix, if specified, causes claims mapping to username to be prefix with the provided value. A value "oidc:" would result in usernames like "oidc:john". ##### `groupsClaim` string [​](#auth-connectors-oidc-groupsClaim "Direct link to auth-connectors-oidc-groupsClaim")GroupsClaim, if specified, causes the OIDCAuthenticator to try to populate the user's groups with an ID Token field. If the GroupsClaim field is present in an ID Token the value must be a string or list of strings. ##### `groups` string\[] [​](#auth-connectors-oidc-groups "Direct link to auth-connectors-oidc-groups")If required groups is non empty, access is denied if the user is not part of at least one of the specified groups. ##### `scopes` string\[] [​](#auth-connectors-oidc-scopes "Direct link to auth-connectors-oidc-scopes")Scopes that should be sent to the server. If empty, defaults to "email" and "profile". ##### `getUserInfo` boolean [​](#auth-connectors-oidc-getUserInfo "Direct link to auth-connectors-oidc-getUserInfo")GetUserInfo, if specified, tells the OIDCAuthenticator to try to populate the user's information from the UserInfo. ##### `groupsPrefix` string [​](#auth-connectors-oidc-groupsPrefix "Direct link to auth-connectors-oidc-groupsPrefix")GroupsPrefix, if specified, causes claims mapping to group names to be prefixed with the value. A value "oidc:" would result in groups like "oidc:engineering" and "oidc:marketing". ##### `type` string [​](#auth-connectors-oidc-type "Direct link to auth-connectors-oidc-type")Type of the OIDC to show in the UI. Only for displaying purposes ##### `resource` string [​](#auth-connectors-oidc-resource "Direct link to auth-connectors-oidc-resource")Resource, if specified, is the value that is set for the "resource" URL parameter when making a request to the /token endpoint of the OIDC provider. #### `github` object [​](#auth-connectors-github "Direct link to auth-connectors-github")Github holds github authentication configuration ##### `clientId` string [​](#auth-connectors-github-clientId "Direct link to auth-connectors-github-clientId")ClientID holds the github client id ##### `clientSecret` required string [​](#auth-connectors-github-clientSecret "Direct link to auth-connectors-github-clientSecret")ClientID holds the github client secret ##### `redirectURI` required string [​](#auth-connectors-github-redirectURI "Direct link to auth-connectors-github-redirectURI")RedirectURI holds the redirect URI. Should be ##### `orgs` object\[] [​](#auth-connectors-github-orgs "Direct link to auth-connectors-github-orgs")Loft queries the following organizations for group information. Group claims are formatted as "(org):(team)". For example if a user is part of the "engineering" team of the "coreos" org, the group claim would include "coreos:engineering".If orgs are specified in the config then user MUST be a member of at least one of the specified orgs to authenticate with loft. ##### `name` string [​](#auth-connectors-github-orgs-name "Direct link to auth-connectors-github-orgs-name")Organization name in github (not slug, full name). Only users in this github organization can authenticate. ##### `teams` string\[] [​](#auth-connectors-github-orgs-teams "Direct link to auth-connectors-github-orgs-teams")Names of teams in a github organization. A user will be able to authenticate if they are members of at least one of these teams. Users in the organization can authenticate if this field is omitted from the config file. ##### `hostName` string [​](#auth-connectors-github-hostName "Direct link to auth-connectors-github-hostName")Required ONLY for GitHub Enterprise. This is the Hostname of the GitHub Enterprise account listed on the management console. Ensure this domain is routable on your network. ##### `rootCA` string [​](#auth-connectors-github-rootCA "Direct link to auth-connectors-github-rootCA")ONLY for GitHub Enterprise. Optional field. Used to support self-signed or untrusted CA root certificates. #### `gitlab` object [​](#auth-connectors-gitlab "Direct link to auth-connectors-gitlab")Gitlab holds gitlab authentication configuration ##### `clientId` required string [​](#auth-connectors-gitlab-clientId "Direct link to auth-connectors-gitlab-clientId")Gitlab client id ##### `clientSecret` required string [​](#auth-connectors-gitlab-clientSecret "Direct link to auth-connectors-gitlab-clientSecret")Gitlab client secret ##### `redirectURI` required string [​](#auth-connectors-gitlab-redirectURI "Direct link to auth-connectors-gitlab-redirectURI")Redirect URI ##### `baseURL` string [​](#auth-connectors-gitlab-baseURL "Direct link to auth-connectors-gitlab-baseURL")BaseURL is optional, default = ##### `groups` string\[] [​](#auth-connectors-gitlab-groups "Direct link to auth-connectors-gitlab-groups")Optional groups whitelist, communicated through the "groups" scope. If `groups` is omitted, all of the user's GitLab groups are returned. If `groups` is provided, this acts as a whitelist - only the user's GitLab groups that are in the configured `groups` below will go into the groups claim. Conversely, if the user is not in any of the configured `groups`, the user will not be authenticated. #### `google` object [​](#auth-connectors-google "Direct link to auth-connectors-google")Google holds google authentication configuration ##### `clientId` required string [​](#auth-connectors-google-clientId "Direct link to auth-connectors-google-clientId")Google client id ##### `clientSecret` required string [​](#auth-connectors-google-clientSecret "Direct link to auth-connectors-google-clientSecret")Google client secret ##### `redirectURI` required string [​](#auth-connectors-google-redirectURI "Direct link to auth-connectors-google-redirectURI")loft redirect uri. E.g. ##### `scopes` string\[] [​](#auth-connectors-google-scopes "Direct link to auth-connectors-google-scopes")defaults to "profile" and "email" ##### `hostedDomains` string\[] [​](#auth-connectors-google-hostedDomains "Direct link to auth-connectors-google-hostedDomains")Optional list of whitelisted domains If this field is nonempty, only users from a listed domain will be allowed to log in ##### `groups` string\[] [​](#auth-connectors-google-groups "Direct link to auth-connectors-google-groups")Optional list of whitelisted groups If this field is nonempty, only users from a listed group will be allowed to log in ##### `serviceAccountFilePath` string [​](#auth-connectors-google-serviceAccountFilePath "Direct link to auth-connectors-google-serviceAccountFilePath")Optional path to service account json If nonempty, and groups claim is made, will use authentication from file to check groups with the admin directory api ##### `adminEmail` string [​](#auth-connectors-google-adminEmail "Direct link to auth-connectors-google-adminEmail")Required if ServiceAccountFilePath The email of a GSuite super user which the service account will impersonate when listing groups #### `microsoft` object [​](#auth-connectors-microsoft "Direct link to auth-connectors-microsoft")Microsoft holds microsoft authentication configuration ##### `clientId` required string [​](#auth-connectors-microsoft-clientId "Direct link to auth-connectors-microsoft-clientId")Microsoft client id ##### `clientSecret` required string [​](#auth-connectors-microsoft-clientSecret "Direct link to auth-connectors-microsoft-clientSecret")Microsoft client secret ##### `redirectURI` required string [​](#auth-connectors-microsoft-redirectURI "Direct link to auth-connectors-microsoft-redirectURI")loft redirect uri. Usually ##### `tenant` string [​](#auth-connectors-microsoft-tenant "Direct link to auth-connectors-microsoft-tenant")tenant configuration parameter controls what kinds of accounts may be authenticated in loft. By default, all types of Microsoft accounts (consumers and organizations) can authenticate in loft via Microsoft. To change this, set the tenant parameter to one of the following:common - both personal and business/school accounts can authenticate in loft via Microsoft (default) consumers - only personal accounts can authenticate in loft organizations - only business/school accounts can authenticate in loft tenant uuid or tenant name - only accounts belonging to specific tenant identified by either tenant uuid or tenant name can authenticate in loft ##### `groups` string\[] [​](#auth-connectors-microsoft-groups "Direct link to auth-connectors-microsoft-groups")It is possible to require a user to be a member of a particular group in order to be successfully authenticated in loft. ##### `onlySecurityGroups` boolean [​](#auth-connectors-microsoft-onlySecurityGroups "Direct link to auth-connectors-microsoft-onlySecurityGroups")configuration option restricts the list to include only security groups. By default all groups (security, Office 365, mailing lists) are included. ##### `useGroupsAsWhitelist` boolean [​](#auth-connectors-microsoft-useGroupsAsWhitelist "Direct link to auth-connectors-microsoft-useGroupsAsWhitelist")Restrict the groups claims to include only the user’s groups that are in the configured groups #### `saml` object [​](#auth-connectors-saml "Direct link to auth-connectors-saml")SAML holds saml authentication configuration ##### `redirectURI` string [​](#auth-connectors-saml-redirectURI "Direct link to auth-connectors-saml-redirectURI")If the response assertion status value contains a Destination element, it must match this value exactly. Usually looks like ##### `ssoURL` string [​](#auth-connectors-saml-ssoURL "Direct link to auth-connectors-saml-ssoURL")SSO URL used for POST value. ##### `caData` string [​](#auth-connectors-saml-caData "Direct link to auth-connectors-saml-caData")CAData is a base64 encoded string that holds the ca certificate for validating the signature of the SAML response. Either CAData, CA or InsecureSkipSignatureValidation needs to be defined. ##### `usernameAttr` string [​](#auth-connectors-saml-usernameAttr "Direct link to auth-connectors-saml-usernameAttr")Name of attribute in the returned assertions to map to username ##### `emailAttr` string [​](#auth-connectors-saml-emailAttr "Direct link to auth-connectors-saml-emailAttr")Name of attribute in the returned assertions to map to email ##### `groupsAttr` string [​](#auth-connectors-saml-groupsAttr "Direct link to auth-connectors-saml-groupsAttr")Name of attribute in the returned assertions to map to groups ##### `ca` string [​](#auth-connectors-saml-ca "Direct link to auth-connectors-saml-ca")CA to use when validating the signature of the SAML response. ##### `insecureSkipSignatureValidation` boolean [​](#auth-connectors-saml-insecureSkipSignatureValidation "Direct link to auth-connectors-saml-insecureSkipSignatureValidation")Ignore the ca cert ##### `entityIssuer` string [​](#auth-connectors-saml-entityIssuer "Direct link to auth-connectors-saml-entityIssuer")When provided Loft will include this as the Issuer value during AuthnRequest. It will also override the redirectURI as the required audience when evaluating AudienceRestriction elements in the response. ##### `ssoIssuer` string [​](#auth-connectors-saml-ssoIssuer "Direct link to auth-connectors-saml-ssoIssuer")Issuer value expected in the SAML response. Optional. ##### `groupsDelim` string [​](#auth-connectors-saml-groupsDelim "Direct link to auth-connectors-saml-groupsDelim")If GroupsDelim is supplied the connector assumes groups are returned as a single string instead of multiple attribute values. This delimiter will be used split the groups string. ##### `allowedGroups` string\[] [​](#auth-connectors-saml-allowedGroups "Direct link to auth-connectors-saml-allowedGroups")List of groups to filter access based on membership ##### `filterGroups` boolean [​](#auth-connectors-saml-filterGroups "Direct link to auth-connectors-saml-filterGroups")If used with allowed groups, only forwards the allowed groups and not all groups specified. ##### `nameIDPolicyFormat` string [​](#auth-connectors-saml-nameIDPolicyFormat "Direct link to auth-connectors-saml-nameIDPolicyFormat")Requested format of the NameID. The NameID value is is mapped to the ID Token 'sub' claim.This can be an abbreviated form of the full URI with just the last component. For example, if this value is set to "emailAddress" the format will resolve to:urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddressIf no value is specified, this value defaults to:urn:oasis:names:tc:SAML:2.0:nameid-format:persistent ### `disableTeamCreation` boolean [​](#auth-disableTeamCreation "Direct link to auth-disableTeamCreation")Prevents from team creation for the new groups associated with the user at the time of logging in through sso, Default behaviour is false, this means that teams will be created for new groups. ### `disableUserCreation` boolean [​](#auth-disableUserCreation "Direct link to auth-disableUserCreation")DisableUserCreation prevents the SSO connectors from creating a new user on a users initial signin through sso. Default behaviour is false, this means that a new user object will be created once a user without a Kubernetes user object logs in. ### `accessKeyMaxTTLSeconds` integer [​](#auth-accessKeyMaxTTLSeconds "Direct link to auth-accessKeyMaxTTLSeconds")AccessKeyMaxTTLSeconds is the global maximum lifespan of an accesskey in seconds. Leaving it 0 or unspecified will disable it. Specifying 2592000 will mean all keys have a Time-To-Live of 30 days. ### `loginAccessKeyTTLSeconds` integer [​](#auth-loginAccessKeyTTLSeconds "Direct link to auth-loginAccessKeyTTLSeconds")LoginAccessKeyTTLSeconds is the time in seconds an access key is kept until it is deleted. Leaving it unspecified will default to 20 days. Setting it to zero will disable the ttl. Specifying 2592000 will mean all keys have a default Time-To-Live of 30 days. ### `customHttpHeaders` object [​](#auth-customHttpHeaders "Direct link to auth-customHttpHeaders")CustomHttpHeaders are additional headers that should be set for the authentication endpoints ### `groupsFilters` string\[] [​](#auth-groupsFilters "Direct link to auth-groupsFilters")GroupsFilters is a regex expression to only save matching sso groups into the user resource ## `oidc` object [​](#oidc "Direct link to oidc")DEPRECATED: Configure the OIDC clients using either the OIDC Client UI or a secret. By default, vCluster Platform as an OIDC Provider is enabled but does not function without OIDC clients. ### `enabled` boolean [​](#oidc-enabled "Direct link to oidc-enabled")If true indicates that loft will act as an OIDC server ### `wildcardRedirect` boolean [​](#oidc-wildcardRedirect "Direct link to oidc-wildcardRedirect")If true indicates that loft will allow wildcard '\*' in client redirectURIs ### `clients` object\[] [​](#oidc-clients "Direct link to oidc-clients")The clients that are allowed to request loft tokens #### `name` string [​](#oidc-clients-name "Direct link to oidc-clients-name")The client name #### `clientId` string [​](#oidc-clients-clientId "Direct link to oidc-clients-clientId")The client id of the client #### `clientSecret` string [​](#oidc-clients-clientSecret "Direct link to oidc-clients-clientSecret")The client secret of the client #### `redirectURIs` required string\[] [​](#oidc-clients-redirectURIs "Direct link to oidc-clients-redirectURIs")A registered set of redirect URIs. When redirecting from dex to the client, the URI requested to redirect to MUST match one of these values, unless the client is "public". ## `audit` object [​](#audit "Direct link to audit")Audit holds audit configuration ### `enabled` boolean [​](#audit-enabled "Direct link to audit-enabled")If audit is enabled and incoming api requests will be logged based on the supplied policy. ### `disableAgentSyncBack` boolean [​](#audit-disableAgentSyncBack "Direct link to audit-disableAgentSyncBack")If true, the agent will not send back any audit logs to Loft itself. ### `level` integer [​](#audit-level "Direct link to audit-level")Level is an optional log level for audit logs. Cannot be used together with policy ### `policy` object [​](#audit-policy "Direct link to audit-policy")The audit policy to use and log requests. By default loft will not log anything #### `rules` object\[] [​](#audit-policy-rules "Direct link to audit-policy-rules")Rules specify the audit Level a request should be recorded at. A request may match multiple rules, in which case the FIRST matching rule is used. The default audit level is None, but can be overridden by a catch-all rule at the end of the list. PolicyRules are strictly ordered. ##### `level` required string [​](#audit-policy-rules-level "Direct link to audit-policy-rules-level")The Level that requests matching this rule are recorded at. ##### `users` string\[] [​](#audit-policy-rules-users "Direct link to audit-policy-rules-users")The users (by authenticated user name) this rule applies to. An empty list implies every user. ##### `userGroups` string\[] [​](#audit-policy-rules-userGroups "Direct link to audit-policy-rules-userGroups")The user groups this rule applies to. A user is considered matching if it is a member of any of the UserGroups. An empty list implies every user group. ##### `verbs` string\[] [​](#audit-policy-rules-verbs "Direct link to audit-policy-rules-verbs")The verbs that match this rule. An empty list implies every verb. ##### `resources` object\[] [​](#audit-policy-rules-resources "Direct link to audit-policy-rules-resources")Resources that this rule matches. An empty list implies all kinds in all API groups. ##### `group` string [​](#audit-policy-rules-resources-group "Direct link to audit-policy-rules-resources-group")Group is the name of the API group that contains the resources. The empty string represents the core API group. ##### `resources` string\[] [​](#audit-policy-rules-resources-resources "Direct link to audit-policy-rules-resources-resources")Resources is a list of resources this rule applies to.For example: 'pods' matches pods. 'pods/log' matches the log subresource of pods. '*' matches all resources and their subresources. 'pods/*' matches all subresources of pods. '\*/scale' matches all scale subresources.If wildcard is present, the validation rule will ensure resources do not overlap with each other.An empty list implies all resources and subresources in this API groups apply. ##### `resourceNames` string\[] [​](#audit-policy-rules-resources-resourceNames "Direct link to audit-policy-rules-resources-resourceNames")ResourceNames is a list of resource instance names that the policy matches. Using this field requires Resources to be specified. An empty list implies that every instance of the resource is matched. ##### `namespaces` string\[] [​](#audit-policy-rules-namespaces "Direct link to audit-policy-rules-namespaces")Namespaces that this rule matches. The empty string "" matches non-namespaced resources. An empty list implies every namespace. ##### `nonResourceURLs` string\[] [​](#audit-policy-rules-nonResourceURLs "Direct link to audit-policy-rules-nonResourceURLs")NonResourceURLs is a set of URL paths that should be audited. *s are allowed, but only as the full, final step in the path. Examples: "/metrics" - Log requests for apiserver metrics "/healthz*" - Log all health checks ##### `omitStages` string\[] [​](#audit-policy-rules-omitStages "Direct link to audit-policy-rules-omitStages")OmitStages is a list of stages for which no events are created. Note that this can also be specified policy wide in which case the union of both are omitted. An empty list means no restrictions will apply. ##### `requestTargets` string\[] [​](#audit-policy-rules-requestTargets "Direct link to audit-policy-rules-requestTargets")RequestTargets is a list of request targets for which events are created. An empty list implies every request. ##### `clusters` string\[] [​](#audit-policy-rules-clusters "Direct link to audit-policy-rules-clusters")Clusters that this rule matches. Only applies to cluster requests. If this is set, no events for non cluster requests will be created. An empty list means no restrictions will apply. #### `omitStages` string\[] [​](#audit-policy-omitStages "Direct link to audit-policy-omitStages")OmitStages is a list of stages for which no events are created. Note that this can also be specified per rule in which case the union of both are omitted. ### `dataStoreEndpoint` string [​](#audit-dataStoreEndpoint "Direct link to audit-dataStoreEndpoint")DataStoreEndpoint is an endpoint to store events in. ### `dataStoreIdentityProvider` string [​](#audit-dataStoreIdentityProvider "Direct link to audit-dataStoreIdentityProvider")DataStoreIdentityProvider is the identity provider to use when generating temporary authentication tokens for the audit datastore, instead of a static password embedded in DataStoreEndpoint. Examples:- aws: RDS IAM Authentication ### `dataStoreCAFile` string [​](#audit-dataStoreCAFile "Direct link to audit-dataStoreCAFile")DataStoreCAFile is the path to a PEM-encoded certificate authority bundle used to verify the audit datastore's server certificate, read directly by the platform pod at startup. Mount it in the platform pod via the chart's top-level volumes/volumeMounts (e.g. from a Secret or ConfigMap), the same way as config.database's caFile. When set, the connection is fully verified (postgres: sslmode=verify-full; mysql: a registered TLS config with the given CA as RootCAs) instead of the default encrypted-but-unverified connection. Optional; has no effect on the sqlite backend.An explicit ?sslmode=/?tls= already present on DataStoreEndpoint always overrides this and is not upgraded, even when DataStoreCAFile is set - e.g. an endpoint carried over from before CA support existed with ?tls=skip-verify stays unverified. The platform logs this case at startup so it isn't silent. ### `dataStoreTTL` integer [​](#audit-dataStoreTTL "Direct link to audit-dataStoreTTL")DataStoreMaxAge is the maximum number of hours to retain old log events in the datastore ### `path` string [​](#audit-path "Direct link to audit-path")The path where to save the audit log files. This is required if audit is enabled. Backup log files will be retained in the same directory. ### `maxAge` integer [​](#audit-maxAge "Direct link to audit-maxAge")MaxAge is the maximum number of days to retain old log files based on the timestamp encoded in their filename. Note that a day is defined as 24 hours and may not exactly correspond to calendar days due to daylight savings, leap seconds, etc. The default is not to remove old log files based on age. ### `maxBackups` integer [​](#audit-maxBackups "Direct link to audit-maxBackups")MaxBackups is the maximum number of old log files to retain. The default is to retain all old log files (though MaxAge may still cause them to get deleted.) ### `maxSize` integer [​](#audit-maxSize "Direct link to audit-maxSize")MaxSize is the maximum size in megabytes of the log file before it gets rotated. It defaults to 100 megabytes. ### `compress` boolean [​](#audit-compress "Direct link to audit-compress")Compress determines if the rotated log files should be compressed using gzip. The default is not to perform compression. ## `loftHost` string [​](#loftHost "Direct link to loftHost")LoftHost holds the domain where the loft instance is hosted. This should not include https or http. E.g. loft.my-domain.com ## `projectNamespacePrefix` string [​](#projectNamespacePrefix "Direct link to projectNamespacePrefix")ProjectNamespacePrefix holds the prefix for loft project namespaces. Omitted defaults to "p-" ## `devPodSubDomain` string [​](#devPodSubDomain "Direct link to devPodSubDomain")DEPRECATED: DevPodSubDomain holds a subdomain in the following form \*.workspace.my-domain.com DevPodSubDomain holds a subdomain in the following form \*.workspace.my-domain.com ## `uiSettings` object [​](#uiSettings "Direct link to uiSettings")UISettings holds the settings for modifying the Loft user interface ### `loftVersion` string [​](#uiSettings-loftVersion "Direct link to uiSettings-loftVersion")LoftVersion holds the current loft version ### `logoURL` string [​](#uiSettings-logoURL "Direct link to uiSettings-logoURL")LogoURL is url pointing to the logo to use in the vCluster Platform UI. This path must be accessible for clients accessing the vCluster Platform UI! ### `faviconURL` string [​](#uiSettings-faviconURL "Direct link to uiSettings-faviconURL")FaviconURL is url pointing to the favicon to use in the vCluster Platform UI. This path must be accessible for clients accessing the vCluster Platform UI! ### `smallLogoURL` string [​](#uiSettings-smallLogoURL "Direct link to uiSettings-smallLogoURL")SmallLogoURL is url pointing to the small logo to use in the vCluster Platform UI. This path must be accessible for clients accessing the vCluster Platform UI! ### `logoBackgroundColor` string [​](#uiSettings-logoBackgroundColor "Direct link to uiSettings-logoBackgroundColor")LogoBackgroundColor is the color value (ex: "#12345") to use as the background color for the logo ### `legalTemplate` string [​](#uiSettings-legalTemplate "Direct link to uiSettings-legalTemplate")LegalTemplate is a text (html) string containing the legal template to prompt to users when authenticating to Loft ### `primaryColor` string [​](#uiSettings-primaryColor "Direct link to uiSettings-primaryColor")PrimaryColor is the color value (ex: "#12345") to use as the primary color ### `sidebarColor` string [​](#uiSettings-sidebarColor "Direct link to uiSettings-sidebarColor")SidebarColor is the color value (ex: "#12345") to use for the sidebar ### `accentColor` string [​](#uiSettings-accentColor "Direct link to uiSettings-accentColor")AccentColor is the color value (ex: "#12345") to use for the accent ### `customCss` string\[] [​](#uiSettings-customCss "Direct link to uiSettings-customCss")CustomCSS holds URLs with custom css files that should be included when loading the UI ### `customJavaScript` string\[] [​](#uiSettings-customJavaScript "Direct link to uiSettings-customJavaScript")CustomJavaScript holds URLs with custom js files that should be included when loading the UI ### `navBarButtons` object\[] [​](#uiSettings-navBarButtons "Direct link to uiSettings-navBarButtons")NavBarButtons holds extra nav bar buttons #### `position` string [​](#uiSettings-navBarButtons-position "Direct link to uiSettings-navBarButtons-position")Position holds the position of the button, can be one of: TopStart, TopEnd, BottomStart, BottomEnd. Defaults to BottomEnd #### `text` string [​](#uiSettings-navBarButtons-text "Direct link to uiSettings-navBarButtons-text")Text holds text for the button #### `link` string [​](#uiSettings-navBarButtons-link "Direct link to uiSettings-navBarButtons-link")Link holds the link of the navbar button #### `icon` string [​](#uiSettings-navBarButtons-icon "Direct link to uiSettings-navBarButtons-icon")Icon holds the url of the icon to display ### `externalURLs` object [​](#uiSettings-externalURLs "Direct link to uiSettings-externalURLs")External URLs that can be called from the UI #### `block` boolean [​](#uiSettings-externalURLs-block "Direct link to uiSettings-externalURLs-block")Block determines if requests to external URLs from the UI should be blocked #### `allow` string\[] [​](#uiSettings-externalURLs-allow "Direct link to uiSettings-externalURLs-allow")Allow specifies which external URLs can be called. In addition to the predefined modules,- "vcluster" (license page, feature descriptions, ...) - "gtm" (google tag manager) - "featurebase" (changelog) any URL can be added to this list. This will allow the UI to make any request to this URL. This is only active when Block is true. ## `vault` object [​](#vault "Direct link to vault")VaultIntegration holds the vault integration configuration ### `enabled` boolean [​](#vault-enabled "Direct link to vault-enabled")Enabled indicates if the Vault Integration is enabled for the project -- this knob only enables the syncing of secrets to or from Vault, but does not setup Kubernetes authentication methods or Kubernetes secrets engines for vclusters. ### `address` string [​](#vault-address "Direct link to vault-address")Address defines the address of the Vault instance to use for this project. Will default to the `VAULT_ADDR` environment variable if not provided. ### `skipTLSVerify` boolean [​](#vault-skipTLSVerify "Direct link to vault-skipTLSVerify")SkipTLSVerify defines if TLS verification should be skipped when connecting to Vault. ### `namespace` string [​](#vault-namespace "Direct link to vault-namespace")Namespace defines the namespace to use when storing secrets in Vault. ### `auth` object [​](#vault-auth "Direct link to vault-auth")Auth defines the authentication method to use for this project. #### `token` string [​](#vault-auth-token "Direct link to vault-auth-token")Token defines the token to use for authentication. #### `tokenSecretRef` object [​](#vault-auth-tokenSecretRef "Direct link to vault-auth-tokenSecretRef")TokenSecretRef defines the Kubernetes secret to use for token authentication. Will be used if `token` is not provided.Secret data should contain the `token` key. ##### `name` string [​](#vault-auth-tokenSecretRef-name "Direct link to vault-auth-tokenSecretRef-name")Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: TODO: Drop `kubebuilder:default` when controller-gen doesn't need it . ##### `key` required string [​](#vault-auth-tokenSecretRef-key "Direct link to vault-auth-tokenSecretRef-key")The key of the secret to select from. Must be a valid secret key. ##### `optional` boolean [​](#vault-auth-tokenSecretRef-optional "Direct link to vault-auth-tokenSecretRef-optional")Specify whether the Secret or its key must be defined ### `syncInterval` string [​](#vault-syncInterval "Direct link to vault-syncInterval")SyncInterval defines the interval at which to sync secrets from Vault. Defaults to `1m.` See for supported formats. ## `disableConfigEndpoint` boolean [​](#disableConfigEndpoint "Direct link to disableConfigEndpoint")DisableLoftConfigEndpoint will disable setting config via the UI and config.management.loft.sh endpoint ## `authenticateVersionEndpoint` boolean [​](#authenticateVersionEndpoint "Direct link to authenticateVersionEndpoint")AuthenticateVersionEndpoint will force authentication for the '/version' endpoint. Will only work with vCluster v0.27 & later ## `cloud` object [​](#cloud "Direct link to cloud")Cloud holds the settings to be used exclusively in vCluster Cloud based environments and deployments. ### `releaseChannel` string [​](#cloud-releaseChannel "Direct link to cloud-releaseChannel")ReleaseChannel specifies the release channel for the cloud configuration. This can be used to determine which updates or versions are applied. ### `maintenanceWindow` object [​](#cloud-maintenanceWindow "Direct link to cloud-maintenanceWindow")MaintenanceWindow specifies the maintenance window for the cloud configuration. This is a structured representation of the time window during which maintenance can occur. #### `dayOfWeek` string [​](#cloud-maintenanceWindow-dayOfWeek "Direct link to cloud-maintenanceWindow-dayOfWeek")DayOfWeek specifies the day of the week for the maintenance window. It should be a string representing the day, e.g., "Monday", "Tuesday", etc. #### `timeWindow` string [​](#cloud-maintenanceWindow-timeWindow "Direct link to cloud-maintenanceWindow-timeWindow")TimeWindow specifies the time window for the maintenance. It should be a string representing the time range in 24-hour format, in UTC, e.g., "02:00-03:00". ## `costControl` object [​](#costControl "Direct link to costControl")CostControl holds the settings related to the Cost Control ROI dashboard and its metrics gathering infrastructure ### `enabled` boolean [​](#costControl-enabled "Direct link to costControl-enabled")Enabled specifies whether the ROI dashboard should be available in the UI, and if the metrics infrastructure that provides dashboard data is deployed ### `global` object [​](#costControl-global "Direct link to costControl-global")Global are settings for globally managed components #### `metrics` object [​](#costControl-global-metrics "Direct link to costControl-global-metrics")Metrics these settings apply to metric infrastructure used to aggregate metrics across all connected clusters ##### `replicas` integer [​](#costControl-global-metrics-replicas "Direct link to costControl-global-metrics-replicas")Replicas is the number of desired replicas. ##### `resources` object [​](#costControl-global-metrics-resources "Direct link to costControl-global-metrics-resources")Resources are compute resource required by the metrics backend ##### `limits` object [​](#costControl-global-metrics-resources-limits "Direct link to costControl-global-metrics-resources-limits")Limits describes the maximum amount of compute resources allowed. More info: ##### `requests` object [​](#costControl-global-metrics-resources-requests "Direct link to costControl-global-metrics-resources-requests")Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: ##### `claims` object\[] [​](#costControl-global-metrics-resources-claims "Direct link to costControl-global-metrics-resources-claims")Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.This field depends on the DynamicResourceAllocation feature gate.This field is immutable. It can only be set for containers. ##### `name` required string [​](#costControl-global-metrics-resources-claims-name "Direct link to costControl-global-metrics-resources-claims-name")Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. ##### `request` string [​](#costControl-global-metrics-resources-claims-request "Direct link to costControl-global-metrics-resources-claims-request")Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. ##### `retention` string [​](#costControl-global-metrics-retention "Direct link to costControl-global-metrics-retention")Retention is the metrics data retention period. Default is 1y ##### `storage` object [​](#costControl-global-metrics-storage "Direct link to costControl-global-metrics-storage") ##### `storageClass` string [​](#costControl-global-metrics-storage-storageClass "Direct link to costControl-global-metrics-storage-storageClass")StorageClass the storage class to use when provisioning the metrics backend's persistent volume If set to "-" or "" dynamic provisioning is disabled If set to undefined or null (the default), the cluster's default storage class is used for provisioning ##### `size` string [​](#costControl-global-metrics-storage-size "Direct link to costControl-global-metrics-storage-size")Size the size of the metrics backend's persistent volume ### `cluster` object [​](#costControl-cluster "Direct link to costControl-cluster")Cluster are settings for each cluster's managed components. These settings apply to all connected clusters unless overridden by modifying the Cluster's spec #### `metrics` object [​](#costControl-cluster-metrics "Direct link to costControl-cluster-metrics")Metrics are settings applied to metric infrastructure in each connected cluster. These can be overridden in individual clusters by modifying the Cluster's spec ##### `replicas` integer [​](#costControl-cluster-metrics-replicas "Direct link to costControl-cluster-metrics-replicas")Replicas is the number of desired replicas. ##### `resources` object [​](#costControl-cluster-metrics-resources "Direct link to costControl-cluster-metrics-resources")Resources are compute resource required by the metrics backend ##### `limits` object [​](#costControl-cluster-metrics-resources-limits "Direct link to costControl-cluster-metrics-resources-limits")Limits describes the maximum amount of compute resources allowed. More info: ##### `requests` object [​](#costControl-cluster-metrics-resources-requests "Direct link to costControl-cluster-metrics-resources-requests")Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: ##### `claims` object\[] [​](#costControl-cluster-metrics-resources-claims "Direct link to costControl-cluster-metrics-resources-claims")Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.This field depends on the DynamicResourceAllocation feature gate.This field is immutable. It can only be set for containers. ##### `name` required string [​](#costControl-cluster-metrics-resources-claims-name "Direct link to costControl-cluster-metrics-resources-claims-name")Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. ##### `request` string [​](#costControl-cluster-metrics-resources-claims-request "Direct link to costControl-cluster-metrics-resources-claims-request")Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. ##### `retention` string [​](#costControl-cluster-metrics-retention "Direct link to costControl-cluster-metrics-retention")Retention is the metrics data retention period. Default is 1y ##### `storage` object [​](#costControl-cluster-metrics-storage "Direct link to costControl-cluster-metrics-storage") ##### `storageClass` string [​](#costControl-cluster-metrics-storage-storageClass "Direct link to costControl-cluster-metrics-storage-storageClass")StorageClass the storage class to use when provisioning the metrics backend's persistent volume If set to "-" or "" dynamic provisioning is disabled If set to undefined or null (the default), the cluster's default storage class is used for provisioning ##### `size` string [​](#costControl-cluster-metrics-storage-size "Direct link to costControl-cluster-metrics-storage-size")Size the size of the metrics backend's persistent volume #### `opencost` object [​](#costControl-cluster-opencost "Direct link to costControl-cluster-opencost")OpenCost are settings applied to OpenCost deployments in each connected cluster. These can be overridden in individual clusters by modifying the Cluster's spec ##### `replicas` integer [​](#costControl-cluster-opencost-replicas "Direct link to costControl-cluster-opencost-replicas")Replicas is the number of desired replicas. ##### `resources` object [​](#costControl-cluster-opencost-resources "Direct link to costControl-cluster-opencost-resources")Resources are compute resource required by the OpenCost backend ##### `limits` object [​](#costControl-cluster-opencost-resources-limits "Direct link to costControl-cluster-opencost-resources-limits")Limits describes the maximum amount of compute resources allowed. More info: ##### `requests` object [​](#costControl-cluster-opencost-resources-requests "Direct link to costControl-cluster-opencost-resources-requests")Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: ##### `claims` object\[] [​](#costControl-cluster-opencost-resources-claims "Direct link to costControl-cluster-opencost-resources-claims")Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.This field depends on the DynamicResourceAllocation feature gate.This field is immutable. It can only be set for containers. ##### `name` required string [​](#costControl-cluster-opencost-resources-claims-name "Direct link to costControl-cluster-opencost-resources-claims-name")Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. ##### `request` string [​](#costControl-cluster-opencost-resources-claims-request "Direct link to costControl-cluster-opencost-resources-claims-request")Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. ### `settings` object [​](#costControl-settings "Direct link to costControl-settings")Settings specify price-related settings that are taken into account for the ROI dashboard calculations. #### `priceCurrency` string [​](#costControl-settings-priceCurrency "Direct link to costControl-settings-priceCurrency")PriceCurrency specifies the currency. #### `averageCPUPricePerNode` object [​](#costControl-settings-averageCPUPricePerNode "Direct link to costControl-settings-averageCPUPricePerNode")AvgCPUPricePerNode specifies the average CPU price per node. ##### `price` number [​](#costControl-settings-averageCPUPricePerNode-price "Direct link to costControl-settings-averageCPUPricePerNode-price")Price specifies the price. ##### `timePeriod` string [​](#costControl-settings-averageCPUPricePerNode-timePeriod "Direct link to costControl-settings-averageCPUPricePerNode-timePeriod")TimePeriod specifies the time period for the price. #### `averageRAMPricePerNode` object [​](#costControl-settings-averageRAMPricePerNode "Direct link to costControl-settings-averageRAMPricePerNode")AvgRAMPricePerNode specifies the average RAM price per node. ##### `price` number [​](#costControl-settings-averageRAMPricePerNode-price "Direct link to costControl-settings-averageRAMPricePerNode-price")Price specifies the price. ##### `timePeriod` string [​](#costControl-settings-averageRAMPricePerNode-timePeriod "Direct link to costControl-settings-averageRAMPricePerNode-timePeriod")TimePeriod specifies the time period for the price. #### `gpuSettings` object [​](#costControl-settings-gpuSettings "Direct link to costControl-settings-gpuSettings")GPUSettings specifies GPU related settings. ##### `enabled` boolean [​](#costControl-settings-gpuSettings-enabled "Direct link to costControl-settings-gpuSettings-enabled")Enabled specifies whether GPU settings should be available in the UI. ##### `averageGPUPrice` object [​](#costControl-settings-gpuSettings-averageGPUPrice "Direct link to costControl-settings-gpuSettings-averageGPUPrice")AvgGPUPrice specifies the average GPU price. ##### `price` number [​](#costControl-settings-gpuSettings-averageGPUPrice-price "Direct link to costControl-settings-gpuSettings-averageGPUPrice-price")Price specifies the price. ##### `timePeriod` string [​](#costControl-settings-gpuSettings-averageGPUPrice-timePeriod "Direct link to costControl-settings-gpuSettings-averageGPUPrice-timePeriod")TimePeriod specifies the time period for the price. #### `controlPlanePricePerCluster` object [​](#costControl-settings-controlPlanePricePerCluster "Direct link to costControl-settings-controlPlanePricePerCluster")ControlPlanePricePerCluster specifies the price of one physical cluster. ##### `price` number [​](#costControl-settings-controlPlanePricePerCluster-price "Direct link to costControl-settings-controlPlanePricePerCluster-price")Price specifies the price. ##### `timePeriod` string [​](#costControl-settings-controlPlanePricePerCluster-timePeriod "Direct link to costControl-settings-controlPlanePricePerCluster-timePeriod")TimePeriod specifies the time period for the price. ## `platformDB` object [​](#platformDB "Direct link to platformDB")PlatformDB holds the settings related to the postgres database that platform uses to store data ### `storageClass` string [​](#platformDB-storageClass "Direct link to platformDB-storageClass")StorageClass sets the storage class for the PersistentVolumeClaim used by the platform database statefulSet. ## `imageBuilder` object [​](#imageBuilder "Direct link to imageBuilder")ImageBuilder holds the settings related to the image builder ### `enabled` boolean [​](#imageBuilder-enabled "Direct link to imageBuilder-enabled")Enabled specifies whether the remote image builder should be available. If it's not available building ad-hoc images from a devcontainer.json is not supported ### `replicas` integer [​](#imageBuilder-replicas "Direct link to imageBuilder-replicas")Replicas is the number of desired replicas. ### `resources` object [​](#imageBuilder-resources "Direct link to imageBuilder-resources")Resources are compute resource required by the buildkit containers #### `limits` object [​](#imageBuilder-resources-limits "Direct link to imageBuilder-resources-limits")Limits describes the maximum amount of compute resources allowed. More info: #### `requests` object [​](#imageBuilder-resources-requests "Direct link to imageBuilder-resources-requests")Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: #### `claims` object\[] [​](#imageBuilder-resources-claims "Direct link to imageBuilder-resources-claims")Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.This field depends on the DynamicResourceAllocation feature gate.This field is immutable. It can only be set for containers. ##### `name` required string [​](#imageBuilder-resources-claims-name "Direct link to imageBuilder-resources-claims-name")Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container. ##### `request` string [​](#imageBuilder-resources-claims-request "Direct link to imageBuilder-resources-claims-request")Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request. ## `database` object [​](#database "Direct link to database")Database represents the database connection settings when deploying the platform with an embedded Kubernetes backed by kine ### `enabled` boolean [​](#database-enabled "Direct link to database-enabled")Enabled defines if the database should be used. ### `dataSource` string [​](#database-dataSource "Direct link to database-dataSource")DataSource is the kine dataSource to use for the database. This depends on the database format. This is optional for the external database. Examples:- mysql: mysql://username:password\@tcp(hostname:3306)/k3s - postgres: postgres\://username:password\@hostname:5432/k3s ### `identityProvider` string [​](#database-identityProvider "Direct link to database-identityProvider")IdentityProvider is the kine identity provider to use when generating temporary authentication tokens for enhanced security. This is optional for the external database. Examples:- aws: RDS IAM Authentication ### `keyFile` string [​](#database-keyFile "Direct link to database-keyFile")KeyFile is the key file to use for the database. This is optional. ### `certFile` string [​](#database-certFile "Direct link to database-certFile")CertFile is the cert file to use for the database. This is optional. ### `caFile` string [​](#database-caFile "Direct link to database-caFile")CaFile is the ca file to use for the database. This is optional. ### `extraArgs` string\[] [​](#database-extraArgs "Direct link to database-extraArgs")ExtraArgs are additional arguments to pass to Kine. --- # SSO Providers | | | Enterprise | | | | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Multiple SSO Providers](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md#multiple-sso-providers) | | | | | | [Single Sign-On](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). The platform supports all major Single-Sign-On (SSO) providers. To configure SSO for the platform, expand the desired SSO provider (for example Google, Okta, GitHub) field listed below and follow the instructions provided. Domain Required Most SSO provider connections do not allow `localhost` and require that you [set up a domain for your platform instance](https://vcluster.com/docs/platform/configure/installation-options/domain.md) first. ## Providers[​](#providers "Direct link to Providers") GitHub 1. **Create A GitHub App** GitHub Apps allow external services to authenticate users through GitHub's OAuth system. To set up SSO for the platform, create a new OAuth App in GitHub: * Navigate to `Settings > Developer Settings > OAuth Apps` in GitHub. * Click on [Create a new OAuth App](https://github.com/settings/applications/new). * Configure the OAuth App with the following settings: * Application name: Platform Login via GitHub * Homepage URL: `https://$VCLUSTER_PRO_HOSTNAME` * Authorization callback URL: `https://$VCLUSTER_PRO_HOSTNAME/auth/github/callback` * After creating the app, note the `$GITHUB_CLIENT_ID` and `$GITHUB_CLIENT_SECRET`. These is required in the next step. info Replace `$VCLUSTER_PRO_HOSTNAME` with your actual platform hostname. 2. **Create platform config for GitHub** To enable GitHub SSO for the platform, use the following YAML configuration to update your platform config (the `config` section of the [`values.yaml`](https://vcluster.com/docs/platform/configure/introduction.md)) or through vCluster Platform UI **Admin > Config**: GitHub SSO Configuration ``` auth: # Tell loft to use github authentication github: # # # REQUIRED CONFIGURATION # # # ClientID of the github application. Can be string literal or pulled from the environment. clientId: $GITHUB_CLIENT_ID # ClientSecret of the github application. Can be string literal or pulled from the environment. clientSecret: $GITHUB_CLIENT_SECRET # Callback URL in the form of https://your-loft-domain/auth/github/callback redirectURI: https://vcluster-platform.my.domain/auth/github/callback # # # OPTIONAL CONFIGURATION # # # (Optional) vCluster Platform queries the following organizations for group information. # Group claims are formatted as "(org):(team)". # For example if a user is part of the "engineering" team of the "coreos" # org, the group claim would include "coreos:engineering". # # If orgs are specified in the config then user MUST be a member of at least one # of the specified orgs to authenticate with loft. orgs: # Organization name in github (not slug, full name) - name: My Organization # (Optional) Names of teams in a github organization. A user will be able to # authenticate if they are members of at least one of these teams. teams: - myteam # (Optional) Required ONLY for GitHub Enterprise. # This is the Hostname of the GitHub Enterprise account listed on the # management console. hostName: my-github.org # (Optional) Required ONLY for GitHub Enterprise. # Used to support self-signed or untrusted CA root certificates. rootCA: /certs/github.ca ``` Configuration options explained: * `clientId`: The Client ID from the GitHub OAuth App. * `clientSecret`: The Client Secret from the GitHub OAuth App. * `redirectURI`: The Authorization callback URL specified in the GitHub OAuth App. * `orgs`: Specify GitHub organizations and teams allowed to access the platform. * `hostName`: Custom GitHub Enterprise hostname (if applicable). * `rootCA`: Path to a custom root certificate for GitHub Enterprise (if applicable). 3. **Authenticate via GitHub** After saving the new configuration, the platform should restart. Users can now log in via GitHub: * Access the platform login page. * Click on the "Login with GitHub" option. * Authenticate with GitHub credentials. * Grant the requested permissions to the GitHub OAuth App. warning Users **must** grant access to the organization configured in step 2. Without this access, they are not going to be able to log in. 4. **Disable Username + Password Authentication** (optional) To turn off password-based authentication, navigate to `Admin > Config` add these two lines to your config: Disable password-based authentication ``` auth: oidc: ... # This is your SSO configuration (make sure this is working!) password: disabled: true # Disable password-based authentication ``` OIDC ## OIDC authentication[​](#oidc-authentication "Direct link to OIDC authentication") [OpenID Connect Specification](https://openid.net/specs/openid-connect-core-1_0.html) is an authentication layer built on top of OAuth 2.0, providing secure and standardized user authentication for applications. Implementing OIDC in the platform offers several benefits: * Enhanced security through token-based authentication * Simplified user management with centralized identity providers * Improved user experience with `single sign-on` (SSO) capabilities 1. **Create platform config for OIDC** To configure OIDC for SSO, use the following YAML configuration to update your platform config (the `config` section of the [`values.yaml`](https://vcluster.com/docs/platform/configure/introduction.md)) or through vCluster Platform UI **Admin > Config**: Platform OIDC configuration ``` auth: # Tell vCluster Platform to allow OIDC for authentication oidc: # # # REQUIRED CONFIGURATION # # # IssuerURL is the URL the provider signs ID Tokens as. # If using Keycloak provide the realm as well, only providing the base URL will return a 404. Ex. https://your-keycloak/realms/YOUR-REALM issuerUrl: https://accounts.google.com # ClientID the JWT must be issued for, the "sub" field. Can be string literal or pulled from the environment. clientId: $CLIENTID # ClientSecret to issue tokens from the OIDC provider. Can be string literal or pulled from the environment. clientSecret: $CLIENTSECRET # Callback URL in the form of https://your-loft-domain/auth/oidc/callback redirectURI: https://vcluster-platform.my.domain/auth/oidc/callback # # # OPTIONAL CONFIGURATION # # # (Optional) Path to a PEM encoded root certificate of the provider. caFile: /tmp/ca-file.crt # (Optional) Specify whether to communicate without validating SSL certificates insecureCa: false # (Optional) UsernameClaim is the JWT field to use as the user's username. # If not set defaults to email. usernameClaim: email # (Optional) If specified, causes claims mapping to username to be prefix with # the provided value. usernamePrefix: my-prefix- # (Optional) If specified, causes the OIDCAuthenticator to try to populate the user's # groups with an ID Token field. groupsClaim: groups # (Optional) If specified, causes claims mapping to group names to be prefixed with the # value. groupsPrefix: group-prefix- # (Optional) If groups is non empty, access is denied if the user is not part of at least one # of the specified groups. This requires groupsClaim to be set! groups: ["my-oidc-group"] # (Optional) If specified, tells the OIDCAuthenticator to try to populate the user's # information from the UserInfo. This might be necessary for slim tokens such as used # by Okta getUserInfo: false # (Optional) Scopes that should be sent to the server. If empty, defaults to "email" and "profile". scopes: ["profile", "email"] # (Optional) Resource is the expected audience for access tokens during token exchange. # Required when using OIDC token exchange with access tokens (RFC 8693). # For Entra ID, set this to the Application (client) ID as a bare GUID. # See: /docs/platform/administer/authentication/token-exchange #resource: "" # (Optional) EmailClaim is the JWT field to use as the user's email. emailClaim: "email" # (Optional) vCluster PlatformUsernameClaim is the JWT field to use as the user's ID (Kubernetes name) and username loftUsernameClaim: "name" # (Optional) PreferredUsername is the JWT field to use as the user's display name preferredUsername: "preferred_username" # (Optional) AllowedExtraClaims are claims of interest that are provided by the OIDC provider but may not already be # covered by a User field. These additional claims of interest will be copied to User.Spec.ExtraClaims. # Like other claims, User.Spec.ExtraClaims is only updated when the User logs in using the OIDC provider. A user must # login after changes to AllowedExtraClaims for them to be reflected in a User's User.Spec.ExtraClaims. allowedExtraClaims: ["department"] ``` Obtaining OIDC credentials To obtain the required `clientId` and `clientSecret`, refer to your OIDC provider's documentation. Common providers include: * [Google Identity](https://developers.google.com/identity/openid-connect/openid-connect) * [Okta OpenID](https://developer.okta.com/docs/concepts/oauth-openid/) * [Keycloak](https://www.keycloak.org/documentation) 2. **Authenticate using OIDC** After saving the new configuration, the platform should restart. You should then be able to log in using your OIDC provider. 3. **Disable Username + Password Authentication** (optional) To turn off password-based authentication, navigate to `Admin > Config` add these two lines to your config: Disable password-based authentication ``` auth: oidc: ... # This is your SSO configuration (make sure this is working!) password: disabled: true # Disable password-based authentication ``` Once OIDC is configured, CI/CD pipelines and automated systems can exchange OIDC tokens for platform access keys without storing static credentials. See [OIDC token exchange](https://vcluster.com/docs/platform/administer/authentication/token-exchange.md). GitLab This guide explains how to integrate GitLab Single Sign-On (SSO) with the platform, allowing users to authenticate using their GitLab credentials. 1. **Create A GitLab App** Navigate to the [GitLab User Settings > Applications](https://gitlab.com/-/profile/applications) page and create a new application with the following settings: * Application name: Platform Login via GitLab * Redirect URI: = `https:// + $VCLUSTER_PRO_HOSTNAME + /auth/gitlab/callback` * Scopes: `read_user` + `openid` info GitLab generates a `$GITLAB_CLIENT_ID` and `$GITLAB_CLIENT_SECRET` for your OAuth application. These credentials are required for the next step. 2. **Create Platform config For GitLab** To set up GitLab SSO in the platform, use the following YAML configuration to update your platform config (the `config` section of the [`values.yaml`](https://vcluster.com/docs/platform/configure/introduction.md)) or through vCluster Platform UI **Admin > Config**: GitLab SSO configuration ``` auth: # Tell loft to use gitlab authentication gitlab: # # # REQUIRED CONFIGURATION # # # ClientID for the gitlab authentication. Can be string literal or pulled from the environment. clientId: $GITLAB_CLIENT_ID # ClientSecret for the gitlab authentication. Can be string literal or pulled from the environment. clientSecret: $GITLAB_CLIENT_SECRET # Callback URL in the form of https://your-loft-domain/auth/gitlab/callback redirectURI: https://vcluster-platform.my.domain/auth/gitlab/callback # # # OPTIONAL CONFIGURATION # # # (Optional) BaseURL of the gitlab instance, # default = https://gitlab.com baseURL: https://my-gitlab-instance.com # (Optional) Optional groups whitelist, communicated through the "groups" scope. # If groups is omitted, all of the user's GitLab groups are returned. # If groups is provided, this acts as a whitelist - only the user's GitLab # groups that are in the configured groups below will go into the groups claim. # Conversely, if the user is not in any of the configured groups, the user will # not be authenticated. groups: - my-group ``` 3. **Authenticate via GitLab** After saving the new configuration, the platform should restart automatically. Users can now log in using their GitLab credentials. 4. **Disable Username + Password Authentication** (optional) To turn off password-based authentication, navigate to `Admin > Config` add these two lines to your config: Disable password-based authentication ``` auth: oidc: ... # This is your SSO configuration (make sure this is working!) password: disabled: true # Disable password-based authentication ``` Microsoft 1. **Create platform config for Microsoft** The platform is able to use Microsoft Single Sign-On (SSO) OAuth2 flow to identify the end user through their Microsoft account. Tenant Must Be Specified in Microsoft Connector Config to Sync SSO Groups The groups claim is only supported when tenant is specified in Microsoft connector config. Furthermore, tenant must also be configured to either ` or `. In order for the platform to be able to list groups on behalf of logged in user, an explicit organization administrator consent is required. To obtain the consent do the following: The platform can use Microsoft OAuth2 flow to identify end users through their Microsoft accounts. The groups claim is only supported when the tenant is specified in the Microsoft connector config. The tenant must be configured to either `` or ``. For the platform to list groups on behalf of the logged-in user, explicit organization administrator consent is required. To obtain consent: * When registering the platform application on , add an explicit Directory.Read.All permission to the list of Delegated Permissions. * Open the following link in your browser and log in under an organization administrator account: `https://login.microsoftonline.com//adminconsent?client_id=` To configure the platform to use Microsoft for SSO, use the following YAML configuration to update your platform config (the `config` section of the [`values.yaml`](https://vcluster.com/docs/platform/configure/introduction.md)) or through vCluster Platform UI **Admin > Config**: Microsoft SSO configuration ``` auth: # Tell the platform to use microsoft authentication microsoft: # # # REQUIRED CONFIGURATION # # # ClientID of the microsoft application. Can be string literal or pulled from the environment. clientId: $CLIENTID # ClientSecret of the microsoft application. Can be string literal or pulled from the environment. clientSecret: $CLIENTSECRET # Callback URL in the form of https://your-loft-domain/auth/microsoft/callback redirectURI: https://vcluster-platform.my.domain/auth/microsoft/callback # # # OPTIONAL CONFIGURATION # # # (Optional) tenant configuration parameter controls what kinds of accounts # may be authenticated in loft. By default, all types of Microsoft # accounts (consumers and organizations) can authenticate in loft via Microsoft. # To change this, set the tenant parameter to one of the following: # common - both personal and business accounts can authenticate via Microsoft (default) # consumers - only personal accounts can authenticate in loft # organizations - only business/school accounts can authenticate in loft # 'tenant uuid' or 'tenant name' - only accounts belonging to specific tenant # identified by either 'tenant uuid' or 'tenant name' can authenticate in loft tenant: common # (Optional) It is possible to require a user to be a member of a particular # group in order to be successfully authenticated in loft. groups: [] # (Optional) Configuration option restricts the list to include only security groups. # By default all groups (security, Office 365, mailing lists) are included. onlySecurityGroups: false # (Optional) Restrict the groups claims to include only the user’s groups # that are in the configured groups useGroupsAsWhitelist: false ``` Optional configuration parameters * `tenant`: Controls what kinds of accounts may be authenticated. Options include: * `common`: Both personal and business accounts (default) * `consumers`: Only personal accounts * `organizations`: Only business/school accounts * `'tenant uuid'` or `'tenant name'`: Only accounts belonging to a specific tenant * `groups`: Require users to be members of specific groups for authentication * `onlySecurityGroups`: Restricts the list to include only security groups * `useGroupsAsWhitelist`: Restricts the groups claims to include only the user's groups that are in the configured groups Obtaining ClientID and ClientSecret To obtain the `$CLIENTID` and `$CLIENTSECRET` values: * Register your application in the Azure Portal * Navigate to "App registrations" * Select your application and find the "Application (client) ID" for `$CLIENTID` * Under "Certificates & secrets", create a new client secret for `$CLIENTSECRET` 2. **Authenticate via Microsoft** After saving the new platform configuration, the platform should restart. Users should then be able to log in via Microsoft. 3. **Disable Username + Password Authentication** (optional) To turn off password-based authentication, navigate to `Admin > Config` add these two lines to your config: Disable password-based authentication ``` auth: oidc: ... # This is your SSO configuration (make sure this is working!) password: disabled: true # Disable password-based authentication ``` Google ## Introduction[​](#introduction "Direct link to Introduction") Google Single Sign-On (SSO) allows users to access the platform using their Google credentials. This integration enhances security and simplifies user management by leveraging Google's authentication infrastructure. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before setting up Google SSO, ensure you have: * A Google Account with admin privileges * Administrator access to the platform * Access to GPC console for creating OAuth2.0 credentials ## Configuration steps[​](#configuration-steps "Direct link to Configuration steps") 1. **Create platform config for Google** The platform can use Google's OpenID Connect provider as an authentication source. Obtaining Google OAuth2.0 credentials To get the required `clientId` and `clientSecret`, follow these steps: * Go to the [GCP Console](https://console.cloud.google.com/) * Navigate to "APIs & Services" > "Credentials" * Click "Create Credentials" and select "OAuth client ID" * Follow the prompts to set up your OAuth consent screen and create your credentials To configure the platform to use Google for SSO, use the following YAML configuration to update your platform config (the `config` section of the [`values.yaml`](https://vcluster.com/docs/platform/configure/introduction.md)) or through vCluster Platform UI **Admin > Config**: Google SSO Configuration ``` auth: # Tell vCluster Platform to use google authentication google: # # # REQUIRED CONFIGURATION # # # ClientID for the google authentication. Can be string literal or pulled from the environment. clientId: $CLIENTID # ClientSecret for the google authentication. Can be string literal or pulled from the environment. clientSecret: $CLIENTSECRET # Callback URL in the form of https://your-loft-domain/auth/google/callback redirectURI: https://vcluster-platform.my.domain/auth/google/callback # # # OPTIONAL CONFIGURATION # # # (Optional) defaults to "profile" and "email" scopes: ["profile", "email"] # (Optional) list of whitelisted domains. If this field is nonempty, # only users from a listed domain will be allowed to log in hostedDomains: [] # (Optional) list of whitelisted groups. If this field is nonempty, # only users from a listed group will be allowed to log in groups: [] # (Optional) path to service account json. If nonempty, # and groups claim is made, will use authentication from file to # check groups with the admin directory api serviceAccountFilePath: /path/to/service/account.json # (Optional) Required if serviceAccountFilePath is set. The email of # a GSuite super user which the service account will impersonate # when listing groups adminEmail: myuser@mydomain.com ``` Security best practice Always store sensitive information like `clientSecret` securely. Consider using environment variables or secure secret management solutions. note Replace the `redirectURI` with your actual domain where the platform is hosted. 2. **Authenticate via Google** After saving the new configuration, the platform should restart itself. To authenticate: * Navigate to the platform login page * Click on the "Login with Google" button * You should be redirected to Google's login page * Enter your Google credentials * Grant the necessary permissions when prompted * You should be redirected back to the platform, now authenticated 3. **Disable Username + Password Authentication** (optional) To turn off password-based authentication, navigate to `Admin > Config` add these two lines to your config: Disable password-based authentication ``` auth: oidc: ... # This is your SSO configuration (make sure this is working!) password: disabled: true # Disable password-based authentication ``` Okta This guide walks you through the process of setting up Single Sign-On (SSO) with Okta for the platform. By following these steps, you'll enable seamless authentication for your users through Okta. 1. **Create A New App In Okta** Start by creating a new Okta App for the platform. ![Create App for the platform](/docs/assets/images/create-app-okta-4681b6b319a5d19336555fc2b8d502f7.png) Okta - Create a new App in Okta Select "Web" App and ensure OpenID Connect is the single sign-on method. ![Create App for the platform](/docs/assets/images/okta-select-web-9a570d1adb3d54ea7cc1d920952f0029.png) Okta - Web App with OpenID Connect as single sign on method On the next screen, configure the login redirect URIs to include your platform instance domain: Configure login redirect URI ``` # Replace ${my-loft-domain.com} with your platform domain https://${my-loft-domain.com}/auth/oidc/callback ``` ![Create App for the platform](/docs/assets/images/okta-app-settings-f5241d9df1230d933fba957201d841b3.png) Okta - The App settings for the platform 2. **Enable Refresh Tokens** After creating the Okta app for the platform, enable "Refresh Token" under "Allowed grant types". This step is crucial as it allows users to maintain their session without frequent re-logins. ![Create App for the platform](/docs/assets/images/okta-refresh-token-37ea20373c65e4627fe8daf786e9d39e.png) Okta - App Settings: Enable Refresh Tokens 3. **Enable Group Claims** If you want to propagate the users groups to the platform, then make sure the Group Filters in Okta are set accordingly. If you want to propagate all groups, add a RegEx filter with '.\*' ![Okta Assign People](/docs/assets/images/okta-group-filter-c4a985dc478b2c166dad34ec6883a375.png) Okta - Propagate User Groups To the platform 4. **Configure platform to use Okta** ![Create App for the platform](/docs/assets/images/okta-client-secret-3cf976917eae122a4a0ee0d6ec0aad3a.png) Okta - Client ID and Secret For App Navigate to `Admin > Config` in the platform and enter the following configuration: Configure the platform for Okta integration ``` auth: oidc: issuerUrl: "https://${MY-OKTA-SUBDOMAIN}.okta.com" clientId: CLIENT_ID clientSecret: CLIENT_SECRET groupsClaim: groups # This is needed because Okta uses thin id tokens # that do not contain the groups directly getUserInfo: true ``` The `groupsClaim` field specifies where to find group information in the token, and `getUserInfo` is set to `true` to retrieve additional user information from Okta. ![Okta Assign Peoples](/docs/assets/images/okta-loft-config-8ba66cb18996bf57b7b78e72295afb9d.png) Configure the platform To Use Okta 5. **Add Users via Okta Assignments** Assign the appropriate Users and Groups to access the platform in Okta. ![Okta Assign People](/docs/assets/images/okta-assign-people-fdb247971cf7e9f76c4df75918b4ad94.png) Okta - Assign Users & Groups To the platform Once users or their groups are assigned to the platform, they can log in via Okta: ![Okta Assign People](/docs/assets/images/okta-loft-login-6a2a10a6fbca21f1a6d2a45cf5add29b.png) The platform - SSO via Okta 6. **Disable Username + Password Authentication** (optional) To turn off password-based authentication, navigate to `Admin > Config` add these two lines to your config: Disable password-based authentication ``` auth: oidc: ... # This is your SSO configuration (make sure this is working!) password: disabled: true # Disable password-based authentication ``` SAML 2.0 ## Introduction to SAML authentication[​](#introduction-to-saml-authentication "Direct link to Introduction to SAML authentication") [Security Assertion Markup Language (SAML) Specification](https://saml.xml.org/saml-specifications) is an open standard for exchanging authentication and authorization data between parties, particularly between an identity provider and a service provider. SAML is useful for implementing secure single sign-on (SSO) in enterprise environments, allowing users to access multiple applications with a single set of credentials. Refresh tokens are not supported The SAML 2.0 connector in the platform does not support refresh tokens as the SAML 2.0 protocol does not provide a way to requery a provider without interaction. For more information, see the [dex documentation for SAML 2.0](https://dexidp.io/docs/connectors/saml/). 1. **Create platform Config For SAML v2.0** The platform can use SAML v2.0 flow to identify the end user. To configure the platform to use SAML for SSO, follow these steps: 1. Use the following YAML configuration to update your platform config (the `config` section of the [`values.yaml`](https://vcluster.com/docs/platform/configure/introduction.md)) or through vCluster Platform UI **Admin > Config**: 2. Obtain the necessary values from your SAML identity provider (IdP) for the required fields. Configure the platform to use SAML ``` auth: # Tell vCluster Platform to use saml authentication saml: # # # REQUIRED CONFIGURATION # # # SSO URL used for POST value. ssoURL: https://saml.example.com/sso # CA to use when validating the signature of the SAML response. ca: /path/to/ca.pem # Alternatively, CA can be provided inline as a base64-encoded blob # caData: (RAW base64-encoded PEM encoded CA) # Callback URL in the form of https://your-vcluster-platform-domain/auth/saml/callback redirectURI: https://vcluster-platform.my.domain/auth/saml/callback # Name of attributes in the returned assertions to map to ID token claims. usernameAttr: name emailAttr: email groupsAttr: groups # optional # # # OPTIONAL CONFIGURATION # # # (Optional) List of groups to filter access based on membership allowedGroups: ["Admins"] # (Optional) To skip signature validation, uncomment the following field. This should # only be used during testing and may be removed in the future insecureSkipSignatureValidation: true # (Optional) Manually specify vCluster Platform's Issuer value. # When provided vCluster Platform will include this as the Issuer value during AuthnRequest. # It will also override the redirectURI as the required audience when evaluating # AudienceRestriction elements in the response. entityIssuer: https://vcluster-platform.my.domain/auth/saml/callback # (Optional) Issuer value expected in the SAML response. ssoIssuer: https://saml.example.com/sso # (Optional) Delimiter for splitting groups returned as a single string. # # By default, multiple groups are assumed to be represented as multiple # attributes with the same name. # # If "groupsDelim" is provided groups are assumed to be represented as a # single attribute and the delimiter is used to split the attribute's value # into multiple groups. groupsDelim: ", " # (Optional) Requested format of the NameID. # # The NameID value is is mapped to the user ID of the user. This can be an # abbreviated form of the full URI with just the last component. For example, # if this value is set to "emailAddress" the format will resolve to: # # urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress # # If no value is specified, this value defaults to: # # urn:oasis:names:tc:SAML:2.0:nameid-format:persistent # nameIDPolicyFormat: persistent ``` 2. **Authenticate via SAML** 1. The platform should restart automatically. 2. You should now be able to log in via SAML. 3. Test the authentication by logging out and logging back in. 3. **Disable Username + Password Authentication** (optional) To turn off password-based authentication, navigate to `Admin > Config` add these two lines to your config: Disable password-based authentication ``` auth: oidc: ... # This is your SSO configuration (make sure this is working!) password: disabled: true # Disable password-based authentication ``` LDAP 1. **Choose DEX\_HOSTNAME And Configure DNS** The platform uses the CNCF project [dex](https://dexidp.io/) for single sign-on. The easiest case is this one: * `$VCLUSTER_PRO_HOSTNAME = vcluster-platform.mycompany.tld` (where platform is running) * `$DEX_HOSTNAME = dex.mycompany.tld` (where dex should be running) 2. **Create Dex Config For LDAP** Create the file `dex-config.yaml` with the following dex configuration: ``` ingress: enabled: true hosts: - host: dex.yourcompany.tld # Use $DEX_HOSTNAME paths: - path: / config: issuer: https://dex.yourcompany.tld # "https://" + $DEX_HOSTNAME connectors: - type: ldap id: ldap name: LDAP config: host: myldap.company.tld:636 # Your LDAP server hostname:port # insecureNoSSL: true # Not recommended but required if not using TLS (port 389) # insecureSkipVerify: true # Not recommended but required for self-signed certificates # rootCAData: ( base64 encoded PEM file ) # startTLS: true # Use ldap:// instead of ldaps:// protocol # The DN and password for an application service account. The connector uses # these credentials to search for users and groups. Not required if the LDAP # server provides access for anonymous auth. # Please note that if the bind password contains a `$`, it has to be saved in an # environment variable which should be given as the value to `bindPW`. bindDN: uid=serviceaccount,cn=users,dc=example,dc=com bindPW: password # User search maps a username and password entered by a user to a LDAP entry. userSearch: # BaseDN to start the search from. It will translate to the query # "(&(objectClass=person)(uid=))". baseDN: cn=users,dc=example,dc=com # Optional filter to apply when searching the directory. filter: "(objectClass=person)" # username attribute used for comparing user entries. This will be translated # and combined with the other filter as "(=)". username: uid # The following three fields are direct mappings of attributes on the user entry. # String representation of the user. idAttr: uid # Required. Attribute to map to Email. emailAttr: mail # Maps to display name of users. No default value. nameAttr: name # Group search queries for groups given a user entry. groupSearch: # BaseDN to start the search from. It will translate to the query # "(&(objectClass=group)(member=))". baseDN: cn=groups,dc=freeipa,dc=example,dc=com # Optional filter to apply when searching the directory. filter: "(objectClass=group)" # Represents group name. nameAttr: name # Following list contains field pairs that are used to match a user to a group. It adds an additional # requirement to the filter that an attribute in the group must match the user's # attribute value. userMatchers: - userAttr: uid groupAttr: member staticClients: - name: vCluster Platform id: loft # Define a $DEX_CLIENT_ID secret: XXXXXXXXXXXXXX # Define a $DEX_CLIENT_SECRET (can be any secret key) redirectURIs: - "https://vcluster-platform.mycompany.tld/auth/oidc/callback" # vCluster Platform URL + /auth/oidc/callback oauth2: skipApprovalScreen: true web: http: 0.0.0.0:5556 storage: type: kubernetes config: inCluster: true ``` For details about configuring dex for LDAP, take a look at the [dex documentation for LDAP](https://dexidp.io/docs/connectors/ldap/). 3. **Deploy Dex via Helm** After creating the file `dex-config.yaml`, you can now install dex via Helm: ``` helm install dex dex --repo https://charts.dexidp.io \ --create-namespace --namespace dex \ -f dex-config.yaml \ --wait ``` 4. **Configure the platform to use dex for authentication** To tell the platform to use dex for SSO, navigate to `Admin > Config` in the platform and adjust your config as shown below: Platform OIDC config for dex ``` auth: oidc: issuerUrl: https://dex.mycompany.tld # Use $DEX_HOSTNAME (see above) clientId: "" # Use $DEX_CLIENT_ID (see above) clientSecret: "" # Use $DEX_CLIENT_SECRET (see above) type: "github" # Optional: SSO Login Button Icon ("", github, gitlab, microsoft, google) usernameClaim: "email" # Optional: Which part of the dex token to use as vCluster Platform username (default: email) usernamePrefix: "" # Optional: Add prefix to usernameClaim for vCluster Platform username groupsClaim: "groups" # Optional: Add Kubernetes groups for this user groupsPrefix: "loft-" # Optional: Prefix for Kubernetes groups caFile: "" # Optional: Path to a CA cert of dex within the vCluster Platform container (default: '') ``` 5. **Authenticate via Dex + LDAP** After saving the new platform configuration, the platform is going to restart itself and you should be able to log in via LDAP and dex. 6. **Disable Username + Password Authentication** (optional) To turn off password-based authentication, navigate to `Admin > Config` add these two lines to your config: Disable password-based authentication ``` auth: oidc: ... # This is your SSO configuration (make sure this is working!) password: disabled: true # Disable password-based authentication ``` Other (OpenShift, LinkedIn ...) Platform supports a variety of other auth providers. You can follow this generic guide to configure SSO for any of the auth providers, including: * OpenShift * LinkedIn * AuthProxy * Bitbucket Cloud * Atlassian Crowd * Gitea 1. **Choose DEX\_HOSTNAME And Configure DNS** The platform uses the CNCF project [dex](https://dexidp.io/) for single sign-on. The easiest case is this one: * `$VCLUSTER_PRO_HOSTNAME = vcluster-platform.mycompany.tld` (where platform is running) * `$DEX_HOSTNAME = dex.mycompany.tld` (where dex should be running) 2. **Create Dex Config For $OTHER\_PROVIDER** Create the file `dex-config.yaml` with the following dex configuration: dex-config.yaml ``` ingress: enabled: true hosts: - host: dex.yourcompany.tld # Use $DEX_HOSTNAME paths: - path: / config: issuer: https://dex.yourcompany.tld # "https://" + $DEX_HOSTNAME connectors: - {YOUR_CONNECTOR_CONFIG} # SEE LINKS BELOW! (do not remove the '-' and indent correctly) staticClients: - name: Loft id: loft # Define a $DEX_CLIENT_ID secret: XXXXXXXXXXXXXX # Define a $DEX_CLIENT_SECRET (can be any secret key) redirectURIs: - 'https://vcluster-platform.mycompany.tld/auth/oidc/callback' # vCluster Platform URL + /auth/oidc/callback oauth2: skipApprovalScreen: true web: http: 0.0.0.0:5556 storage: type: kubernetes config: inCluster: true ``` Check the official [artifacthub.io](https://artifacthub.io/packages/helm/dex/dex) page for all available options. To fill the `connectors` section shown above, go to the appropriate docs page for the auth provider you want to use in combination with dex: * [Microsoft](https://dexidp.io/docs/connectors/microsoft/) * [Google](https://dexidp.io/docs/connectors/google/) * [OpenID Connect](https://dexidp.io/docs/connectors/oidc/) * [OpenShift](https://dexidp.io/docs/connectors/openshift/) * [LinkedIn](https://dexidp.io/docs/connectors/linkedin/) * [AuthProxy](https://dexidp.io/docs/connectors/authproxy/) * [Bitbucket Cloud](https://dexidp.io/docs/connectors/bitbucketcloud/) * [Atlassian Crowd](https://dexidp.io/docs/connectors/atlassian-crowd/) * [Gitea](https://dexidp.io/docs/connectors/gitea/) 3. **Deploy Dex via Helm** After creating the file `dex-config.yaml`, you can now install dex via Helm: ``` helm install dex dex --repo https://charts.dexidp.io \ --create-namespace --namespace dex \ -f dex-config.yaml \ --wait ``` 4. **Configure the platform to use dex for authentication** To tell the platform to use dex for SSO, navigate to `Admin > Config` in the platform and adjust your config as shown below: Platform OIDC config for dex ``` auth: oidc: issuerUrl: https://dex.mycompany.tld # Use $DEX_HOSTNAME (see above) clientId: "" # Use $DEX_CLIENT_ID (see above) clientSecret: "" # Use $DEX_CLIENT_SECRET (see above) type: "github" # Optional: SSO Login Button Icon ("", github, gitlab, microsoft, google) usernameClaim: "email" # Optional: Which part of the dex token to use as vCluster Platform username (default: email) usernamePrefix: "" # Optional: Add prefix to usernameClaim for vCluster Platform username groupsClaim: "groups" # Optional: Add Kubernetes groups for this user groupsPrefix: "loft-" # Optional: Prefix for Kubernetes groups caFile: "" # Optional: Path to a CA cert of dex within the vCluster Platform container (default: '') ``` 5. **Authenticate via dex + $OTHER\_PROVIDER** After saving the new platform configuration, platform is going to restart itself and you should be able to log in via $OTHER\_PROVIDER and dex. 6. **Disable Username + Password Authentication** (optional) To turn off password-based authentication, navigate to `Admin > Config` add these two lines to your config: Disable password-based authentication ``` auth: oidc: ... # This is your SSO configuration (make sure this is working!) password: disabled: true # Disable password-based authentication ``` ## Multiple SSO providers[​](#multiple-sso-providers "Direct link to Multiple SSO providers") In the Platform, you can also configure *multiple* SSO providers together. This can be a very handy feature if you have multiple groups using different authentication providers based on team or geographical region. Configuration of multiple SSO providers *of the same type* is also possible. The latter is configured by using the "connectors" option. Below is an example platform config "auth" section that contains three SSO providers (one OIDC, and two GitHub providers). Users logging into the platform with such a configuration present see three options for authentication as well as the default username/password box (which you can turn off if desired). Add the following YAML configuration to your platform config (the `config` section of the [`values.yaml`](https://vcluster.com/docs/platform/configure/introduction.md)) or through vCluster Platform UI **Admin > Config**: Multiple SSO providers ``` auth: github: clientId: test123 clientSecret: test123 redirectURI: https://my-vcluster-platform-url/auth/github/callback oidc: clientId: test clientSecret: test issuerUrl: https://example-issuer.com redirectURI: https://my-vcluster-platform-url/auth/oidc/callback # If you want to use multiple SSO configs of the same type you can use the connectors option connectors: # the id of the connector that will show up in the vcluster-platform callback url - id: test # name to show in the UI displayName: Test GitHub # any regular sso config github: clientId: test clientSecret: test # Make sure to exchange test here with the connector id if you change that redirectURI: https://my-vcluster-platform-url/auth/test/callback ``` ## Token exchange for CI/CD[​](#token-exchange-for-cicd "Direct link to Token exchange for CI/CD") OIDC connectors support [RFC 8693](https://datatracker.ietf.org/doc/html/rfc8693) token exchange, which lets CI/CD pipelines authenticate with the platform without storing long-lived access keys. Instead of distributing static credentials, a pipeline requests a short-lived OIDC token from its runtime environment (GitHub Actions, Azure DevOps, and similar providers) and exchanges it for a platform access key scoped to that pipeline's identity. Token exchange works with any OIDC connector configured under `auth.connectors`, and with the primary OIDC provider configured under `auth.oidc`. The exchanged access key inherits the permissions of the platform user that maps to the token's identity claims. See [OIDC token exchange](https://vcluster.com/docs/platform/administer/authentication/token-exchange.md) for endpoint reference, configuration examples, and troubleshooting. ## SSO group sync[​](#sso-group-sync "Direct link to SSO group sync") The platform can be configured to allow user authentication using Single-Sign-On (SSO). SSO is an authentication method that enables users to access multiple applications with a single set of login credentials. This feature allows users with a valid account on another service (for example, GitHub) to authenticate and log into the platform using that service. While this approach simplifies user management for administrators, a mechanism is still needed to ensure appropriate permissions are applied to these users. SSO groups address this need. Most SSO providers allow administrators to configure data shared with authenticating platforms. Among this shared data, the list of groups to which the authenticating user belongs is particularly important. Upon SSO authentication in the platform, this group data is inspected. The user is automatically joined to any Teams that include any of the provided group names in the `SSO Groups as Members` field. Any SSO group names not set in any Team's `SSO Groups as Members` field is going to be dynamically created as a new Team, with the group name automatically set in the `SSO Groups as Members` field. This group behavior allows administrators to create Teams in the platform that correspond to teams (groups) in the SSO identity provider, set the appropriate policy for those Teams, and enable users to be automatically assigned to the appropriate team, and thus privileges, upon logging in using SSO. info If a user belongs to multiple SSO groups that correspond to different Teams in the platform, they are going to be added to all matching Teams. ## Create a team with SSO group membership[​](#create-a-team-with-sso-group-membership "Direct link to Create a team with SSO group membership") 1. Go to Access & Secrets > Users & Roles and select the Teams tab. 2. Click the Add Team button. 3. In the configuration sheet that opens, give your new team a name by replacing the 'my-team' placeholder name, or by updating the manifest YAML 'metadata.name' field. 4. In the Team Members tab enter any desired groups into the SSO groups field. You can add as many groups as you would like here. These group names must exactly match the group name that the SSO provider shares with the platform during SSO authentication. 5. Make any additional desired modifications to your new Team. 6. Click on the Save Changes button. --- # Disable the local admin account after SSO setup After configuring [single sign-on (SSO)](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md) for vCluster Platform, disabling the local admin account is a recommended security practice for enterprise environments. This prevents unauthorized access through the default credentials and ensures all authentication flows through your identity provider. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before disabling the local admin account: * SSO is configured and tested successfully * At least one SSO user has global admin permissions * You have verified SSO login works with the expected permissions * You have kubectl access to the cluster running vCluster Platform Verify SSO access first Ensure you can log in using SSO with global admin permissions before disabling the local admin. Locking yourself out requires recovery steps. ## Disable the admin account[​](#disable-the-admin-account "Direct link to Disable the admin account") There are two steps to fully disable the local admin: 1. Lock the admin user in the UI 2. Prevent the admin from being recreated on upgrades ### Lock the admin user[​](#lock-the-admin-user "Direct link to Lock the admin user") 1. Log in to vCluster Platform using your SSO credentials with global admin permissions. 2. Go to Access & Secrets > Users & Roles. 3. In the admin user row, click the actions menu and select Lock User. The admin account is now locked. The user cannot log in and any access keys generated for the admin will stop working. ### Prevent admin recreation[​](#prevent-admin-recreation "Direct link to Prevent admin recreation") To prevent the admin account from being recreated during upgrades, update your Helm values: vcluster-platform-values.yaml ``` admin: create: false ``` Apply the configuration: ``` helm upgrade vcluster-platform loft/vcluster-platform \ --namespace vcluster-platform \ --values vcluster-platform-values.yaml \ --reuse-values ``` note If you previously had admin credentials in your values file, remove them after locking the account. Leaving stale credentials in configuration files is a security risk. ## Recovery[​](#recovery "Direct link to Recovery") If SSO becomes unavailable and you need to regain access: 1. Re-enable password authentication if it was disabled. Run the following command to generate a new configuration with password login enabled: ``` kubectl get secrets/loft-manager-config -n vcluster-platform \ -o jsonpath="{.data.config}" | base64 -d | \ yq "del(.auth.password.disabled)" | base64 ``` Copy the output, then edit the secret: ``` kubectl edit secret loft-manager-config -n vcluster-platform ``` Replace `.data.config` with the new value and restart the vCluster Platform pods. 2. Reset the admin password using kubectl. See [Reset Admin Password](https://vcluster.com/docs/platform/administer/authentication/reset-admin-password.md) for detailed instructions. 3. Unlock the admin user: * Log in with the reset password * Go to Access & Secrets > Users & Roles * Click the actions menu on the admin user row and select **Unlock User** After regaining access, investigate the SSO issue, and fix it before re-locking the admin account. --- # Enforce storage quotas per tenant cluster vCluster Platform [project quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md) cap aggregate storage consumption across all tenant clusters in a project, but they don't distribute limits across namespaces or enforce per-PVC caps inside individual tenant clusters. Per-cluster enforcement fills that gap. A ResourceQuota or Kyverno policy applied inside the tenant cluster controls how much storage any single namespace can consume and how individual PVCs may be sized. This guide covers two patterns for per-cluster enforcement. Both can be applied automatically at provisioning time through templates so that every new tenant cluster starts with the right guardrails in place. Use ResourceQuota for namespace-level aggregate limits. Use Kyverno when you need per-PVC size limits or storage class allowlists. Use both when tenants need bounded total usage and per-object guardrails. ## Pattern 1: ResourceQuota inside the tenant cluster[​](#pattern-1-resourcequota-inside-the-tenant-cluster "Direct link to Pattern 1: ResourceQuota inside the tenant cluster") A Kubernetes ResourceQuota object enforces aggregate limits on PVC count and total storage requests. It's namespace-scoped, so enforcing limits across all namespaces in the tenant cluster requires one ResourceQuota per namespace. Adding one to a tenant cluster template deploys it inside every tenant cluster created from that template. ### Add a ResourceQuota to a tenant cluster template[​](#add-a-resourcequota-to-a-tenant-cluster-template "Direct link to Add a ResourceQuota to a tenant cluster template") In the `objects` field of a VirtualClusterTemplate, include a ResourceQuota for each namespace you want to constrain. vCluster Platform applies the objects inside the tenant cluster at provisioning time. ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterTemplate metadata: name: standard-tenant spec: template: helmRelease: chart: version: "0.25.0" objects: |- apiVersion: v1 kind: ResourceQuota metadata: name: storage-quota namespace: default spec: hard: requests.storage: "100Gi" count/persistentvolumeclaims: "20" ``` These two constraints are independent aggregate limits. `requests.storage` caps the combined total storage across all PVCs in the namespace, not the size of any individual PVC. In this example, the namespace cannot have PVCs that total more than 100Gi of space. `count/persistentvolumeclaims` caps the number of PVCs regardless of their size. In this example, the namespace cannot have more than 20 PVCs, even if their total combined storage is less than 100Gi. With this example, a tenant could create 20 PVCs of 5Gi each (100Gi combined, both limits reached simultaneously), or 2 PVCs of 50Gi each (100Gi combined, only the storage limit reached). What is not possible is more than 20 PVCs or any number of PVCs totaling more than 100Gi. ### Scope to a storage class[​](#scope-to-a-storage-class "Direct link to Scope to a storage class") To limit storage from a specific storage class, add a scoped resource name under the same `spec.hard` key: ``` spec: hard: ceph-rbd.storageclass.storage.k8s.io/requests.storage: "50Gi" ``` ### Apply across namespaces[​](#apply-across-namespaces "Direct link to Apply across namespaces") Repeat the ResourceQuota block for each namespace that requires enforcement. Namespace templates For per-namespace enforcement that applies when a namespace is created inside a tenant cluster, use a Space template with a ResourceQuota in the `objects` field instead. See [Create a template](https://vcluster.com/docs/platform/administer/templates/create-templates.md). ### Enforcement errors[​](#enforcement-errors "Direct link to Enforcement errors") When a tenant or namespace admin creates a PVC that would push usage past the configured limit, the API server rejects the request with an error: ``` Error from server (Forbidden): persistentvolumeclaims "my-pvc" is forbidden: exceeded quota: storage-quota, requested: requests.storage=20Gi, used: requests.storage=90Gi, limited: requests.storage=100Gi ``` The count limit produces a similar error, replacing `requests.storage` with `count/persistentvolumeclaims` in the output. ## Pattern 2: Kyverno policies[​](#pattern-2-kyverno-policies "Direct link to Pattern 2: Kyverno policies") ResourceQuota enforces aggregate limits but cannot cap individual PVC sizes or restrict which storage classes tenants may use. [Kyverno](https://kyverno.io) fills those gaps. Install Kyverno inside the tenant cluster as an [App](https://vcluster.com/docs/platform/administer/templates/create-templates.md) in the tenant cluster template. The examples below use `kyverno.io/v1`. Adjust the API version and validation syntax for the Kyverno version installed in your tenant clusters. ### Restrict individual PVC size[​](#restrict-individual-pvc-size "Direct link to Restrict individual PVC size") The following ClusterPolicy rejects any PVC whose storage request exceeds 50Gi: ``` apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: restrict-pvc-size spec: validationFailureAction: Enforce background: false rules: - name: max-pvc-storage-request match: any: - resources: kinds: - PersistentVolumeClaim validate: message: "PVC storage requests must not exceed 50Gi." deny: conditions: any: - key: "{{ request.object.spec.resources.requests.storage }}" operator: GreaterThan value: "50Gi" ``` ### Restrict allowed storage classes[​](#restrict-allowed-storage-classes "Direct link to Restrict allowed storage classes") The following ClusterPolicy limits PVCs to an approved set of storage classes, preventing tenants from requesting storage from arbitrary provisioners such as Ceph RBD: ``` apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: restrict-storage-class spec: validationFailureAction: Enforce background: false rules: - name: allowed-storage-classes match: any: - resources: kinds: - PersistentVolumeClaim validate: message: "Only 'standard' and 'fast-ssd' storage classes are permitted." pattern: spec: storageClassName: "standard | fast-ssd" ``` ### Embed Kyverno policies in a tenant cluster template[​](#embed-kyverno-policies-in-a-tenant-cluster-template "Direct link to Embed Kyverno policies in a tenant cluster template") Add ClusterPolicy manifests to the `objects` field of a VirtualClusterTemplate to deploy them inside every tenant cluster created from that template. Multiple manifests are separated by `---`. ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterTemplate metadata: name: standard-tenant spec: template: helmRelease: chart: version: "0.25.0" objects: |- apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: restrict-pvc-size spec: validationFailureAction: Enforce background: false rules: - name: max-pvc-storage-request match: any: - resources: kinds: - PersistentVolumeClaim validate: message: "PVC storage requests must not exceed 50Gi." deny: conditions: any: - key: "{{ request.object.spec.resources.requests.storage }}" operator: GreaterThan value: "50Gi" --- apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: restrict-storage-class spec: validationFailureAction: Enforce background: false rules: - name: allowed-storage-classes match: any: - resources: kinds: - PersistentVolumeClaim validate: message: "Only 'standard' and 'fast-ssd' storage classes are permitted." pattern: spec: storageClassName: "standard | fast-ssd" ``` ## Template application order[​](#template-application-order "Direct link to Template application order") The platform applies `objects` before `apps`. ClusterPolicy manifests in the `objects` field are applied before the Kyverno App finishes installing its CRDs and admission webhook, so the apply will fail if Kyverno is not already present. To avoid this, either pre-install Kyverno in the tenant cluster before using this template, or package the ClusterPolicy manifests as a dedicated App listed after the Kyverno App rather than using the `objects` field. Storage quota enforcement flow for tenant PVC creation ## Comparison[​](#comparison "Direct link to Comparison") | Capability | ResourceQuota | Kyverno | | ------------------------------------- | ------------- | -------------------------- | | Aggregate storage limit per namespace | Yes | No | | PVC count limit per namespace | Yes | No | | Storage class-scoped aggregate limit | Yes | No | | Per-PVC maximum size | No | Yes | | Restrict allowed storage classes | No | Yes | | Applied automatically using templates | Yes | Yes (requires Kyverno App) | Use ResourceQuota for aggregate enforcement and Kyverno for per-object rules and storage class governance. The two approaches work together and can be included in the same template. ## Related[​](#related "Direct link to Related") * [Manage Quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md): project-level quota configuration including aggregate `requests.storage` and `count/persistentvolumeclaims` limits * [Create a template](https://vcluster.com/docs/platform/administer/templates/create-templates.md): add objects and Apps to a tenant cluster template --- # Expose the platform with Istio This guide explains how to expose the platform using the Istio ingress gateway. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before exposing the platform with Istio, ensure you have: * A running Kubernetes cluster where the platform is deployed * Administrative access to this cluster (`kubectl` configured with appropriate permissions) * The `istioctl` command-line tool installed on your local machine * A domain name that you can configure for accessing the platform (to replace `org.vclusterplatform.com` in examples) * Optional: TLS certificates for secure HTTPS access, stored in a Kubernetes secret * The platform service running in the `vcluster-platform` namespace ## Install Istio[​](#install-istio "Direct link to Install Istio") Install Istio on your platform control plane cluster: Install Istio on your cluster ``` istioctl install ``` ## Configure the Istio Gateway[​](#configure-the-istio-gateway "Direct link to Configure the Istio Gateway") Create a `Gateway` resource to handle incoming traffic: Create Gateway resource ``` apiVersion: networking.istio.io/v1 kind: Gateway metadata: name: platform-gateway namespace: vcluster-platform spec: selector: istio: ingressgateway servers: - port: number: 80 name: http protocol: HTTP hosts: - "org.vclusterplatform.com" # Replace with your domain - port: name: https number: 443 protocol: HTTPS hosts: - "org.vclusterplatform.com" # Replace with your domain tls: credentialName: tls-secret # Replace with your TLS certificate secret mode: SIMPLE ``` ## Configure Virtual Service[​](#configure-virtual-service "Direct link to Configure Virtual Service") Create a `VirtualService` to route traffic to your platform service: Create VirtualService resource ``` apiVersion: networking.istio.io/v1 kind: VirtualService metadata: name: vcluster-platform namespace: vcluster-platform spec: hosts: - "org.vclusterplatform.com" # Replace with your domain gateways: - platform-gateway http: - route: - destination: port: number: 80 host: loft.vcluster-platform.svc.cluster.local # Replace with your platform service ``` For more information about setting up TLS certificates, see the [Istio Secure Gateways documentation](https://istio.io/latest/docs/tasks/traffic-management/ingress/secure-ingress/). ## Enable Tailscale connectivity[​](#enable-tailscale-connectivity "Direct link to Enable Tailscale connectivity") For external tenant clusters or platform agents on connected clusters to communicate with the platform, apply this `EnvoyFilter`: Apply EnvoyFilter for Tailscale connectivity ``` apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: vcluster-platform-tailscale namespace: istio-system spec: configPatches: - applyTo: NETWORK_FILTER match: listener: filterChain: filter: name: "envoy.filters.network.http_connection_manager" patch: operation: MERGE value: typed_config: "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager" "upgradeConfigs": - { "upgradeType": "tailscale-control-protocol" } - { "upgradeType": "derp" } ``` This configuration enables the Tailscale protocols required for secure cross-cluster communication. --- # End-to-end GitOps with ArgoCD This guide walks through the full lifecycle of managing platform resources with ArgoCD, covering platform installation, cluster connections, project creation, and tenant cluster deployment. Each section references the detailed documentation for that step and provides the ArgoCD-specific configuration needed. info While this guide uses ArgoCD as the example, the same principles apply to other GitOps tools like Flux. Each section links to detailed documentation that includes tabs for alternative tooling. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you begin, ensure you have: * A Kubernetes cluster with administrative access * [ArgoCD installed](https://argo-cd.readthedocs.io/en/stable/getting_started/) and configured on the cluster * `helm` v3.10 or later * `kubectl` configured with admin access to the target cluster ## 1. Install the platform[​](#1-install-the-platform "Direct link to 1. Install the platform") Deploy the platform as an ArgoCD Application. This uses the platform Helm chart with your desired configuration values. Create an ArgoCD Application that points to the platform Helm chart: Modify the following with your specific values to generate a copyable command: PLATFORM\_VERSION4.6.0 ADMIN\_PASSWORDyour-secure-password PLATFORM\_HOSTvcluster-platform.example.com platform-application.yaml ``` apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: vcluster-platform namespace: argocd spec: destination: namespace: vcluster-platform server: "https://kubernetes.default.svc" source: repoURL: "https://charts.loft.sh" targetRevision: 4.6.0 chart: vcluster-platform helm: parameters: - name: admin.create value: "true" - name: admin.username value: admin - name: admin.password value: "your-secure-password" - name: ingress.enabled value: "true" - name: ingress.host value: "vcluster-platform.example.com" - name: config.loftHost value: "https://vcluster-platform.example.com" project: default syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ``` Apply this to your cluster or commit it to a Git repository that ArgoCD monitors. ``` kubectl apply -f platform-application.yaml ``` ArgoCD and Helm releases ArgoCD does not deploy the Helm release secret. Platform configuration updates must be managed through your Git repository, not the platform UI. For the full set of Helm configuration options and alternative installation methods, see the [platform installation with ArgoCD guide](https://vcluster.com/docs/platform/install/gitops.md). ## 2. Connect a control plane cluster[​](#2-connect-a-control-plane-cluster "Direct link to 2. Connect a control plane cluster") After the platform is running, connect control plane clusters where tenant clusters will be deployed. In a GitOps workflow, you can manage connected clusters as Kubernetes resources alongside your platform installation. Each connected cluster requires two resources: a Cluster object and a Secret containing the cluster credentials. Modify the following with your specific values to generate a copyable command: CLUSTER\_NAMEmy-host-cluster cluster.yaml ``` apiVersion: management.loft.sh/v1 kind: Cluster metadata: name: my-host-cluster annotations: argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec: displayName: "my-host-cluster" owner: user: admin config: secretName: "loft-cluster-config-my-host-cluster" secretNamespace: "vcluster-platform" access: - subresources: - "*" users: - admin verbs: - "*" ``` Modify the following with your specific values to generate a copyable command: CLUSTER\_NAMEmy-host-cluster CLUSTER\_API\_SERVERhttps\://1.2.3.4:6443 CA\_DATAYOUR-CA-DATA CLUSTER\_TOKENYOUR-TOKEN cluster-secret.yaml ``` apiVersion: v1 kind: Secret metadata: name: "loft-cluster-config-my-host-cluster" namespace: vcluster-platform type: Opaque stringData: config: | apiVersion: v1 kind: Config clusters: - name: my-host-cluster cluster: server: https://1.2.3.4:6443 certificate-authority-data: YOUR-CA-DATA contexts: - name: my-host-cluster context: cluster: my-host-cluster user: my-host-cluster-user current-context: my-host-cluster users: - name: my-host-cluster-user user: token: YOUR-TOKEN ``` Cluster secrets The Secret contains authentication data for the remote cluster. Handle this appropriately in your GitOps pipeline. Consider using sealed secrets or an external secrets operator. The `SkipDryRunOnMissingResource=true` annotation is required because the Cluster CRD is installed by the platform itself. Without it, ArgoCD cannot sync the resource on initial deployment. You also need to install the platform agent on each connected cluster. If you manage agents through GitOps, set the `DISABLE_AGENT=true` environment variable in your platform deployment and create a separate ArgoCD Application for the agent: Modify the following with your specific values to generate a copyable command: CLUSTER\_NAMEmy-host-cluster CLUSTER\_API\_SERVERhttps\://1.2.3.4:6443 PLATFORM\_VERSION4.6.0 agent-application.yaml ``` apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: platform-agent-my-host-cluster namespace: argocd spec: destination: namespace: vcluster-platform server: "https://1.2.3.4:6443" source: repoURL: "https://charts.loft.sh" targetRevision: 4.6.0 chart: vcluster-platform helm: parameters: - name: agentOnly value: "true" project: default syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ``` To also register the control plane cluster with Argo CD so it can receive shared infrastructure applications directly, add `spec.argoCD` to the Cluster object after creating a connector (see [Step 3](#3-create-a-project)): ``` spec: argoCD: enabled: true connector: argocd-main # name of your connector Secret ``` For interactive setup methods using the UI or CLI, see the [connect a cluster guide](https://vcluster.com/docs/platform/administer/clusters/connect-cluster.md). ## 3. Create a project[​](#3-create-a-project "Direct link to 3. Create a project") Projects organize tenant clusters and control access, templates, and quotas. In a GitOps workflow, define projects as Project custom resources. Modify the following with your specific values to generate a copyable command: PROJECT\_NAMEteam-alpha TEAM\_NAMEdev-team project.yaml ``` apiVersion: management.loft.sh/v1 kind: Project metadata: name: team-alpha annotations: argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec: displayName: team-alpha owner: user: admin allowedClusters: - name: "*" allowedTemplates: - kind: VirtualClusterTemplate group: storage.loft.sh name: "*" members: - kind: Team group: storage.loft.sh name: dev-team clusterRole: loft-management-project-admin ``` ### Connect to Argo CD[​](#connect-to-argo-cd "Direct link to Connect to Argo CD") To register tenant clusters with Argo CD, create a connector. A connector is a Kubernetes Secret in the Platform namespace that holds your Argo CD server credentials and endpoint. Once a connector exists, you reference it by name from each cluster you want registered. See [Connect to Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md) for the full setup, including required RBAC permissions and connector YAML. If you are using the project-level Argo CD integration from an earlier version of vCluster Platform, see [Argo CD (legacy)](https://vcluster.com/docs/platform/integrations/argocd/legacy.md). management.loft.sh vs storage.loft.sh Define resources using the `management.loft.sh` API group. The `storage.loft.sh` group is used internally by platform controllers and appears in some spec fields as references to existing resources. For interactive project creation using the UI, see the [create a project guide](https://vcluster.com/docs/platform/administer/projects/create.md). ## 4. Deploy tenant clusters[​](#4-deploy-tenant-clusters "Direct link to 4. Deploy tenant clusters") With the platform, cluster, and project in place, deploy tenant clusters through ArgoCD. There are two approaches depending on how you want to manage the tenant cluster lifecycle. ### Option A: Deploy with VirtualClusterInstance[​](#option-a-deploy-with-virtualclusterinstance "Direct link to Option A: Deploy with VirtualClusterInstance") Create a VirtualClusterInstance resource that the platform manages. This gives you full platform features like sleep mode, auto-delete, and template enforcement. Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEdev-vcluster PROJECT\_NAMEteam-alpha CLUSTER\_NAMEmy-host-cluster VCLUSTER\_VERSION0.31.0 virtualclusterinstance.yaml ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterInstance metadata: name: dev-vcluster namespace: p-team-alpha annotations: argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec: clusterRef: cluster: my-host-cluster external: false owner: user: admin template: helmRelease: chart: version: 0.31.0 values: | controlPlane: backingStore: etcd: deploy: enabled: true integrations: argoCD: enabled: true connector: argocd-main ``` tip The namespace follows the pattern `p-`. The platform uses this convention to associate tenant clusters with projects. When the VirtualClusterInstance reconciles, Platform registers the tenant cluster with Argo CD using the connector credentials. You can then declare Argo CD Applications in the tenant cluster configuration. See [Deploy applications](https://vcluster.com/docs/platform/integrations/argocd/deploy-applications.md). ### Option B: Deploy with Helm chart directly[​](#option-b-deploy-with-helm-chart-directly "Direct link to Option B: Deploy with Helm chart directly") Deploy the vCluster Helm chart through an ArgoCD Application. This approach gives you direct control over the Helm release but does not provide platform lifecycle features unless you [register the tenant cluster afterward](https://vcluster.com/docs/platform/use-platform/virtual-clusters/add-virtual-clusters.md). Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEdev-vcluster VCLUSTER\_VERSION0.31.0 NAMESPACEteam-alpha vcluster-application.yaml ``` apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: dev-vcluster namespace: argocd spec: project: default source: chart: vcluster repoURL: https://charts.loft.sh targetRevision: 0.31.0 helm: releaseName: dev-vcluster valueFiles: - vcluster.yaml destination: server: https://kubernetes.default.svc namespace: team-alpha ``` For the full list of deployment options including CLI, Terraform, and Flux, see the [deploy vCluster guide](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/basics.md). ## Manage stacks declaratively[​](#manage-stacks-declaratively "Direct link to Manage stacks declaratively") A [Stack](https://vcluster.com/docs/platform/understand/what-are-stacks.md) is defined by two resources, `StackTemplate` and `StackInstance`. You can store both with the other Platform resources in your Git repository. A StackTemplate is cluster-scoped. Create each StackInstance in its project's `p-` namespace. This example defines a reusable Stack with two App tasks, then deploys it to a tenant cluster. Argo CD sync waves create the referenced template before the instance. The Stack controller, not Argo CD, enforces the task dependency and health gate. stack-template.yaml ``` apiVersion: management.loft.sh/v1 kind: StackTemplate metadata: name: example-platform annotations: argocd.argoproj.io/sync-wave: "0" spec: displayName: Example platform parameters: - variable: namespace label: Namespace defaultValue: example tasks: - name: database app: templateRef: name: postgresql parameters: namespace: "{{ .Values.namespace }}" - name: api dependsOn: - database app: templateRef: name: example-api parameters: namespace: "{{ .Values.namespace }}" --- apiVersion: management.loft.sh/v1 kind: StackInstance metadata: name: example-platform namespace: p-team-alpha annotations: argocd.argoproj.io/sync-wave: "1" spec: destination: virtualCluster: name: dev-vcluster templateRef: name: example-platform parameters: namespace: example owner: user: admin ``` Set `spec.owner` on the StackInstance. App tasks deploy as their owner, and a GitOps controller applies the manifest with its own service account rather than a Platform identity. The owner is immutable once set, so a later commit can't change it. Declare the referenced App or Argo CD application templates in an earlier sync wave or manage them separately. A referenced StackTemplate is resolved on every StackInstance reconciliation, so a committed template change can update existing instances. Review `prunePolicy` before removing tasks. Its default value, `Retain`, reports removed child applications as orphans instead of deleting them. For the complete task and output schema, see [Create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md). For Stack resource permissions, see [Stack permissions](https://vcluster.com/docs/platform/administer/users-permissions/permissions/stacks.md). ## Manage GitOps drift[​](#manage-gitops-drift "Direct link to Manage GitOps drift") Platform controllers may modify certain fields on resources after creation, causing drift in ArgoCD. Configure `ignoreDifferences` in your ArgoCD Applications to handle this: ``` spec: ignoreDifferences: # Ignore status for all Platform custom resources - group: management.loft.sh kind: "*" jsonPointers: - /status # Ignore controller-managed fields - group: management.loft.sh kind: "*" managedFieldsManagers: - loft syncPolicy: syncOptions: - RespectIgnoreDifferences=true ``` For a complete breakdown of drift-causing fields by resource type and tool-specific configuration, see the [GitOps drift management section](https://vcluster.com/docs/platform/install/gitops.md#manage-gitops-drift). ## Repository structure[​](#repository-structure "Direct link to Repository structure") A typical Git repository for this workflow looks like: ``` platform-gitops/ applications/ platform.yaml # ArgoCD Application for vCluster Platform agent-cluster-1.yaml # ArgoCD Application for agent on cluster-1 resources/ clusters/ cluster-1.yaml # Cluster + Secret resources projects/ team-alpha.yaml # Project resource virtual-clusters/ dev-vcluster.yaml # VirtualClusterInstance stacks/ example-platform.yaml # StackTemplate + StackInstance ``` ## Next steps[​](#next-steps "Direct link to Next steps") * [Configure platform settings](https://vcluster.com/docs/platform/configure/introduction.md) for authentication, audit logging, and more * [Set up templates](https://vcluster.com/docs/platform/administer/templates/create-templates.md) to standardize tenant cluster configurations * [Configure project quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md) to control resource usage * [Enable sleep mode](https://vcluster.com/docs/vcluster/manage/sleep-wakeup.md) to reduce costs on idle tenant clusters --- # Build a product control plane on the Platform API When you build a product that provisions tenant clusters for your customers, your product's control plane is what customers interact with, not Platform directly. Your automation drives Platform through its API. This guide maps common product concepts to Platform resources, shows how to call and watch the API, and covers how to keep usage metering consistent across tenants. This pattern applies to any product built on Platform. For a managed Kubernetes service, see the [AI Cloud production path](https://vcluster.com/docs/vcluster/production-guide/ai-cloud.md). For a managed inference platform, see [Building an inference platform](https://vcluster.com/docs/vcluster/introduction/inference-platform.md). ## Map product concepts to Platform resources[​](#map-product-concepts-to-platform-resources "Direct link to Map product concepts to Platform resources") Decide which actions your product exposes, such as instance creation, tier selection, quota changes, updates, and deletion. Map those customer-facing concepts to the Platform and tenant resources that implement them: | Product concept | Platform or tenant resource | | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | Customer account, organization, or workspace | Platform project, quotas, allowed templates, and allowed node types | | Service tier or plan, such as a dedicated or shared tier | Template parameters, node type constraints, and quota policy | | Provisioned instance, such as a cluster or endpoint | Tenant cluster, namespace, `InferenceService` or direct workload, Service, route, autoscaler, secrets, and metrics resources | | Instance URL and API key | Generated or directly managed Gateway API or Ingress route, DNS entry, certificate, and provider authentication layer | | Usage and billing record | Node-hour, GPU-hour, storage, egress, request, or token usage events mapped to customer, instance, and tier | | Lifecycle event | Platform API call, template update, tenant workload rollout, drain, scale, or delete operation | Keep this mapping in your product control plane. Customers should see instance status, URL, usage, errors, and the current provisioning stage, such as waiting for capacity or rolling out workloads. Your operators should see the backing project, tenant cluster, node claim, template version, route, and rollout state. Your product automation calls the [Platform API](https://vcluster.com/docs/platform/api/use-api.md) with an internal [access key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) to create or select the tenant cluster and template for an instance. It then watches cluster, node, route, and workload rollout status. Build this as an internal provisioning path first, then decide which operations become customer-facing. ## Keep billing labels stable[​](#keep-billing-labels-stable "Direct link to Keep billing labels stable") Keep billing labels, such as customer, instance, and tier, stable across tenant clusters, routes, workloads, and metrics. That way usage maps back to the product object the customer sees. Node-hours or GPU-hours per SKU work for dedicated node tiers. Shared pools usually need finer-grained signals such as requests, generated tokens, or pod-level GPU allocation. [vBilling](https://www.vcluster.com/vbilling) is an experimental vCluster Labs project that meters tenant clusters. It tracks GPU-hours per SKU, GPU utilization, storage, network egress, and node hours. It streams those as usage events to an adapter such as Lago, Stripe Meters, or Metronome. See [this deep dive](https://www.vcluster.com/blog/vbilling-an-open-source-billing-pipe-for-ai-clouds-running-v) for the full metric list and adapter configuration. Pricing, invoicing, and customer billing logic belong in your billing system, not vBilling. ## Confirm license tiers[​](#confirm-license-tiers "Direct link to Confirm license tiers") Confirm which parts of the stack require Platform activation or a paid tier before you build on top of them. Sidebar badges show feature tier requirements. See [Open Source vs Free tier](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md) and [vCluster pricing](https://www.vcluster.com/pricing). ## Next steps[​](#next-steps "Direct link to Next steps") * [Use API](https://vcluster.com/docs/platform/api/use-api.md) — connect to the Platform management API * [Owned Access Key](https://vcluster.com/docs/platform/api/resources/ownedaccesskey.md) — create the internal key your automation uses * [End-to-end GitOps with ArgoCD](https://vcluster.com/docs/platform/how-to/gitops-end-to-end.md) — deliver platform and tenant resources declaratively * [Building an inference platform](https://vcluster.com/docs/vcluster/introduction/inference-platform.md) — the inference-specific version of this pattern --- # Use Go Template Functions ### Overview[​](#overview "Direct link to Overview") A Go template function is a built-in or user-defined helper function. It can be used within Go templates to transform data when rendering text. For example, in a Helm chart (which uses Go templates), functions such as `default`, `upper`, and `quote` are commonly used to process values dynamically. Templates in vCluster Platform standardize configurations for tenant clusters, namespaces, applications, and [Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md). You can create templates through the vCluster Platform UI or by applying manifests directly to your cluster using vCluster CLI or Helm charts. The platform provides template resources for each workload type, including VirtualClusterTemplate and StackTemplate. For more information, see [Creating Templates](https://vcluster.com/docs/platform/administer/templates/create-templates.md). When defining a VirtualClusterTemplate resource, use Go template functions to enable conditional logic and dynamic configuration within your template values and customize deployments based on context or user input. ### Use Go template operators[​](#use-go-template-operators "Direct link to Use Go template operators") Go template operators let you inject variables, conditions, and loops into text files to generate YAML or HTML dynamically. They don't use `==` syntax but use prefix operators like `and`, `eq`, and `gt` as follows: Use eq operator ``` {{- if and (eq .Values.env "prod") (gt .Values.replicas 2) }} replicas: {{ .Values.replicas }} {{- else }} replicas: 1 {{- end }} ``` You can find more information [here](https://developer.hashicorp.com/nomad/tutorials/archive/go-template-syntax#function-list). Below is an example of using `eq` operator: Use eq operator to enable or disable ingresses synchronization ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterTemplate metadata: name: vcluster-pro-template spec: displayName: Virtual Cluster Pro Template template: helmRelease: values: sync: toHost: ingresses: {{ if eq .Values.loft.project "auth-core" }} enabled: true {{ else }} enabled: false {{ end }} ``` ### Use template variables[​](#use-template-variables "Direct link to Use template variables") The vCluster Platform provides [platform-specific parameter values](https://vcluster.com/docs/platform/administer/templates/parameters.md#platform-specific-parameter-values) and you can use these parameter values to render the template file dynamically: Use template variables to fill the configuration ``` metadata: name: "{{ .Values.loft.virtualClusterName }}" spec: ingress: host: "{{ .Values.loft.virtualClusterName }}.example.com" parameters: cpu: "{{ .Values.cpuLimit }}" ``` For example, You can use [connected control plane cluster parameter values](https://vcluster.com/docs/platform/administer/templates/parameters.md#connected-control-plane-cluster-parameter-values) to specify the sleep mode timezone automatically: ``` sleepMode: enabled: true autoSleep: afterInactivity: "35m" timeZone: '{{ .Values.loft.clusterAnnotations.sleepTimeZone }}' ``` ## Template values and outputs in Stacks[​](#template-values-and-outputs-in-stacks "Direct link to Template values and outputs in Stacks") A Stack task can use StackInstance parameters through `{{ .Values. }}`. It can also consume a captured value from an earlier task through `{{ .Outputs.. }}`. The consuming task must list the producing task in `dependsOn`. ``` tasks: - name: database app: templateRef: name: postgresql outputs: - name: address fromResource: apiVersion: v1 kind: Service namespace: database name: postgresql jsonPath: "{.spec.clusterIP}" - name: api dependsOn: - database app: templateRef: name: example-api parameters: environment: "{{ .Values.environment }}" databaseAddress: "{{ .Outputs.database.address }}" ``` Stack parameters and outputs are rendering contexts, not general Go template functions. See [What are Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md) for the task orchestration concept. For their validation rules and supported locations, see [Create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md#pass-outputs-between-tasks). --- # With Offline License Key | | | Enterprise | | | | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ------ | ------ | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Air-Gapped Mode](https://vcluster.com/docs/platform/install/air-gapped.md) | | Add-On | Add-On | Add-On | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). This guide walks you through installing vCluster Platform and deploying tenant clusters in environments without internet access, referred to as air-gapped environments. It covers setting up a private OCI-compliant registry, populating it with container images and Helm charts, configuring vCluster Platform and agents to pull from the registry, and deploying tenant clusters securely and offline. If you want to run a dedicated offline license service in-cluster, see [With Offline License Server](https://vcluster.com/docs/platform/install/air-gapped/with-offline-license-server.md). For how the platform decides between these modes, and how it caches and refreshes what it retrieves, see [How licensing works](https://vcluster.com/docs/platform/understand/licensing.md). ## Overview[​](#overview "Direct link to Overview") There are several artifacts that are typically accessed using an internet connection. To complete an air-gapped install, these artifacts need to be accessible by the Kubernetes cluster through a private registry. * **vCluster Platform Helm chart** - This Helm chart is typically accessed through the vCluster Labs charts repository and is used for installing vCluster Platform and connecting hosts (for example installing agents). * **vCluster Helm chart** - This Helm chart is typically accessed through the vCluster Labs charts repository and is used for deploying tenant clusters. * **Images used in the Helm charts** - These images are typically accessed through different container registries. After populating the registry with the artifacts, install vCluster Platform, connect control plane clusters and then deploy tenant clusters. Each step requires additional configuration to use your private registry. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Ensure you have the following: * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Your current kube-context must have administrative privileges, which you can verify with `kubectl auth can-i create clusterrole -A` info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm` installed: Helm v3.10.0+ is required for deploying the platform. Refer to the [Helm Installation Guide](https://helm.sh/docs/intro/install/) if you need to install it. * `kubectl` installed: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. Registry prerequisites * **OCI-compliant private registry with a `/charts` folder** - A private registry accessible to both the Kubernetes cluster for the platform and connected control plane clusters and a separate, internet-connected machine. * **Ability to push to your OCI-compliant private registry** - Write access for uploading images and charts. Licensing prerequisite * **A license key for vCluster Platform** - A vCluster Labs license key is required to install vCluster Platform. info Contact to purchase a license key or request a trial license key for offline use. Internet access prerequisites * **Access to the internet** - Required on the machine used for downloading and preparing artifacts. * **Logged in to GitHub Container Registry** - Authentication for pulling images from public registries. Kubernetes access prerequisite * **Administrator access to a Kubernetes cluster** - Current kube-context must have administrative privileges, verifiable with `kubectl auth can-i create clusterrole -A` Tooling prerequisites * **`wget` installed** - For downloading release assets and scripts * **`docker` installed** - For pulling, packaging, and pushing container images Network and resource prerequisites * **Control plane cluster access to platform domain** - Connected control plane clusters must reach the domain where vCluster Platform is hosted. * **vCluster Platform Pod Resource Requirements** * Requests * memory: `256Mi` * cpu: `200m` * Limits * memory: `4Gi` * cpu: `2` - Ensure that the required network ports are open. info The platform requires open ports for API extensions, webhooks, and cost monitoring. In the infrastructure of private Kubernetes clusters, ensure firewall rules allow traffic from the Kubernetes master to platform components. * `8443`—API service extension for cluster communication (`v1.cluster.loft.sh`) * `9443`—Webhook validation and enforcement (`loft webhook`) * `9444`—Management API for platform administration (`v1.management.loft.sh`) * `9090`—Prometheus metrics proxy for cost monitoring For more information, see [Networking](https://vcluster.com/docs/platform/administer/clusters/advanced/networking.md) to learn about port and firewall rules. ## Populate images to private registry[​](#populate-images-to-private-registry "Direct link to Populate images to private registry") Each [vCluster Platform GitHub release](https://github.com/loft-sh/loft/releases) and [vCluster GitHub release](https://github.com/loft-sh/vcluster/releases) provides multiple release assets to help you upload images to a private registry. These assets include text files listing all necessary images and scripts to automate downloading (pulling) and pushing them. ### Image text files[​](#image-text-files "Direct link to Image text files") Each [vCluster Platform release](https://github.com/loft-sh/loft/releases) includes a file that lists all container images required for installing vCluster Platform: * `images.txt` — Required images for installing vCluster Platform. In vCluster releases v0.25.0 and later, two files list the container images needed to deploy a vCluster: * `images.txt` — Required images for deploying vCluster with the default Kubernetes version. * `images-optional.txt` - An optional set of images needed for additional features, including different Kubernetes distributions and versions. In vCluster releases **earlier than v0.25.0**, a single file is provided: * `vcluster-images.txt` — Required images for deploying vCluster with the default Kubernetes version. ### Scripts to pull and push images[​](#scripts-to-pull-and-push-images "Direct link to Scripts to pull and push images") Both product releases contain one script which helps download and package the list of images and another script that pushes images to the private registry. The script from either release can be used to populate the registry. The difference between the scripts in the different releases is the default name of the tarball that is created. These instructions override the defaults to ensure the correct image text file and tarball is created. * `download-images.sh` - A bash script that quickly iterates over all the images files to pull them and package them into a tarball to a machine that has internet access. * `push-images.sh` - A bash script that takes the tarball generated from the download script to push them to your private registry. tip Both releases include `download-images.sh` and `push-images.sh`, but you only need to download the scripts once. ### Pull and push images for vCluster Platform[​](#pull-and-push-images-for-vcluster-platform "Direct link to Pull and push images for vCluster Platform") Download the required platform images from public registries and upload them to your private registry for air-gapped installation. #### Populate the registry with images[​](#populate-the-registry-with-images "Direct link to Populate the registry with images") Follow these steps to populate the registry with the required platform images. 1. Set environment variables for the version of vCluster Platform that you want to deploy and the private registry. Export environment variables ``` export PLATFORM_VERSION=4.12.0 # Replace with desired version export REGISTRY=ecr.io/myteam # This should be a prefix; do not include any image paths ``` 2. Download the assets from the [vCluster Platform GitHub release](https://github.com/loft-sh/loft/releases) and make the scripts executable. warning The `images.txt` contains a `vcluster-pro` image for the default vCluster version, but doesn't contain all the images required to deploy vCluster. An additional step is required to retrieve all vCluster images. Download assets and prepare scripts ``` wget https://github.com/loft-sh/loft/releases/download/v${PLATFORM_VERSION}/images.txt wget https://github.com/loft-sh/loft/releases/download/v${PLATFORM_VERSION}/download-images.sh wget https://github.com/loft-sh/loft/releases/download/v${PLATFORM_VERSION}/push-images.sh chmod +x ./download-images.sh chmod +x ./push-images.sh ``` 3. Run `download-images.sh` to pull all images and create a tarball of the images. Review the output to confirm all images were pulled successfully and packaged in the tarball. Download and package images ``` ./download-images.sh --image-list images.txt --images loft-images.tar.gz ``` 4. Run `push-images.sh` to upload all required images to your private registry. When pushing images to your private registry, references to the original (public) image registry are removed. Only the image name and repository are preserved. This enables vCluster to pull all images from your configured private registry. Push images to private registry ``` ./push-images.sh --registry ${REGISTRY} --images loft-images.tar.gz --image-list images.txt ``` ### Pull and push images for vCluster[​](#pull-and-push-images-for-vcluster "Direct link to Pull and push images for vCluster") Populate the registry with the specific images needed for deploying vCluster: 1. Set environment variables for the version of vCluster that you want to deploy and the private registry. tip You can review the `tag` of the `vcluster-pro` image in `images.txt` to pick the default vCluster version. Export environment variables ``` export VCLUSTER_VERSION=0.37.4 # Replace with desired version # Setting the REGISTRY environment is redundant from the steps in populating vCluster Platform images # export REGISTRY=ecr.io/myteam # This should be a prefix; do not include any image paths ``` 2. Download the text files from the [vCluster GitHub release](https://github.com/loft-sh/vcluster/releases). The `images.txt` or `vcluster-images.txt` files contain multiple Kubernetes versions and distributions. You can edit the file and remove the images for any unwanted versions and distributions. Download image text file for vCluster v0.25.0+ versions ``` wget https://github.com/loft-sh/vcluster/releases/download/v${VCLUSTER_VERSION}/images.txt ``` Download image text file for vCluster < v0.25.0 versions ``` wget https://github.com/loft-sh/vcluster/releases/download/v${VCLUSTER_VERSION}/vClusterimages.txt ``` 3. **Optional**: Download the scripts from the [vCluster GitHub release](https://github.com/loft-sh/vcluster/releases) and make them executable. note If you've already completed this as part of populating the platform images, you can skip this step. Download prepare scripts ``` wget https://github.com/loft-sh/vcluster/releases/download/v${VCLUSTER_VERSION}/download-images.sh wget https://github.com/loft-sh/vcluster/releases/download/v${VCLUSTER_VERSION}/push-images.sh chmod +x ./download-images.sh chmod +x ./push-images.sh ``` 4. Run `download-images.sh` to pull all images and create a tarball of the images. Review the output to confirm all images were pulled successfully and packaged in the tarball. Download and package images for vCluster v0.25.0+ versions ``` ./download-images.sh --image-list images.txt --images vcluster-images.tar.gz ``` Download and package images for vCluster < v0.25.0 versions ``` ./download-images.sh --image-list vcluster-images.txt --images vcluster-images.tar.gz ``` Multi-platform support If you are using the scripts from vCluster v0.31.0 or later and need to support multiple CPU architectures (e.g., AMD64 and ARM64), use the `--platforms` flag to specify which platforms to download. This is useful when your cluster has nodes with different architectures. Download images for multiple platforms ``` ./download-images.sh --platforms "linux/amd64,linux/arm64" --image-list vcluster-images.txt --images vcluster-images.tar.gz ``` 5. Run `push-images.sh` to upload all required images to your private registry. Images are pushed to your private registry without public registry references, preserving only the repository and image name. You can configure vCluster to use your private registry for all deployments. Push images to private registry ``` ./push-images.sh --registry ${REGISTRY} --images vcluster-images.tar.gz ``` Additional push options These options are available when using scripts from vCluster v0.31.0 or later. **Multi-platform support**: If you downloaded images for multiple platforms, specify the same `--platforms` flag when pushing: ``` ./push-images.sh --registry ${REGISTRY} --images vcluster-images.tar.gz --platforms "linux/amd64,linux/arm64" ``` **Insecure registry**: If your private registry uses HTTP instead of HTTPS, add the `--insecure` flag: ``` ./push-images.sh --registry ${REGISTRY} --images vcluster-images.tar.gz --insecure ``` 6. **Optional**: If you want to deploy vCluster v0.25.0 or later with a different Kubernetes version, download the `images-optional.txt` file from the [vCluster GitHub release](https://github.com/loft-sh/vcluster/releases). Additional container images for the version must be downloaded and uploaded to your private registry. The `images-optional.txt` contains multiple Kubernetes distributions and versions. You can edit the file and remove the images for the unwanted distributions and versions. Pull, package, and push all optional images ``` wget https://github.com/loft-sh/vcluster/releases/download/v${VCLUSTER_VERSION}/images-optional.txt ./download-images.sh --image-list images-optional.txt --images vcluster-images-optional.tar.gz ./push-images.sh --registry ${REGISTRY} --images vcluster-images-optional.tar.gz ``` tip You can also use the `--platforms` and `--insecure` flags with these commands if using scripts from vCluster v0.31.0 or later. ## Populate the Helm charts to a private registry[​](#populate-the-helm-charts-to-a-private-registry "Direct link to Populate the Helm charts to a private registry") You can push the following Helm charts to your OCI-compliant private registry: * **vCluster Platform Helm chart**: The core chart for deploying the vCluster platform. * **Version-specific Helm charts**: Charts required for each vCluster version you intend to deploy. * **Application Helm charts**: Charts for applications you plan to deploy with vCluster. For more details, see the [Apps documentation](https://vcluster.com/docs/platform/understand/what-are-apps.md) for details. * **Experimental configuration charts**: Charts needed when using `experimental.deploy.vcluster.helm` [configuration settings](https://vcluster.com/docs/vcluster/configure/vcluster-yaml#experimental-deploy-vcluster). warning This guide only includes detailed steps for pulling and pushing the vCluster Platform image and Helm chart. ### Pull and push the vCluster Platform Helm chart[​](#pull-and-push-the-vcluster-platform-helm-chart "Direct link to Pull and push the vCluster Platform Helm chart") 1. note If you haven’t already set the environment variables, set them now before continuing. The private registry requires a `/charts` folder, where you must push all Helm charts. Export environment variables ``` export PLATFORM_VERSION=4.12.0 # Replace with desired version export REGISTRY=ecr.io/myteam # A charts folder is expected ``` 2. Pull the platform Helm chart and push it to your private registry with the OCI protocol: Pull and push the Helm chart to your private registry ``` helm pull vcluster-platform --repo https://charts.loft.sh --version ${PLATFORM_VERSION} helm push vcluster-platform-${PLATFORM_VERSION}.tgz oci://${REGISTRY}/charts ``` ### Pull and push the vCluster Helm chart[​](#pull-and-push-the-vcluster-helm-chart "Direct link to Pull and push the vCluster Helm chart") 1. note If you haven’t already set the environment variables, set them before continuing. The private registry assumes having a `/charts` folder, which is where to push all the Helm charts. Export environment variables ``` export VCLUSTER_VERSION=0.37.4 # Replace with desired version export REGISTRY=ecr.io/myteam # A charts folder is expected ``` 2. Pull the vCluster Helm chart and push it into your private registry with the OCI protocol. Pull and push the Helm chart to your private registry ``` helm pull vcluster --repo https://charts.loft.sh --version ${VCLUSTER_VERSION} helm push vcluster-${VCLUSTER_VERSION}.tgz oci://${REGISTRY}/charts ``` ## Configure and install vCluster Platform and agent[​](#configure-install-platform-agent "Direct link to Configure and install vCluster Platform and agent") To install the platform and connect hosts, you must configure a `vcluster-platform.yaml` file. This YAML file is used as the `values.yaml` for the vCluster Platform Helm chart. ### vCluster Platform and agent configuration[​](#vcluster-platform-and-agent-configuration "Direct link to vCluster Platform and agent configuration") The `vcluster-platform.yaml` file contains all configuration settings for your vCluster Platform and connected hosts (agents) installation. Reference the offline license key Reference the license key so that the platform can start with all your enterprise features enabled. Offline license configuration ``` env: LICENSE_KEY: "YOUR_LICENSE_KEY" ``` Alternatively, a Kubernetes `Secret` may be used in the following manner: Create license secret ``` kubectl create namespace vcluster-platform kubectl create secret generic license-key-secret -n vcluster-platform --from-file=licenseKey="./license.txt" ``` Offline license configuration with secret ``` envValueFrom: LICENSE_KEY: secretKeyRef: name: license-key-secret key: licenseKey ``` Use a private registry without credentials * Set the **imageRef** to be used by the vCluster Platform Helm chart to the image in the private registry. * Set the **default registry** for pulling images used in installing vCluster Platform or connecting hosts (i.e. deploying agents). * Set the **default chart registry** in the `vcluster-platform.yaml` as the default Helm repository when creating tenant clusters. This should prefix with `oci://` to ensure it's being deployed using OCI protocol as well as the `/charts` folder. Setting the image for vCluster Platform and the default private registry ``` imageRef: registry: ecr.io # Replace with your private registry repository: myteam/loft-sh/vcluster-platform # Replace with your repository path tag: # Replace with your version env: DEFAULT_IMAGE_REGISTRY: ecr.io/myteam DEFAULT_VCLUSTER_CHART_REPO: oci://ecr.io/myteam/charts # The agent values are needed to keep the agent up to date with the same settings agentValues: env: DEFAULT_IMAGE_REGISTRY: ecr.io/myteam DEFAULT_VCLUSTER_CHART_REPO: oci://ecr.io/myteam/charts ``` Use an authenticated private registry * Set the **imageRef** to be used by the vCluster Platform Helm chart to the image in the private registry. * Set the **default registry** for pulling images used in installing vCluster Platform or connecting hosts (i.e. deploying agents). * Set the **default chart registry** in the `vcluster-platform.yaml` as the default Helm repository when creating tenant clusters. This should prefix with `oci://` to ensure it's being deployed using OCI protocol as well as the `/charts` folder. * For registries that require authentication, create a Kubernetes secret in the namespace where you deploy the vCluster Platform or agent. * Reference the **secret** to use those credentials for the registries. Setting the image for vCluster Platform and the default private registry ``` imageRef: registry: ecr.io # Replace with your private registry repository: myteam/loft-sh/vcluster-platform # Replace with your repository path tag: # Replace with your version env: DEFAULT_IMAGE_REGISTRY: ecr.io/myteam DEFAULT_VCLUSTER_CHART_REPO: oci://ecr.io/myteam/charts HELM_REGISTRY_CONFIG: /loft/helm/.dockerconfigjson volumes: - name: helm-pull-secret secret: secretName: registry-credentials-secret-name # Replace with the name of the secret deployed on the host cluster of where the Platform is deployed volumeMounts: - name: helm-pull-secret mountPath: /loft/helm readOnly: true # The agent values are needed to keep the agent up to date with the same settings agentValues: env: DEFAULT_IMAGE_REGISTRY: ecr.io/myteam DEFAULT_VCLUSTER_CHART_REPO: oci://ecr.io/myteam/charts HELM_REGISTRY_CONFIG: /loft/helm/.dockerconfigjson volumes: - name: helm-pull-secret secret: secretName: registry-credentials-secret-name # Replace with the name of the secret deployed on the host cluster of where the platform is deployed volumeMounts: - name: helm-pull-secret mountPath: /loft/helm readOnly: true ``` warning This only configures the registry authentication for managing tenant clusters with Helm. * See [App Reference](https://vcluster.com/docs/platform/api/resources/apps.md#app-reference) to configure the chart [repository URL](https://vcluster.com/docs/platform/api/resources/apps.md#spec-config-chart-repoURL) with [username](https://vcluster.com/docs/platform/api/resources/apps.md#spec-config-chart-usernameRef) and [password](https://vcluster.com/docs/platform/api/resources/apps.md#spec-config-chart-passwordRef) authentication. * See `experimental.deploy.vcluster.helm` [configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml#experimental-deploy-vcluster) to configure [repository](https://vcluster.com/docs/vcluster/configure/vcluster-yaml#experimental-deploy-vcluster-helm-chart-repo) with [username](https://vcluster.com/docs/vcluster/configure/vcluster-yaml#experimental-deploy-vcluster-helm-chart-username) and [password](https://vcluster.com/docs/vcluster/configure/vcluster-yaml#experimental-deploy-vcluster-helm-chart-password) authentication. Use different vCluster Platform login credentials If the username and password are not set in your `vcluster-platform.yaml`, the default credentials are: * Username: `admin` * Password: `my-password` Set different vCluster Platform login credentials ``` admin: create: true username: newadmin password: "newpassword" ``` Restrict Platform UI requests When the platform runs in an airgapped environment the pods usually don't have a connection to the outside world, but the platform UI running in your browser can still make outbound requests to external services. You can configure the platform to restrict the requests browsers will make when running the platform UI. Add this to your `vcluster-platform.yaml` as a sensible baseline: vcluster-platform.yaml ``` config: uiSettings: externalURLs: block: true allow: - vcluster - gtm - featurebase ``` You can add any origin you need to access from the browser UI to the `allow` list. The platform uses [Content Security Policies (CSPs)](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CSP) to restrict access to external URLs. For example, if you add custom JavaScript or CSS files from another domain, add `assets.my-domain.com/*` to the allow list. In addition to plain URLs we've bundled commonly used ones into predefined "modules". It's recommended to add them to the allow list: * `vcluster`: Allows the UI to make requests to `*.loft.sh` to fetch the latest license information and assets. * `gtm`: Allows the UI to include Google Tag Manager to improve the user experience and add in-product guides * `featurebase`: Allows the UI to fetch and display the changelog of new releases Configure the platform host domain (required) Set `config.loftHost` to the domain where users access the platform. This is required because the Loft Router cannot auto-provision a domain without internet connectivity. Configure the platform host domain ``` config: loftHost: vcluster-platform.example.com ``` warning The `loftHost` value must be resolvable by all connected control plane clusters. ### Install the platform[​](#install-platform "Direct link to Install the platform") It is recommended to install vCluster Platform on its own control plane cluster. #### Install vCluster Platform[​](#install-vcluster-platform "Direct link to Install vCluster Platform") 1. On the control plane cluster, create the namespace for the platform, where vCluster Platform and optionally login credentials are to be deployed in. Create vCluster Platform namespace ``` export PLATFORM_NAMESPACE=vcluster-platform kubectl create namespace ${PLATFORM_NAMESPACE} ``` 2. **Optional**: Create a Secret to authenticate to your private registry. If your private registry requires authentication, create a Kubernetes secret that stores your login credentials. Place this Secret in the same namespace where the vCluster Platform is deployed. There are multiple ways to create your registry credentials as a secret. For more information, see how to quickly create a [Secret from your Docker login](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_create/kubectl_create_secret_docker-registry/). Example of how to create a Kubernetes secret for login credentials ``` kubectl create secret docker-registry registry-credentials-secret-name \ --namespace ${PLATFORM_NAMESPACE} \ --docker-server= \ --docker-username= \ --docker-password= \ --docker-email= ``` 3. Deploy vCluster Platform into the same namespace. Deploy vCluster with Helm ``` export PLATFORM_VERSION=4.12.0 # Replace with the chart version that was pushed into your private registry export REGISTRY=myecr.io/team # Replace with the private registry helm upgrade --install vcluster-platform oci://${REGISTRY}/charts/vcluster-platform:${PLATFORM_VERSION} \ --version ${PLATFORM_VERSION} \ --values vcluster-platform.yaml \ --namespace ${PLATFORM_NAMESPACE} ``` warning vCluster Platform exclusively supports the default `secret` backend for storing Helm releases. Alternative `HELM_DRIVER` configurations (such as `configmap` or `sql`) are not supported. ### Connect control plane clusters[​](#connect-control-plane-clusters "Direct link to Connect control plane clusters") After installing the platform, connect control plane clusters by deploying an agent on each host. The control plane clusters must be able to access the domain where vCluster Platform is hosted. Deploying an agent is deploying the vCluster Platform Helm chart and `vcluster-platform.yaml`. The only difference is when deploying the vCluster Platform Helm chart is adding additional variables to figure out how to connect to the platform. #### Deploy agents to connect control plane clusters[​](#deploy-agents-to-connect-control-plane-clusters "Direct link to Deploy agents to connect control plane clusters") 1. On the control plane cluster, create the namespace for the agent installation, where the agent and optionally login credentials are to be deployed in. Create vCluster Platform namespace ``` export AGENT_NAMESPACE=vcluster-platform kubectl create namespace ${AGENT_NAMESPACE} ``` 2. **Optional**: Create a Secret to authenticate with your private registry. If your private registry requires authentication, create a Kubernetes Secret that stores your login credentials. Place this secret in the same namespace where the agent is deployed. There are multiple ways to create your registry credentials as a Secret. For more details, see [Create a Secret from your Docker login](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_create/kubectl_create_secret_docker-registry/). Create a Kubernetes secret for login credentials example ``` kubectl create secret docker-registry registry-credentials-secret-name \ --namespace ${AGENT_NAMESPACE} \ --docker-server= \ --docker-username= \ --docker-password= \ --docker-email= ``` 3. Create the Cluster resource on the Platform cluster (not the new control plane cluster you're trying to connect) Create Cluster resource on the Platform cluster ``` export CLUSTER_NAME= cat < helm upgrade --install vcluster-platform oci://${REGISTRY}/charts/vcluster-platform:${PLATFORM_VERSION} \ --version ${PLATFORM_VERSION} \ --values vcluster-platform.yaml \ --namespace ${AGENT_NAMESPACE} \ --set agentOnly=true \ --set url=$PLATFORM_DOMAIN \ --set token=$PLATFORM_TOKEN ``` warning vCluster Platform exclusively supports the default `secret` backend for storing Helm releases. Alternative `HELM_DRIVER` configurations (such as `configmap` or `sql`) are not supported. Upgrading later When you're ready to upgrade vCluster Platform or connected control plane clusters, see [Upgrade in an air-gapped environment](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade-air-gapped.md). ## Deploy tenant clusters in air-gapped environments[​](#deploy-tenant-clusters-in-air-gapped-environments "Direct link to Deploy tenant clusters in air-gapped environments") With your vCluster Platform now installed and configured for air-gapped environments, you can deploy tenant clusters. The platform automatically uses your configured private registry settings. Choose the Right Guide After you complete the platform setup, stay on this page if you need to complete additional platform configuration or proceed to the vCluster guide for [deploying individual tenant clusters](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/air-gapped.md): **Continue with this platform air-gapped guide if you need to:** * Create tenant clusters and connect more control plane cluster agents without internet access. * Configure additional platform-wide private registry settings. **Use the [vCluster air-gapped deployment guide](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/air-gapped.md) to:** * Create tenant clusters after the platform is set up. * Review detailed vCluster configuration options for air-gapped environments. ## Configure and deploy tenant clusters[​](#configure-deploy-virtual-clusters "Direct link to Configure and deploy tenant clusters") ### Tenant cluster configuration[​](#virtual-cluster-configuration "Direct link to Tenant cluster configuration") The `vcluster.yaml` file defines all configuration settings for your tenant cluster deployment. Use a private registry without credentials Set the default private registry that does **not** have authentication. Setting the private registry ``` controlPlane: advanced: defaultImageRegistry: ecr.io/myteam # Replace with your private registry ``` Use an authenticated private registry For registries that require authentication, create a Kubernetes secret in the namespace where you deploy the vCluster. Assign the secret as an image pull secret for the vCluster control plane to access required images. Optionally, you can use the same image pull secret for workloads inside the vCluster that pull from the same registry. Image pull secrets configuration ``` controlPlane: advanced: defaultImageRegistry: ecr.io/myteam # Replace with your private image registry serviceAccount: imagePullSecrets: # Uses credentials for the vCluster control plane - name: registry-credentials-secret-name # Replace with the name of the secret deployed on the host cluster of where the vCluster is deployed workloadServiceAccount: # Uses credentials for any workloads created in the vCluster imagePullSecrets: - name: registry-credentials-secret-name # Replace with the name of the secret deployed on the host cluster of where your vCluster is deployed ``` warning When using tenant clusters in air-gapped environments, the `config.experimental.deploy.vcluster.helm` [configuration setting](https://vcluster.com/docs/vcluster/configure/vcluster-yaml#experimental-deploy-vcluster) does not work with external Helm repositories since they cannot be accessed. This means custom Helm charts from repositories like `charts.bitnami.com` cannot be used for tenant cluster deployments. Use a non-default Kubernetes version For vCluster v0.25.0+, if you downloaded `images-optional.txt` to use a different Kubernetes version, configure vCluster to use that version in your deployment. Specific Kubernetes version configuration ``` controlPlane: distro: k8s: image: tag: v1.31.1 # Replace with the Kubernetes version that you have chosen ``` For other versions, refer to the vCluster docs on how to set your image versions. Replace the alpine image If you use the `sync.toHost.pods.rewriteHosts` feature, manually replace the full path of the [alpine image](https://hub.docker.com/_/alpine) with your private registry path. The `defaultImageRegistry` setting does not apply to this image. Replace the alpine image ``` sync: toHost: pods: rewriteHosts: initContainer: image: registry: ecr.io repository: myteam/library/alpine tag: 3.20 ``` ### Deploy vCluster through the platform[​](#deploy-vcluster-through-the-platform "Direct link to Deploy vCluster through the platform") When deploying vCluster through the platform console, you must add your `vcluster.yaml`. #### Use an authenticated private registry[​](#use-an-authenticated-private-registry "Direct link to Use an authenticated private registry") If you are using a private registry with authentication, you must create a Kubernetes secret that stores your login credentials. Place this secret in the same namespace where the vCluster is deployed so the control plane can pull the required container images. In the options for the config, there is a section called `Host namespace`, add the yaml for the secret in the `Kubernetes objects` section. Kubernetes secret for login credentials ``` apiVersion: v1 kind: Secret metadata: name: registry-credentials-secret-name namespace: ${VCLUSTER_NAMESPACE} type: Opaque data: # Credentials based on your private registry ``` ### Deploy vCluster using Helm[​](#deploy-vcluster-using-helm "Direct link to Deploy vCluster using Helm") 1. In the platform UI, create an [access key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) to connect vCluster to the platform. Save the **access key**. 2. Create a Kubernetes secret using the **access key** in a namespace on the control plane cluster. Create platform API key namespace and secret for platform API key ``` export APIKEY_NAMESPACE=api-key-namespace kubectl create namespace ${APIKEY_NAMESPACE} kubectl create -f - < \ --docker-username= \ --docker-password= \ --docker-email= ``` 4. Review your `vcluster.yaml` file. vcluster.yaml configuration ``` controlPlane: advanced: defaultImageRegistry: ecr.io/myteam # Replace with your private image registry serviceAccount: imagePullSecrets: - name: registry-credentials-secret-name # Replace with the name of the registry credentials secret workloadServiceAccount: imagePullSecrets: - name: registry-credentials-secret-name # Replace with the name of the registry credentials secret platform: apiKey: namespace: api-key-namespace # Replace with the namespace that the secret of the access key was deployed in secretName: vcluster-platform-api-key # Replace with the name of the secret of the access key createRBAC: true # Sets up RBAC so that the vcluster can access the secret of the access key ``` 5. Deploy vCluster. Deploy vCluster with Helm ``` export VCLUSTER_NAME="my-vcluster" # Replace with what you want to name your vCluster export VCLUSTER_VERSION=0.37.4 # Replace with the chart version that was pushed into your private registry export REGISTRY=myecr.io/team # Replace with the private registry # Uncomment and create a namespace if you didn't already create one for a secret for the registry # export VCLUSTER_NAMESPACE=vcluster-my-vcluster # Replace with the name of the namespace you want to deploy your vCluster into # kubectl create namespace ${VCLUSTER_NAMESPACE} helm upgrade --install "${VCLUSTER_NAME}" oci://${REGISTRY}/charts/vcluster:${VCLUSTER_VERSION} \ --version ${VCLUSTER_VERSION} \ --values vcluster.yaml \ --namespace ${VCLUSTER_NAMESPACE} ``` --- # Platform Database info This feature is available from the **Platform** version v4.4.0 The platform database is a local PostgreSQL instance provisioned only in air-gapped environments running an [offline license key](https://vcluster.com/docs/platform/install/air-gapped.md). It stores usage snapshots, which are periodic records of CPU and GPU resource consumption used for license billing. The platform attempts to transmit these snapshots to the vCluster license server every 15 minutes. The local database acts as a staging area. Once a snapshot is successfully transmitted, vCluster's license server holds the authoritative record. Data that has not yet been transmitted exists only in this database. The platform database does not store core platform state. Tenant cluster definitions, project settings, users, and all other platform objects are stored as Kubernetes custom resources on the control plane cluster. This database exists solely for usage reporting. Unrelated to the External Database feature This page covers the local PostgreSQL instance used for usage reporting in air-gapped environments. If you need high availability with multiple platform replicas, see [Platform External Database](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/.md). That is a separate feature using a Kine-backed PostgreSQL database as a shared backing store for multi-replica HA deployments, and it requires a fresh installation configured before the platform first starts. ## When the platform database is provisioned[​](#when-the-platform-database-is-provisioned "Direct link to When the platform database is provisioned") The platform database is only created when all of the following conditions are true at startup: * The platform is configured with an [offline license key](https://vcluster.com/docs/platform/install/air-gapped.md) (air-gapped mode). * The platform is not running as a hosted instance (managed and operated by vCluster Labs on your behalf). * The active license plan does not disable the platform database. Standard online-licensed installations do not provision this database. If you do not see a `platform-db` PVC or StatefulSet in your platform namespace, the database is not active in your environment and this page does not apply. ## How the platform database surfaces to administrators[​](#how-the-platform-database-surfaces-to-administrators "Direct link to How the platform database surfaces to administrators") Administrators encounter the platform database in several situations: **Namespace inspection.** The platform database runs as a StatefulSet in the platform namespace. A routine `kubectl get all` or `kubectl get pvc` will show the `platform-db` StatefulSet, Service, PVC, and Secret alongside platform resources. **Cluster conditions.** The controller sets `EmbeddedPostgresDeployed` and `EmbeddedPostgresAvailable` conditions on the `Cluster` resource. If the database fails to start, `kubectl describe cluster.storage.loft.sh ` shows a condition message directing you to investigate. **Platform logs.** If the database is unreachable, the platform logs `error sending usage snapshot` every 15 minutes. Administrators tailing logs during an incident will see this repeating. **Platform upgrades.** The controller restarts the postgres pods during platform upgrades. Administrators watching pod activity during an upgrade will see `platform-db` pods cycling, which is expected behavior. See [Troubleshoot Platform Database](https://vcluster.com/docs/platform/troubleshoot/platform-database.md) for steps to resolve conditions and log errors. ## What usage snapshots record[​](#what-usage-snapshots-record "Direct link to What usage snapshots record") Each snapshot captures a point-in-time count of resource capacity across the platform, not live workload utilization. The data is used to calculate consumption against the committed amounts in your license. | Metric | Description | | ------------------------- | ------------------------------------------------------------------------ | | Tenant cluster count | Total number of tenant clusters across the platform | | Node count | Total number of nodes across all connected clusters | | CPU count | Total CPU capacity across all nodes | | GPU count | Total GPU capacity across all nodes | | Per-node detail | Machine ID, creation timestamp, and full resource capacity for each node | | Per-tenant-cluster detail | Name, namespace, UID, creation timestamp, and node associations | ### GPU metering in Platform 4.12[​](#gpu-metering-in-platform-412 "Direct link to GPU metering in Platform 4.12") Platform 4.12 calculates the metered `gpu` total from physical accelerators advertised in node capacity. It also records a breakdown by accelerator vendor. Earlier versions only included `nvidia.com/gpu` in the total. The following device plugin resource names count toward the total: | Vendor | Counted resource names | | ----------- | ---------------------------------------- | | NVIDIA | `nvidia.com/gpu` | | AMD | `amd.com/gpu` | | Intel GPU | `gpu.intel.com/i915`, `gpu.intel.com/xe` | | Intel Gaudi | `habana.ai/gaudi` | | AWS Neuron | `aws.amazon.com/neuron` | | Google TPU | `google.com/tpu` | Metered totals can increase after an upgrade Existing environments with non-NVIDIA accelerators can report a higher billable GPU total after upgrading. The previous snapshot remains in effect until the platform writes a new usage snapshot, normally within 15 minutes. The breakdown groups accelerators by vendor in Platform 4.12. The `model` value remains empty because the platform doesn't collect the required hardware labels yet. Intel GPUs and Intel Gaudi therefore appear in one `intel` entry. The following accelerator configurations aren't fully represented: * The platform doesn't count NVIDIA sharing resources named `nvidia.com/gpu.shared` or MIG resources named `nvidia.com/mig-*`. * A time-sliced GPU that remains advertised as `nvidia.com/gpu` is counted exactly as advertised. This can exceed the number of physical cards. * The platform doesn't count GPUs allocated through Kubernetes Dynamic Resource Allocation (DRA) ResourceSlices. * The platform counts `aws.amazon.com/neuron` devices and ignores `aws.amazon.com/neuroncore` to avoid counting the same hardware twice. Use the [tenant cluster resource-usage API](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance/resourceusage.md) to inspect the live per-vendor breakdown. Read the `gpus` field for cross-vendor data. The existing `capacity` field only includes `nvidia.com/gpu` for GPU capacity. ## Configure[​](#configure "Direct link to Configure") By default, the platform database uses the default StorageClass. To specify a different StorageClass, navigate to `Admin > Config` and edit the `platformDB.storageClass` field. The value can also be configured in the vCluster Platform Helm values during install or upgrade: ``` platformDB: storageClass: ``` --- # With Offline License Server | | | Enterprise | | | | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ------ | ------ | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Air-Gapped Mode](https://vcluster.com/docs/platform/install/air-gapped.md) | | Add-On | Add-On | Add-On | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). This guide explains how to deploy the offline license server and expose it through an ingress. It also covers configuring vCluster Platform with `LICENSE_SERVER` and exporting usage data. ## Overview[​](#overview "Direct link to Overview") The offline license server runs inside your cluster and handles license validation locally. Use this approach instead of an [offline license key](https://vcluster.com/docs/platform/install/air-gapped.md) when you need: * Usage tracking and data export * A dedicated in-cluster service for license validation without external connectivity The license server stores state in PostgreSQL (embedded or external) and exposes a `/license/v2` API endpoint that vCluster Platform calls for license checks. The platform contacts it on the same schedule it would use for the hosted license server. For that schedule and the surrounding caching behavior, see [How licensing works](https://vcluster.com/docs/platform/understand/licensing.md). ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Kubernetes 1.28+ * Helm 3+ * An offline license key provided by vCluster Labs ## Deploy the license server[​](#deploy-the-license-server "Direct link to Deploy the license server") * Embedded PostgreSQL * External PostgreSQL Use this option to run PostgreSQL as part of the Helm deployment. For production deployments, use an externally managed PostgreSQL. Create namespace and secret ``` kubectl create namespace license-server kubectl create secret generic license-server-config \ --namespace license-server \ --from-literal=postgres-password='my-password' \ --from-literal=static-api-key='' \ --from-literal=offline-license-key='' ``` Install license server with embedded PostgreSQL ``` helm install license-server license-server \ --repo https://charts.loft.sh \ --namespace license-server \ --set postgresql.enabled=true \ --set existingSecret=license-server-config ``` Use this option if your environment already provides PostgreSQL. Create namespace and secret ``` kubectl create namespace license-server kubectl create secret generic license-server-config \ --namespace license-server \ --from-literal=postgres-dsn='postgres://:@:5432/license_server?sslmode=require' \ --from-literal=static-api-key='' \ --from-literal=offline-license-key='' ``` Install license server with external PostgreSQL ``` helm install license-server license-server \ --repo https://charts.loft.sh \ --namespace license-server \ --set postgresql.enabled=false \ --set existingSecret=license-server-config ``` tip `` can be any value you choose. It's used to export usage data later. ## License server endpoint[​](#license-server-endpoint "Direct link to License server endpoint") The license server API base path is `/license/v2`. Use one of these endpoint styles depending on your network setup: * In-cluster: `http://license-server.license-server.svc.cluster.local/license/v2` * Ingress or load balancer: `https:///license/v2` ## Expose the license server using ingress[​](#expose-the-license-server-using-ingress "Direct link to Expose the license server using ingress") If the platform doesn't run in the same Kubernetes cluster as the license server, or you want a stable DNS endpoint, expose the license server through an ingress. * Ingress * Ingress with TLS Install with ingress enabled ``` helm install license-server license-server \ --repo https://charts.loft.sh \ --namespace license-server \ --set postgresql.enabled=true \ --set existingSecret=license-server-config \ --set ingress.enabled=true \ --set ingress.className=nginx \ --set ingress.hosts[0].host=license-server.example.com \ --set ingress.hosts[0].paths[0].path=/ \ --set ingress.hosts[0].paths[0].pathType=Prefix ``` Your platform should target: * `http://license-server.example.com/license/v2` or `https://license-server.example.com/license/v2` (based on your ingress setup) Install with ingress and TLS ``` helm install license-server license-server \ --repo https://charts.loft.sh \ --namespace license-server \ --set postgresql.enabled=true \ --set existingSecret=license-server-config \ --set ingress.enabled=true \ --set ingress.className=nginx \ --set ingress.hosts[0].host=license-server.example.com \ --set ingress.hosts[0].paths[0].path=/ \ --set ingress.hosts[0].paths[0].pathType=Prefix \ --set ingress.tls[0].secretName=license-server-tls \ --set ingress.tls[0].hosts[0]=license-server.example.com ``` Your platform should target: * `https://license-server.example.com/license/v2` ## Configure vCluster Platform to use the offline license server[​](#configure-vcluster-platform-to-use-the-offline-license-server "Direct link to Configure vCluster Platform to use the offline license server") Set the platform environment variable `LICENSE_SERVER` to your license server base URL including `/license/v2`. vcluster-platform.yaml ``` env: LICENSE_SERVER: "https://license-server.example.com/license/v2" ``` If the platform can reach the service directly in-cluster: vcluster-platform.yaml (in-cluster service endpoint) ``` env: LICENSE_SERVER: "http://license-server.license-server.svc.cluster.local/license/v2" ``` If your license server endpoint uses a self-signed certificate, set `insecureSkipVerify: true` in the platform Helm values: vcluster-platform.yaml (self-signed certificate) ``` env: LICENSE_SERVER: "https://license-server.example.com/license/v2" insecureSkipVerify: true ``` insecureSkipVerify disables verification everywhere This value turns off TLS verification for the platform and all its managed components, not just the license server connection. Prefer trusting the signing CA instead, by supplying it as a base64-encoded value in the `additionalCA` Helm value. See [Configure a custom domain](https://vcluster.com/docs/platform/configure/installation-options/domain.md) for how to produce that value. Setting `insecureSkipVerify: true` causes `additionalCA` to be ignored, so use one or the other. Apply the updated values through your normal upgrade flow (for example, `helm upgrade`). See [Upgrade in an air-gapped environment](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade-air-gapped.md) to upgrade vCluster Platform itself. ## Export usage data[​](#export-usage-data "Direct link to Export usage data") If you included `static-api-key` in `license-server-config`, you can export usage data from the license server: ``` curl -X GET -H "Authorization: Bearer " https://my-license-server.com/license/v2/instance/usage/export -o usage.csv ``` ## Uninstall[​](#uninstall "Direct link to Uninstall") ``` helm uninstall license-server --namespace license-server ``` --- # Install vCluster Platform on Amazon EKS This guide provides instructions for installing the platform on [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Ensure you have the following: * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Your current kube-context must have administrative privileges, which you can verify with `kubectl auth can-i create clusterrole -A` info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm` installed: Helm v3.10.0+ is required for deploying the platform. Refer to the [Helm Installation Guide](https://helm.sh/docs/intro/install/) if you need to install it. * `kubectl` installed: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. - vCluster CLI installed: * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` - `AWS CLI` installed: Command-line tool for interacting with Amazon Web Services (AWS) resources and services. For more details on installation, see the [AWS CLI user guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). - `eksctl`: Command-line utility that simplifies creating and managing Kubernetes clusters on Amazon Elastic Kubernetes Service (EKS). For more details on installation, see the [eksctl documentation](https://eksctl.io/installation/). note Ensure you have the necessary IAM permissions to create clusters and manage cloud services. ## Create an AWS EKS cluster[​](#create-an-aws-eks-cluster "Direct link to Create an AWS EKS cluster") 1. Configure the AWS CLI. Run the following command and provide your AWS Access Key ID, Secret Access Key, and preferred output format when prompted: Configure AWS CLI ``` aws configure ``` This sets up local credentials in `~/.aws/credentials` and stores your default region and output format in `~/.aws/config`. This enables the AWS CLI and tools like eksctl to authenticate and access AWS services. 2. Set environment variables. Modify the following with your specific values to generate a copyable command: CLUSTER\_NAMEvcluster-demo REGIONeu-central-1 Set environment variables ``` export CLUSTER_NAME=vcluster-demo export REGION=eu-central-1 ``` 3. [Create an AWS EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) using the `eksctl` CLI: Modify the following with your specific values to generate a copyable command: CLUSTER\_NAMEvcluster-demo REGIONeu-central-1 Create AWS EKS cluster ``` eksctl create cluster \ -name=vcluster-demo \ --enable-auto-mode \ --region=eu-central-1 ``` info This process typically takes about 10-15 minutes. This command creates an AWS EKS cluster named `vcluster-demo` in the region `eu-central-1`. kubeconfig update This command automatically updates your `kubeconfig` file with the new cluster configuration. 4. Verify the cluster creation. Verify that the AWS EKS was created successfully by getting the cluster info: List cluster nodes ``` kubectl cluster-info ``` You should see an output similar to the following: Example output ``` Kubernetes control plane is running at https://8F3D76A1C205B94E32D18FC96E04B73D.gr7.eu-central-1.eks.amazonaws.com ``` ## Set up the platform[​](#set-up-the-platform "Direct link to Set up the platform") After the AWS EKS cluster is running, install the platform. ### Install the platform[​](#install-the-platform "Direct link to Install the platform") 1. [Install the platform](https://vcluster.com/docs/platform/install/quick-start-guide.md) using the vCluster CLI: idempotency The following command is *idempotent*, meaning that running it again does not result it creating another cluster with the same name. ``` vcluster platform start ``` The command prompts you to enter the email address for the admin user: installation expected output ``` By providing your email, you accept our Terms of Service and Privacy Statement: Terms of Service: https://www.loft.sh/legal/terms Privacy Statement: https://www.loft.sh/legal/privacy ? Please specify an email address for the admin user ``` tip If the command takes too long to execute—such as when other cluster operations are in progress—rerun the command. 2. Connect to the platform. After the platform is installed, your default browser opens with the platform UI, and you should see output similar to the following: platform installation output ``` ########################## LOGIN ############################ Username: admin Password: 9758c908-b931-4edd-b3cb-3f034e50651a # Change via UI or via: vcluster platform reset password Login via UI: https://hyx4907.loft.host Login via CLI: vcluster platform login https://hyx4907.loft.host ################################################################# vCluster Platform was successfully installed and can now be reached at: https://hyx4907.loft.host Thanks for using vCluster platform! 19:34:46 done You are successfully logged into vCluster Platform! - Use `vcluster platform create vcluster` to create a new virtual cluster - Use `vcluster platform add vcluster` to add an existing virtual cluster to a vCluster platform instance ``` When logging in via the UI, provide the following details: * **First Name** * **Last Name** * **Email** (pre-filled with the address you supplied earlier) * **Organization** To log in via the CLI, run the `Login via CLI` command provided above. This completes the basic platform installation. For additional configuration and available features, see the [Next steps section](#post-install-steps). You can *optionally* perform additional configuration steps: * [Expose the platform UI using the load balancer](#load-balancer) * [Set up a custom domain and configure DNS](#setup-domain-dns) ### Expose platform UI using the load balancer[​](#load-balancer "Direct link to Expose platform UI using the load balancer") Optionally, you can expose the platform UI using a LoadBalancer service to make it accessible outside the cluster. 1. Create a LoadBalancer service to expose the platform UI. note This assumes the platform is installed in the `vcluster-platform` namespace which is a default installation namespace. ``` cat <`. tip The platform uses a self-signed certificate, so you must accept the warning in your browser. For production use, replace the default self-signed certificate with a valid [TLS certificate](https://vcluster.com/docs/platform/configure/installation-options/domain.md#configure-tls). ### Set up custom domain and configure DNS[​](#setup-domain-dns "Direct link to Set up custom domain and configure DNS") Optionally, you can set up a custom domain and configure DNS to provide a secure URL for accessing the platform. note This example assumes that your cluster has the [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/) installed. You must also have an SSL/TLS certificate issued by [AWS Certificate Manager (ACM)](https://aws.amazon.com/certificate-manager/) that is [created and verified](https://docs.aws.amazon.com/acm/latest/userguide/setup.html). 1. Configure DNS in Amazon Route 53. To use a custom domain, you must create a new Route 53 hosted zone: Modify the following with your specific values to generate a copyable command: DOMAIN NAMEvcluster-platform.yourdomain.tld CALLER REFERENCEvcluster-demo Create a Route 53 Hosted Zone ``` aws route53 create-hosted-zone \ --name=vcluster-platform.yourdomain.tld \ --caller-reference=vcluster-demo ``` 2. Install the `external-dns` EKS add-on. This add-on automatically manages DNS records based on Kubernetes resources such as `Ingress` and `Service`. It integrates with Route 53 and eliminates the need to manually create or update DNS entries. Modify the following with your specific values to generate a copyable command: CLUSTER\_NAMEvcluster-demo Deploy external-dns ``` eksctl create addon \ --name=external-dns \ --cluster=vcluster-demo \ --auto-apply-pod-identity-associations ``` 3. Create an `Ingress` object. Replace `arn:aws:acm:region:account-id:certificate/certificate-id` with the correct value of the ACM certificate Amazon Resource Name (ARN). Modify the following with your specific values to generate a copyable command: INGRESS NAMEvcluster-platform-ingress NAMESPACEvcluster-platform ACM CERTIFICATE ARNarn:aws:acm:region:account-id:certificate/certificate-id DNS HOSTNAMEvcluster-platform.yourdomain.tld ALB GROUP NAMEvcluster-platform HEALTHCHECK PATH/healthz SERVICE NAMEloft SERVICE PORT10443 Create Ingress for platform UI ``` cat < 6m4s v1.31.8 aks-nodepool1-41068841-vmss000001 Ready 6m11s v1.31.8 ``` ## Set up the platform[​](#set-up-the-platform "Direct link to Set up the platform") After the AKS cluster is running, install the platform. ### Install the platform[​](#install-the-platform "Direct link to Install the platform") 1. [Install the platform](https://vcluster.com/docs/platform/install/quick-start-guide.md) using the vCluster CLI. idempotency The following command is *idempotent*, meaning that running it again does not result in creating another cluster with the same name. ``` vcluster platform start ``` The command prompts you to enter the email address for the admin user: Expected installation output ``` By providing your email, you accept our Terms of Service and Privacy Statement: Terms of Service: https://www.loft.sh/legal/terms Privacy Statement: https://www.loft.sh/legal/privacy ? Please specify an email address for the admin user ``` tip If the command takes too long to execute—such as when other cluster operations are in progress—rerun the command. 2. Connect to the platform. After the platform is installed, your default browser opens with the platform UI, and you should see output similar to the following: platform installation output ``` ########################## LOGIN ############################ Username: admin Password: 9758c908-b931-4edd-b3cb-3f034e50651a # Change via UI or via: vcluster platform reset password Login via UI: https://hyx4907.loft.host Login via CLI: vcluster platform login https://hyx4907.loft.host ################################################################# vCluster Platform was successfully installed and can now be reached at: https://hyx4907.loft.host Thanks for using vCluster Platform! 19:34:46 done You are successfully logged into vCluster Platform! - Use `vcluster platform create vcluster` to create a new virtual cluster - Use `vcluster platform add vcluster` to add an existing virtual cluster to a vCluster platform instance ``` When logging in using the UI, provide the following details: * **First Name** * **Last Name** * **Email** (pre-filled with the address you supplied earlier) * **Organization** To log in using the CLI, run the `Login via CLI` command provided above. This completes the basic platform installation. For additional configuration and available features, see the [Next steps section](#post-install-steps). You can *optionally* perform additional configuration steps: * [Expose the platform UI using the load balancer](#load-balancer) * [Set up a custom domain and configure DNS](#setup-domain-dns) ### Expose the platform UI using LoadBalancer[​](#load-balancer "Direct link to Expose the platform UI using LoadBalancer") 1. Optionally, you can expose the platform UI using a LoadBalancer service to make it accessible outside the cluster. note This assumes the platform is installed in the `vcluster-platform` namespace which is a default installation namespace. ``` cat <`. tip The platform uses a self-signed certificate, so you should accept the warning in your browser. For production use, replace the default self-signed certificate with a valid [TLS certificate](https://vcluster.com/docs/platform/configure/installation-options/domain.md#configure-tls). ### Set up custom domain and configure DNS[​](#setup-domain-dns "Direct link to Set up custom domain and configure DNS") Optionally, you can set up a custom domain, SSL certificate, and configure DNS to provide a secure URL for accessing the platform. [Official Azure guide](https://learn.microsoft.com/en-us/azure/aks/app-routing-dns-ssl) 1. All necessary steps for configuring DNS and SSL for Azure Kubernetes Service (AKS) App Routing are thoroughly described in the official Azure documentation. Please refer to the guide here: [Configure DNS and SSL for AKS App Routing](https://learn.microsoft.com/en-us/azure/aks/app-routing-dns-ssl). 2. Connect the platform to the custom domain. After DNS is set up, you can start the platform using the following command: ``` vcluster platform start --host=vcluster-platform.yourdomain.tld ``` info For more information on how to configure a custom domain, see the [Configure external access and TLS documentation](https://vcluster.com/docs/platform/configure/installation-options/domain.md). ## Control pod scheduling on AKS node pools[​](#control-pod-scheduling "Direct link to Control pod scheduling on AKS node pools") Azure Kubernetes Service (AKS) uses two types of node pools with different purposes: * **System node pools**: Host critical infrastructure pods such as CoreDNS and metrics-server. These pools use the label `kubernetes.azure.com/mode: system`. * **User node pools**: Run application workloads and can scale to zero when not needed. By default, the platform pods can be scheduled on any available node. For production environments, you may want to ensure platform pods run on user node pools to keep system node pools reserved for critical infrastructure. ### Available scheduling options[​](#scheduling-options "Direct link to Available scheduling options") You can control where platform pods are scheduled using Helm values during installation or upgrade. Three main approaches are available: * **nodeSelector**: Simplest approach using node labels * **Node affinity**: More flexible with soft and hard scheduling rules * **Tolerations**: Works with node taints to allow pods on specific nodes ### Use nodeSelector for basic scheduling[​](#use-nodeselector "Direct link to Use nodeSelector for basic scheduling") The `nodeSelector` field is the simplest way to constrain platform pods to specific nodes using labels. This example schedules platform pods only on user node pools (labeled with `kubernetes.azure.com/mode: user`), keeping system node pools reserved for critical infrastructure. Modify the following with your specific values to generate a copyable command: NODE\_POOL\_TYPEuser values.yaml - Using nodeSelector to target user node pools ``` # Deploy platform pods only on user node pools nodeSelector: kubernetes.azure.com/mode: user ``` Apply this configuration during platform installation: Install platform with nodeSelector ``` helm upgrade loft vcluster-platform \ --install \ --repo https://charts.loft.sh/ \ --namespace vcluster-platform \ --create-namespace \ --values values.yaml ``` info The `nodeSelector` requires nodes to have the exact label specified. If no matching nodes exist, pods will remain in a pending state. ### Use node affinity for flexible scheduling[​](#use-node-affinity "Direct link to Use node affinity for flexible scheduling") Node affinity provides more control than `nodeSelector`, allowing you to specify both required and preferred scheduling rules. Modify the following with your specific values to generate a copyable command: NODE\_POOL\_TYPEuser values.yaml - Using node affinity with preference rules ``` affinity: nodeAffinity: # Hard requirement: pods must be scheduled on nodes matching this rule requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.azure.com/mode operator: In values: - user # Soft preference: prefer nodes that are not spot instances preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 preference: matchExpressions: - key: kubernetes.azure.com/scalesetpriority operator: NotIn values: - spot ``` The example above: 1. Requires pods to be scheduled only on user node pools 2. Prefers non-spot instances when available tip Use `preferredDuringSchedulingIgnoredDuringExecution` for soft preferences that won't block pod scheduling if the preference can't be satisfied. ### Use tolerations with tainted system pools[​](#use-tolerations "Direct link to Use tolerations with tainted system pools") Microsoft recommends applying the `CriticalAddonsOnly=true:NoSchedule` taint to system node pools. This prevents application pods from being scheduled there unless they have a matching toleration. If your system node pools use this taint, configure platform pods to run on user node pools without requiring tolerations. note By default, platform pods don't include the `CriticalAddonsOnly` toleration, so they won't be scheduled on tainted system node pools. You only need to configure tolerations if you want platform pods to run on tainted nodes. If your user node pools also have custom taints, add the appropriate tolerations: Modify the following with your specific values to generate a copyable command: TAINT\_KEYworkload-type TAINT\_VALUEplatform values.yaml - Using tolerations for custom tainted nodes ``` # Allow scheduling on user node pools with custom taints tolerations: - key: workload-type operator: Equal value: platform effect: NoSchedule ``` ### Combine scheduling mechanisms[​](#combine-scheduling "Direct link to Combine scheduling mechanisms") For precise control, combine multiple scheduling mechanisms: Modify the following with your specific values to generate a copyable command: LABEL\_KEYworkload-type LABEL\_VALUEplatform TAINT\_KEYworkload-type TAINT\_VALUEplatform values.yaml - Combining nodeSelector and tolerations ``` # Target specific node pool by label nodeSelector: workload-type: platform # Allow scheduling on nodes with this taint tolerations: - key: workload-type operator: Equal value: platform effect: NoSchedule ``` This configuration: 1. Selects only nodes labeled with `workload-type: platform` 2. Tolerates the matching taint on those nodes info For more information about platform configuration options, see [platform configuration](https://vcluster.com/docs/platform/configure/installation-options/overview.md). ## Next steps[​](#post-install-steps "Direct link to Next steps") After logging into the UI, you'll be able to start creating virtual clusters immediately. You're automatically part of a [project](https://vcluster.com/docs/platform/understand/what-are-projects.md) called `Default Project`. Click **New Virtual Cluster** and **Create** to spin one up to try out. tip Find more information about creating virtual clusters in the [create virtual clusters](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-no-template.md) section. Otherwise, read more about some primary concepts: * [Projects](https://vcluster.com/docs/platform/understand/what-are-projects.md) - How resources can be grouped together into different projects. * [Tenant clusters](https://vcluster.com/docs/platform/introduction/vcluster_intro.md) - What tenant clusters are and how Platform manages them. * [Templates](https://vcluster.com/docs/platform/understand/what-are-templates.md) - How to use templates to control what type of resources that can be made. * [Connected clusters](https://vcluster.com/docs/platform/understand/what-are-clusters.md) - How to connect Control Plane Clusters to the platform. * [Auto sleep and wakeup](https://vcluster.com/docs/vcluster/manage/sleep-wakeup.md) - How to temporarily scale down unused virtual clusters and bring them back up. You can also use Google as an identity provider and [configure SSO](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md#providers) to enable user authentication to the platform. ## Clean up resources[​](#clean-up-resources "Direct link to Clean up resources") If you installed the AKS cluster using this guide, and want to clean up the resources, run the following command: Clean up resources ``` az aks delete --resource-group $RESOURCE_GROUP_NAME --name $CLUSTER_NAME ``` --- # Install vCluster Platform on Google Cloud using GKE This guide provides instructions for installing the platform on Google Cloud using [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine). ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Ensure you have the following: * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Your current kube-context must have administrative privileges, which you can verify with `kubectl auth can-i create clusterrole -A` info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm` installed: Helm v3.10.0+ is required for deploying the platform. Refer to the [Helm Installation Guide](https://helm.sh/docs/intro/install/) if you need to install it. * `kubectl` installed: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. - vCluster CLI installed: * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` - [Google Cloud SDK (`gcloud` CLI)](https://cloud.google.com/sdk/docs/install) installed. note Ensure you have the necessary IAM permissions to create clusters and manage cloud services. ## Create a GKE cluster[​](#create-a-gke-cluster "Direct link to Create a GKE cluster") 1. Prepare the environment. Start by creating a zonal GKE cluster using the `gcloud` CLI. Set up your environment variables: tip Project ID can be found in the Google Cloud Console under the project name. Alternatively use `gcloud project list` to list all projects and their IDs. To check which project is active, use `gcloud config get-value project`. Modify the following with your specific values to generate a copyable command: PROJECT\_IDdevelopment CLUSTER\_NAMEvcluster-demo ZONEeurope-west1-b MACHINE\_TYPEe2-standard-4 Set environment variables ``` export PROJECT_ID=development export CLUSTER_NAME=vcluster-demo export ZONE=europe-west1-b export MACHINE_TYPE=e2-standard-4 ``` Configure `gcloud`, enable the required APIs, and set the default project: Modify the following with your specific values to generate a copyable command: PROJECT\_IDyour-project-id Configure gcloud ``` gcloud config set project=your-project-id gcloud services enable container.googleapis.com ``` 2. Create the cluster. Modify the following with your specific values to generate a copyable command: CLUSTER\_NAMEvcluster-demo ZONEeurope-west1-b MACHINE\_TYPEe2-standard-4 Create GKE cluster ``` gcloud container clusters create=vcluster-demo \ --zone=europe-west1-b \ --machine-type=e2-standard-4 \ --num-nodes=2 ``` info This process typically takes about 10-15 minutes. Using the default values, this command creates a GKE cluster named `vcluster-demo` in the `europe-west1-b` zone with two nodes of type `e2-standard-4`. kubeconfig update This command automatically updates your `kubeconfig` file with the new cluster configuration. 3. Verify the cluster creation. Verify that GKE was created successfully by listing the nodes: List cluster nodes ``` kubectl get nodes ``` You should see an output similar to the following: Example output ``` NAME STATUS ROLES AGE VERSION gke-vcluster-demo-default-pool-a1b2c3d4-e5f6 Ready 5m v1.30.5-gke.1443001 gke-vcluster-demo-default-pool-a1b2c3d4-g7h8 Ready 5m v1.30.5-gke.1443001 ``` ## Set up the platform[​](#set-up-the-platform "Direct link to Set up the platform") After the GKE cluster is running, install the platform. ### Install the platform[​](#install-the-platform "Direct link to Install the platform") 1. [Install the platform](https://vcluster.com/docs/platform/install/quick-start-guide.md) using the vCluster CLI. idempotency The following command is *idempotent*, meaning that running it again does not result it creating another cluster with the same name. ``` vcluster platform start ``` The command prompts you to enter the email address for the admin user: Expected installation output ``` By providing your email, you accept our Terms of Service and Privacy Statement: Terms of Service: https://www.loft.sh/legal/terms Privacy Statement: https://www.loft.sh/legal/privacy ? Please specify an email address for the admin user ``` tip If the command takes too long to execute—such as when other cluster operations are in progress—rerun the command. 2. Connect to the platform. After the platform is installed, your default browser opens with the platform UI, and you should see output similar to the following: platform installation output ``` ########################## LOGIN ############################ Username: admin Password: 9758c908-b931-4edd-b3cb-3f034e50651a # Change via UI or via: vcluster platform reset password Login via UI: https://hyx4907.loft.host Login via CLI: vcluster platform login https://hyx4907.loft.host ################################################################# vCluster Platform was successfully installed and can now be reached at: https://hyx4907.loft.host Thanks for using vCluster Platform! 19:34:46 done You are successfully logged into vCluster Platform! - Use `vcluster platform create vcluster` to create a new virtual cluster - Use `vcluster platform add vcluster` to add an existing virtual cluster to a vCluster platform instance ``` When logging in using the UI, provide the following details: * **First Name** * **Last Name** * **Email** (pre-filled with the address you supplied earlier) * **Organization** To log in using the CLI, run the `Login via CLI` command provided above. This completes the basic platform installation. For additional configuration and available features, see the [Next steps section](#post-install-steps). You can *optionally* perform additional configuration steps: * [Expose the platform UI using the load balancer](#load-balancer) * [Set up a custom domain and configure DNS](#setup-domain-dns) ### Expose the platform UI using load balancer[​](#load-balancer "Direct link to Expose the platform UI using load balancer") 1. Optionally, you can expose the platform UI using a LoadBalancer service to make it accessible outside the cluster. note This assumes the platform is installed in the `vcluster-platform` namespace which is a default installation namespace. ``` cat <`. tip The platform uses a self-signed certificate, so you should accept the warning in your browser. For production use, replace the default self-signed certificate with a valid [TLS certificate](https://vcluster.com/docs/platform/configure/installation-options/domain.md#configure-tls). ### Set up custom domain and configure DNS[​](#setup-domain-dns "Direct link to Set up custom domain and configure DNS") Optionally, you can set up a custom domain and configure DNS to provide a secure URL for accessing the platform. 1. [Configure DNS in Google Cloud DNS](https://cloud.google.com/dns/docs/set-up-dns-records-domain-name). To use a custom domain, you must configure the DNS to point to the LoadBalancer IP address. Modify the following with your specific values to generate a copyable command: EXTERNAL\_IPload-balancer-external-IP DNS\_ZONE\_NAMEvcluster-platform DOMAINyourdomain.tld. FQDNvcluster-platform.yourdomain.tld. Configure DNS ``` EXTERNAL_IP=load-balancer-external-IP gcloud dns managed-zones create vcluster-platform --dns-name="yourdomain.tld." gcloud dns record-sets transaction start --zone=vcluster-platform gcloud dns record-sets transaction add "${EXTERNAL_IP}" \ --name="vcluster-platform.yourdomain.tld." \ --ttl=300 \ --type=A \ --zone=vcluster-platform gcloud dns record-sets transaction execute --zone=vcluster-platform ``` 2. Connect the platform to the custom domain. After DNS is set up, you can start the platform using the following command: \`vcluster platform start --host=vcluster-platform.yourdomain.tld info For more information on how to configure a custom domain, see the [Configure external acess and TLS documentation](https://vcluster.com/docs/platform/configure/installation-options/domain.md). If you do not have a custom domain setup, follow the [Set up a domain by using Cloud DNS tutorial](https://cloud.google.com/dns/docs/tutorials/create-domain-tutorial). ## Next steps[​](#post-install-steps "Direct link to Next steps") After logging into the UI, you'll be able to start creating virtual clusters immediately. You're automatically part of a [project](https://vcluster.com/docs/platform/understand/what-are-projects.md) called `Default Project`. Click **New Virtual Cluster** and **Create** to spin one up to try out. tip Find more information about creating virtual clusters in the [create virtual clusters](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-no-template.md) section. Otherwise, read more about some primary concepts: * [Projects](https://vcluster.com/docs/platform/understand/what-are-projects.md) - How resources can be grouped together into different projects. * [Tenant clusters](https://vcluster.com/docs/platform/introduction/vcluster_intro.md) - What tenant clusters are and how Platform manages them. * [Templates](https://vcluster.com/docs/platform/understand/what-are-templates.md) - How to use templates to control what type of resources that can be made. * [Connected clusters](https://vcluster.com/docs/platform/understand/what-are-clusters.md) - How to connect Control Plane Clusters to the platform. * [Auto sleep and wakeup](https://vcluster.com/docs/vcluster/manage/sleep-wakeup.md) - How to temporarily scale down unused virtual clusters and bring them back up. You can also use Google as an identity provider and [configure SSO](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md) to enable user authentication to the platform. ## Cleanup[​](#cleanup "Direct link to Cleanup") If you installed the GKE cluster with this tutorial, and want to clean up the resources, run the following command: Clean up resources ``` gcloud container clusters delete $CLUSTER_NAME --zone $ZONE --quiet ``` --- # Install with ArgoCD The platform can be installed using [GitOps](https://argoproj.github.io/cd/), similar to how you would for installing applications in your Kubernetes clusters. This guide provides a quick overview of deploying the platform with GitOps. Since the platform is similar to other applications, standard GitOps practices are equally applicable. info This guide details installing the platform using GitOps practices, specifically with ArgoCD. Although ArgoCD is used as an example, the principles are applicable to other GitOps tools. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Your current kube-context must have administrative privileges, which you can verify with `kubectl auth can-i create clusterrole -A` info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm` installed: Helm v3.10.0+ is required for deploying the platform. Refer to the [Helm Installation Guide](https://helm.sh/docs/intro/install/) if you need to install it. * `kubectl` installed: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. ### ArgoCD[​](#argocd "Direct link to ArgoCD") ArgoCD needs to be installed and configured on the control plane cluster. Follow the [Argo CD Installation Guide](https://argo-cd.readthedocs.io/en/stable/getting_started/) to install it. ### Resource requirements[​](#resource-requirements "Direct link to Resource requirements") * Ensure that the required network ports are open. info The platform requires open ports for API extensions, webhooks, and cost monitoring. In the infrastructure of private Kubernetes clusters, ensure firewall rules allow traffic from the Kubernetes master to platform components. * `8443`—API service extension for cluster communication (`v1.cluster.loft.sh`) * `9443`—Webhook validation and enforcement (`loft webhook`) * `9444`—Management API for platform administration (`v1.management.loft.sh`) * `9090`—Prometheus metrics proxy for cost monitoring For more information, see [Networking](https://vcluster.com/docs/platform/administer/clusters/advanced/networking.md) to learn about port and firewall rules. ## Deployment[​](#deployment "Direct link to Deployment") Required labels for manual namespace creation If you are manually creating the vCluster namespace as part of your GitOps workflow, ensure you apply the following labels: ``` metadata: labels: loft.sh/vcluster-instance-name: dev-reliability # Identifies the specific vCluster instance name loft.sh/vcluster-instance-namespace: p-default # Specifies the parent namespace in Loft loft.sh/vcluster-instance-project: default # Defines the Loft project this vCluster belongs to ``` In other workflows, these labels are automatically applied to the namespace created for you. ### Basic installation[​](#basic-installation "Direct link to Basic installation") note Before beginning, take a few minutes to review the [installation with Helm](https://vcluster.com/docs/platform/install/helm.md) section of the documentation. Helm is particularly well-suited for GitOps as it provides a standardized way to package and version Kubernetes applications. The most basic GitOps platform installation consists of an [ArgoCD Application](https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#applications) with the platform chart and your desired values. #### Create ArgoCD application[​](#create-argocd-application "Direct link to Create ArgoCD application") Execute the following command to create a simple ArgoCD Application. Create platform ArgoCD Application ``` PLATFORM_VERSION=4.12.0 cat < gitops-application.yaml --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: vcluster-platform namespace: argocd spec: destination: name: "" namespace: vcluster-platform server: "https://kubernetes.default.svc" source: path: "" repoURL: "https://charts.loft.sh" targetRevision: $PLATFORM_VERSION chart: vcluster-platform helm: parameters: # admin - name: admin.create value: "true" - name: admin.username value: admin - name: admin.password value: password # ingress - name: ingress.enabled value: "true" - name: ingress.host value: "vcluster-platform.example" - name: ingress.name value: "vcluster-platform-ingress" # audit - name: audit.enableSideCar value: "true" - name: config.audit.level value: "1" # config - name: config.loftHost value: "https://vcluster-platform.example" project: default syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true EOF ``` #### Apply ArgoCD application[​](#apply-argocd-application "Direct link to Apply ArgoCD application") tip To retrieve all available versions of the platform Helm chart, run the following command: List available platform versions ``` helm search repo loft/vcluster-platform ``` Apply the ArgoCD Application to your cluster using the `kubectl apply` command. Apply the ArgoCD Application ``` kubectl apply -f gitops-application.yaml ``` Alternatively, follow the regular GitOps process and push the `gitops-application.yaml` file to your Git repository. The ArgoCD `Application` would install the platform into the current-context configured for your Kubernetes cluster, in the `vcluster-platform` namespace. In this example, values are being passed to set some basic configurations including: * Administrator user/password * Ingress configuration * Audit configuration The platform Deployment automatically installs the platform agent in the local cluster. To manage the agent installation manually, refer to the [agent](#agents) section. ArgoCD and Helm Releases If you are using ArgoCD to manage your platform instance, you won't be able to update the platform configuration through the platform UI. This shouldn't be a problem if you're following a GitOps approach, as manual updates are typically avoided. However, it's important to note that ArgoCD does not deploy the Helm release secret. ## Configuration[​](#configuration "Direct link to Configuration") One of the core tenets of the platform is that all platform resources are just "normal" Kubernetes resources. This means that you can manage any platform objects in the same way you'd manage any other Kubernetes resources in your GitOps workflow. To generate the appropriate manifests to manage each platform resource type, check out the [API documentation](https://vcluster.com/docs/platform/api/resources/project/.md) where you can find example manifests and argument references for nearly all platform resource types. Alternatively, you can use the platform UI to build out your required objects and simply snag the YAML output from the build pane and use that as your manifest content. Here is an example of creating a `Team`, and a `Project` that the `Team` is a member of. You could manage these resources in your GitOps platform, then allow project users to create resources in the platform as they wish. This would put the core pieces under GitOps, freeing teams to control their own environments in the platform manually if they wish. The following manifests could be added into a custom Helm chart, or used as manifests in a repo connected to ArgoCD. This is a contrived example, of course, but the main point here is that all platform resources are just "normal" Kubernetes (custom) resources that can be managed with your GitOps tooling, or any other Kubernetes-centric tooling. management.loft.sh vs storage.loft.sh When working with Platform custom resources in GitOps workflows, you'll encounter two API groups: * **management.loft.sh**: Use this to define the desired state of resources. All resources you create (Teams, Projects, and so on) should use this API group. * **storage.loft.sh**: Used internally by Platform controllers to store actual state. This is auto-generated and should not be manually created. For GitOps, always define resources using `management.loft.sh`. You'll see `storage.loft.sh` referenced in some spec fields (like project members) - this is correct as it references existing storage-backed resources. Example Team and Project resources ``` --- apiVersion: management.loft.sh/v1 kind: Team metadata: name: acme-team spec: displayName: acme-team owner: user: admin access: - verbs: - "*" subresources: - "*" users: - admin - name: vcluster-platform-access verbs: - get - bind subresources: - clusters teams: - acme-team --- kind: Project apiVersion: management.loft.sh/v1 metadata: name: acme-team-project spec: displayName: acme-team-project owner: user: admin quotas: {} allowedClusters: - name: "*" allowedTemplates: - kind: VirtualClusterTemplate group: storage.loft.sh name: "*" - kind: SpaceTemplate group: storage.loft.sh name: "*" members: - kind: Team group: storage.loft.sh name: acme-team clusterRole: loft-management-project-admin access: - name: vcluster-platform-admin-access verbs: - get - update - patch - delete subresources: - "*" users: - admin teams: - acme-team ``` ### Connected clusters[​](#connected-clusters "Direct link to Connected clusters") One of the benefits of the platform is that you can easily manage resources located in many physical clusters by adding them to the platform and using it as your central point of management. When you add a "connected" cluster to the platform, a `Cluster` resource is created and a platform Agent is installed in the cluster to handle local reconciliation tasks. If you are managing the platform using GitOps, you may also wish to manage these connected clusters in a similar fashion, rather than letting the platform install and manage the Agent. tip Managing connected clusters using GitOps offers several advantages: * Consistency: Ensures all clusters are configured identically. * Version Control: Keeps track of changes to cluster configurations over time. * Automation: Reduces manual intervention and potential for human error. * Auditability: Provides a clear record of who made changes and when. ### Cluster resources[​](#cluster-resources "Direct link to Cluster resources") If you would like to manage the platform *and* its agents using your GitOps tooling, you likely also want to manage the connected cluster configurations that live inside the platform. For information about networking requirements, see [Networking](https://vcluster.com/docs/platform/administer/clusters/advanced/networking.md). These configuration elements inform the platform of: * The remotely connected clusters * How to connect to those clusters to validate the agent installation * How to proxy Kubernetes commands from the central platform instance to the remote clusters Each cluster requires two resources: * a `Cluster` object that simply defines the cluster name and the owner of the cluster inside the platform * an associated Secret that contains relevant configuration information for the platform to connect to the cluster. A common strategy for bundling cluster data with your primary platform `Application` (as in an ArgoCD `Application` object) is to write a simple "parent" Helm chart that includes the platform as a dependency. This parent chart can contain anything you'd like, but in this case would be used to include the `Cluster` objects and their associated Secret objects. Cluster Secrets The secret associated with a Cluster object necessarily includes authentication data to access the remote cluster, so take care to ensure that this information is handled appropriately. A simple `Chart.yaml` that includes the base platform chart as a dependency may look similar to this, but you'll need to update `$PLATFORM_VERSION` with a valid platform version. Chart.yaml ``` apiVersion: v2 name: vcluster-platform-manager description: A parent Helm chart for vCluster Platform type: application version: 1.0.0 dependencies: - name: vcluster-platform version: $PLATFORM_VERSION repository: https://charts.loft.sh ``` Values that need to be passed to the dependent platform chart can be passed by referring to the dependency name, in this case, `vcluster-platform`. For example, if you wanted to set the `replicaCount` value on the platform chart, you could do as follows in a `values.yaml` file: ``` vcluster-platform: replicaCount: 3 ``` Who is managing the agent? Remember, if you want to manage the platform agent using your GitOps workflow, make sure you set the `export DISABLE_AGENT=true` environment variable to `true` for your platform installation. The "parent" Helm chart can now include any additional resources that you may want to install with your platform instance. In this case, the chart should include both the Cluster and Secret resources for any connected clusters. You can accomplish this by having a basic template that iterates over an array of clusters that users can provide using values, something like the following: Template for Cluster and Secret resources ``` {{ range .Values.clusters }} --- apiVersion: management.loft.sh/v1 kind: Cluster metadata: name: "{{ .name }}" annotations: argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true spec: access: - subresources: - '*' users: - admin verbs: - '*' config: secretName: "loft-cluster-config-{{ .name }}" secretNamespace: "vcluster-platform" displayName: "{{ .name }}" owner: user: "admin" {{- end }} --- {{- range .Values.clusters }} apiVersion: v1 kind: Secret metadata: name: "loft-cluster-config-{{ .name }}" namespace: "vcluster-platform" type: Opaque data: config: "{{ .config | default "" | b64enc }}" {{- end }} ``` With the preceding template, users would provide an array of maps containing a `name` and a `config` field. For example: Example cluster configuration ``` clusters: - name: my-connected-cluster config: | apiVersion: v1 kind: Config clusters: - name: my-connected-cluster cluster: certificate-authority-data: YOUR-CA-DATA-HERE server: https://1.2.3.4:6443 contexts: - name: my-connected-cluster-context context: cluster: my-connected-cluster namespace: kube-system user: my-connected-cluster-user current-context: my-connected-cluster-context preferences: {} users: - name: my-connected-cluster-user user: token: YOUR-TOKEN-HERE ``` Missing CRDs? You may have noticed a strange annotation on the `Cluster` resource `argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true`. This annotation tells ArgoCD to skip the dry run/check of the resource. You need this for initial installations as the platform itself is going to install the `Cluster` CRD into the cluster. Without this annotation, the `Application` is not going to sync. #### Agents[​](#agents "Direct link to Agents") If you've turned off agent installation on your platform, ensure you manually install the platform agent in each connected cluster. Without this, the platform cannot operate within the connected cluster. The most obvious difference in managing the agents compared to the platform manager is that the agents require installation in the connected "remote" clusters, not local to the platform manager. After adding the relevant clusters as available clusters in your ArgoCD configuration, you can simply create another ArgoCD `Application` to manage the agent. Here is a basic example Remember to update `$PLATFORM_VERSION` variable with a valid platform version. ArgoCD Application for platform agent ``` --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: vcluster-platform-agent-my-other-cluster namespace: argocd spec: destination: name: "" namespace: vcluster-platform server: "https://my-other-cluster:6443" source: path: "" repoURL: "https://charts.loft.sh" targetRevision: "$PLATFORM_VERSION" chart: vcluster-platform helm: parameters: # Required parameter - name: agentOnly value: true # Custom parameters - name: env.SOMEVAR value: my-value project: default syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ``` Once again, note that when managing the agent installations using ArgoCD or your GitOps tooling of choice, ensure that the `DISABLE_AGENT` environment variable is set to `true` for your platform deployment. ## Manage GitOps drift[​](#manage-gitops-drift "Direct link to Manage GitOps drift") When managing Platform resources through GitOps, you may encounter drift detection issues where resources appear out-of-sync even immediately after a successful sync. This typically occurs because Platform controllers modify certain fields after resource creation. ### Why drift occurs[​](#why-drift-occurs "Direct link to Why drift occurs") The Platform controllers may add or modify fields on your resources for several reasons: * **Status fields**: Controllers continuously update status subresources to reflect current state (quota usage, conditions, health checks) * **Default values**: Empty or omitted spec fields may be populated with default values * **Access rules**: Default access rules may be added to resources if not explicitly specified * **Annotations**: Controllers add tracking annotations like `loft.sh/apply`, `loft.sh/project-namespace` * **Finalizers**: The `loft.sh/cleanup` finalizer is auto-added to managed resources ### Drift-causing fields by resource type[​](#drift-causing-fields-by-resource-type "Direct link to Drift-causing fields by resource type") The following table summarizes fields that commonly cause drift for each Platform resource type: | Resource type | Common drift fields | Description | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | **Project** | `status`, `spec.access`, `spec.quotas`, `spec.requireTemplate`, `spec.requirePreset`, `metadata.finalizers`, `metadata.annotations` | Status, default access rules, empty objects get defaults, finalizers, and tracking annotations auto-added | | **Team** | `status`, `spec.access`, `metadata.finalizers`, `metadata.annotations` | Status, default access rules, finalizers, and tracking annotations auto-added | | **VirtualClusterTemplate** | `status`, `spec.template.metadata`, `spec.template.instanceTemplate.metadata`, `spec.template.spaceTemplate.metadata`, `spec.template.accessPoint.ingress`, `metadata.annotations` | Empty nested objects, tracking annotations like `loft.sh/last-applied-parameter-defaults` | | **SpaceTemplate** | `status`, `spec.template.metadata` | Status and metadata defaults | | **SharedSecret** | `status` | Status field updates | | **Cluster** | `status`, `spec.access` | Connection status and default access rules | | **VirtualClusterInstance** | `status`, `spec.clusterRef` | Instance status continuously updated | ### GitOps tool configuration[​](#gitops-tool-configuration "Direct link to GitOps tool configuration") * ArgoCD * Flux ArgoCD provides several mechanisms to handle drift from Platform resources. #### Application-level ignore differences[​](#application-level-ignore-differences "Direct link to Application-level ignore differences") Configure `ignoreDifferences` in your ArgoCD Application to ignore specific fields: ``` apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: platform-resources namespace: argocd spec: # ... other config ... ignoreDifferences: # Ignore status for all Platform custom resources - group: management.loft.sh kind: "*" jsonPointers: - /status # Project-specific: ignore empty spec objects that get defaults - group: management.loft.sh kind: Project jsonPointers: - /spec/quotas - /spec/requireTemplate - /spec/requirePreset - /spec/vault # Team-specific: ignore access modifications - group: management.loft.sh kind: Team jsonPointers: - /spec/access # VirtualClusterTemplate: ignore nested empty metadata objects - group: management.loft.sh kind: VirtualClusterTemplate jsonPointers: - /spec/template/metadata - /spec/template/instanceTemplate/metadata - /spec/template/spaceTemplate/metadata - /spec/template/pro syncPolicy: syncOptions: # Ensures ignored differences are also ignored during sync - RespectIgnoreDifferences=true ``` RespectIgnoreDifferences required Always include `RespectIgnoreDifferences=true` in sync options. Without this, ArgoCD ignores differences for display purposes only but still applies the desired state during sync, overwriting controller-managed fields. #### Use managed fields managers[​](#use-managed-fields-managers "Direct link to Use managed fields managers") For more dynamic drift handling, configure ArgoCD to trust specific field managers: ``` ignoreDifferences: - group: management.loft.sh kind: "*" managedFieldsManagers: - loft ``` This tells ArgoCD to ignore any fields managed by the `loft` controller, automatically adapting to field changes without explicit paths. #### System-level configuration[​](#system-level-configuration "Direct link to System-level configuration") For cluster-wide settings, configure the `argocd-cm` ConfigMap: ``` apiVersion: v1 kind: ConfigMap metadata: name: argocd-cm namespace: argocd data: # Ignore status field for all Platform custom resources resource.customizations.ignoreDifferences.management.loft.sh_Project: | jsonPointers: - /status - /spec/quotas resource.customizations.ignoreDifferences.management.loft.sh_Team: | jsonPointers: - /status resource.customizations.ignoreDifferences.management.loft.sh_VirtualClusterTemplate: | jsonPointers: - /status resource.customizations.ignoreDifferences.management.loft.sh_SharedSecret: | jsonPointers: - /status # Alternative: ignore status for all resources resource.compareoptions: | ignoreResourceStatusField: all ``` Flux uses server-side apply for drift detection and correction, which handles many drift scenarios automatically through Kubernetes managed fields semantics. #### Best practices[​](#best-practices "Direct link to Best practices") 1. **Omit fields you don't manage**: Remove fields that controllers manage (like `status: {}`) from your manifests entirely. Server-side apply only manages fields you explicitly set. 2. **Use field managers**: Flux's `kustomize-controller` acts as a distinct field manager. Fields managed by Platform controllers won't conflict as long as you don't include them in your manifests. 3. **Minimal manifests**: Define only the fields you want to control: ``` # Good: Minimal manifest - Flux won't conflict with controller-managed fields apiVersion: management.loft.sh/v1 kind: Project metadata: name: my-project spec: displayName: My Project allowedClusters: - name: "*" members: - kind: User group: storage.loft.sh name: admin clusterRole: loft-management-project-admin ``` ``` # Avoid: Including empty objects that controllers may populate apiVersion: management.loft.sh/v1 kind: Project metadata: name: my-project spec: displayName: My Project quotas: {} # Remove - controllers will manage this requireTemplate: {} # Remove - let the controller set defaults vault: {} # Remove - unless explicitly configuring Vault status: {} # Remove - never include status in manifests ``` 4. **Force reconciliation for immutable fields**: If drift correction fails due to immutable fields, use the `force` option in your Kustomization: ``` apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization spec: force: true # Recreates resources when patching fails ``` ### Troubleshoot drift issues[​](#troubleshoot-drift-issues "Direct link to Troubleshoot drift issues") If resources continue showing as out-of-sync: 1. **Examine the diff**: Check which specific fields are causing drift in your GitOps tool's UI or CLI 2. **Review controller logs**: Platform controller logs may indicate why fields are being modified 3. **Compare live vs desired state**: Use `kubectl get -o yaml` to see the full live state 4. **Check for webhooks**: Mutating admission webhooks may modify resources after apply ## Upgrade[​](#upgrade "Direct link to Upgrade") To upgrade a GitOps-managed platform deployment, update the chart version in your `Application` or `HelmRelease` manifest. Let your GitOps tool apply it rather than running `helm upgrade` directly. See [Upgrade a GitOps-managed deployment](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade.md#upgrade-a-gitops-managed-deployment). ## Login[​](#login "Direct link to Login") If the `loftHost` is not configured in the platform settings, a random domain is automatically provisioned. You can retrieve this domain from the `loft-router-domain` secret located in the installation namespace. This domain is essential for accessing the platform interface and configuring other services. info You can easily configure your own [custom domain](https://vcluster.com/docs/platform/configure/installation-options/domain.md). Follow these steps to retrieve the domain name from the secret: * Linux and WSL * macOS To retrieve the domain from the secret on **Linux**, run: Retrieve domain value. ``` # Set the namespace where the platform is installed PLATFORM_NAMESPACE=vcluster-platform echo "https://$(kubectl get secret loft-router-domain \ -n "$PLATFORM_NAMESPACE" \ -o jsonpath="{.data.domain}" \ | base64 --decode)" ``` To retrieve the domain from the secret on **macOS**, run: Retrieve domain value. ``` # Set the namespace where the platform is installed PLATFORM_NAMESPACE=vcluster-platform echo "https://$(kubectl get secret loft-router-domain \ -n "$PLATFORM_NAMESPACE" \ -o jsonpath="{.data.domain}" \ | base64 -D)" ``` note If you encounter an error or the secret is not found, ensure that the platform is properly installed and that you have the necessary permissions to access secrets in the specified namespace. ### Default credentials[​](#default-credentials "Direct link to Default credentials") If the username and password are not set in your `values.yaml`, the default credentials are: * Username: `admin` * Password: `my-password` info It is strongly recommended to change these default credentials for security reasons. You can [reset the administrator password](https://vcluster.com/docs/platform/administer/authentication/reset-admin-password.md). ## Next steps[​](#next-steps "Direct link to Next steps") ### Create tenant clusters[​](#create-tenant-clusters "Direct link to Create tenant clusters") After logging into the UI, you'll be able to start creating virtual clusters immediately. You're automatically part of a [project](https://vcluster.com/docs/platform/understand/what-are-projects.md) called `Default Project`. Click **New Virtual Cluster** and **Create** to spin one up to try out. tip Find more information about creating virtual clusters in the [create virtual clusters](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-no-template.md) section. Otherwise, read more about some primary concepts: * [Projects](https://vcluster.com/docs/platform/understand/what-are-projects.md) - How resources can be grouped together into different projects. * [Tenant clusters](https://vcluster.com/docs/platform/introduction/vcluster_intro.md) - What tenant clusters are and how Platform manages them. * [Templates](https://vcluster.com/docs/platform/understand/what-are-templates.md) - How to use templates to control what type of resources that can be made. * [Connected clusters](https://vcluster.com/docs/platform/understand/what-are-clusters.md) - How to connect Control Plane Clusters to the platform. * [Auto sleep and wakeup](https://vcluster.com/docs/vcluster/manage/sleep-wakeup.md) - How to temporarily scale down unused virtual clusters and bring them back up. --- # Install with Helm The platform can be installed directly using Helm commands. Managing the platform installation with Helm directly can be a great way to "GitOps" your platform installation, by using ArgoCD or other GitOps tools to manage the platform installation. This section outlines the basics of deploying and managing the platform with Helm, and can serve as a solid starting point for managing the platform in a GitOps fashion as well. info See the [Platform CLI hub](https://vcluster.com/docs/platform/reference/platform-cli.md) for UI-to-CLI equivalents across common Platform tasks. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Your current kube-context must have administrative privileges, which you can verify with `kubectl auth can-i create clusterrole -A` info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm` installed: Helm v3.10.0+ is required for deploying the platform. Refer to the [Helm Installation Guide](https://helm.sh/docs/intro/install/) if you need to install it. * `kubectl` installed: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. ### Resource requirements[​](#resource-requirements "Direct link to Resource requirements") * Allow egress traffic from vCluster Platform pods to \* (HTTPS, port 443) for license retrieval and usage reporting. If the license response names a different analytics endpoint, allow the platform pods to reach that endpoint too. Only the platform pods need this. Connected cluster agents and clusters don't contact the license server, so they need no licensing-specific egress. For environments without internet access, see the [air-gapped environment guide](https://vcluster.com/docs/platform/install/air-gapped.md). For the full model, see [How licensing works](https://vcluster.com/docs/platform/understand/licensing.md). - Ensure that the required network ports are open. info The platform requires open ports for API extensions, webhooks, and cost monitoring. In the infrastructure of private Kubernetes clusters, ensure firewall rules allow traffic from the Kubernetes master to platform components. * `8443`—API service extension for cluster communication (`v1.cluster.loft.sh`) * `9443`—Webhook validation and enforcement (`loft webhook`) * `9444`—Management API for platform administration (`v1.management.loft.sh`) * `9090`—Prometheus metrics proxy for cost monitoring For more information, see [Networking](https://vcluster.com/docs/platform/administer/clusters/advanced/networking.md) to learn about port and firewall rules. For more detailed information about networking requirements, see [Networking](https://vcluster.com/docs/platform/administer/clusters/advanced/networking.md). ## Deployment[​](#deployment "Direct link to Deployment") The platform installation using Helm is similar to any Helm chart installation. ### Add the repository[​](#add-the-repository "Direct link to Add the repository") If you haven't done so, add the vCluster Helm repository from `https://charts.loft.sh` and update. Add vCluster Helm repository ``` helm repo add loft https://charts.loft.sh helm repo update ``` ### Deploy the platform[​](#deploy-the-platform "Direct link to Deploy the platform") Modify the following with your specific values to generate a copyable command: RELEASE\_NAMEvcluster-platform RELEASE\_NAMESPACEvcluster-platform PLATFORM\_VERSION4.12.0 Deploy the platform ``` helm upgrade vcluster-platform vcluster-platform \ --install \ --repo https://charts.loft.sh/ \ --namespace vcluster-platform \ --create-namespace \ --version 4.12.0 ``` warning vCluster Platform exclusively supports the default `secret` backend for storing Helm releases. Alternative `HELM_DRIVER` configurations (such as `configmap` or `sql`) are not supported. You should see output similar to the following: Output of the Helm deploy command ``` Release "vcluster-platform" does not exist. Installing it now. NAME: vcluster-platform LAST DEPLOYED: Fri Oct 4 13:10:52 2024 NAMESPACE: vcluster-platform STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Thank you for installing vcluster-platform. ``` tip To retrieve all available versions of the platform Helm chart, run the following command: List available platform versions ``` helm search repo loft/vcluster-platform ``` ## Configuration[​](#configuration "Direct link to Configuration") ### Customize values[​](#customize-values "Direct link to Customize values") As with most Helm charts, the `vcluster-platform` chart supports many user-configurable values to modify the installation of resources. You can find the `vcluster-platform` chart in the [vCluster Helm repository](https://charts.loft.sh) (source: [loft-sh/loft on GitHub](https://github.com/loft-sh/loft/tree/master/chart)). The `values.yaml` file in the chart repository contains the default values, as well as many comments showing optional values and additional information. To customize your installation, you can either: 1. Copy the default values file and modify it to suit your installation needs, or 2. Create a new YAML file containing only the desired values settings. ### Configure platform options[​](#configure-platform-options "Direct link to Configure platform options") Inside the `values.yaml` file, there is a `config` field, which contains the configuration options for the platform. These options are available in the UI under `Platform` > `Platform Config` > `Config` section and can be edited either through the `values.yaml` or the UI. There is a detailed list of [available configuration options](https://vcluster.com/docs/platform/configure/platform-configs/overview.md) in the docs. ### Set recommended values[​](#set-recommended-values "Direct link to Set recommended values") There are a set of fields in the `values.yaml` that are recommended to explicitly set: * `config.loftHost` - Sets a publicly resolvable hostname. This value should be the same as the `ingress.host` if an ingress is configured for the installation. Loft Router vCluster provides a hosted domain for the vCluster platform, referred to as the Loft Router. The Loft Router enables you to trial the vCluster platform using a hosted domain by vCluster. This is particularly useful when you cannot use your own DNS to create an ingress for the vCluster platform but still wants the trial environment accessible over the public internet. It is enabled by default. You can disable it by either: * Explicitly setting `config.loftHost` and `config.devPodSubDomain`, or * Setting the environment variable `DISABLE_LOFT_ROUTER=true`. Set the environment variable to disable Loft Router ``` helm upgrade vcluster-platform vcluster-platform --install \ --repo https://charts.loft.sh/ \ --namespace vcluster-platform \ --create-namespace \ --values values.yaml \ --set env.DISABLE_LOFT_ROUTER='true' ``` * `admin.username` - Sets the username of an administrator user. By default, the username is `admin`. * `admin.password` - Sets the password of an administrator user. By default, the password is `my-password`. Recommended values.yaml and config section. ``` admin: username: my-own-username password: my-admin-password config: audit: enabled: true loftHost: vcluster-platform.mytld.com ingress: enabled: true host: vcluster-platform.mytld.com ``` * `env`—Defines environment variables for the vCluster platform components. By default, this field is empty. A common use case is configuring network access between the vCluster platform and managed tenant clusters to allow API communication. If both the platform and the tenant clusters run on the same control plane cluster, you should also configure the `NO_PROXY` (or `no_proxy`) environment variable to bypass proxies for internal traffic if you are using proxies. Otherwise, the platform will be unable to reach the vCluster control plane Service, and features such as the Cost Control dashboard will fail to function properly. values.yaml ``` env: NO_PROXY: localhost,127.0.0.1,.svc,.svc.cluster.local,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 no_proxy: localhost,127.0.0.1,.svc,.svc.cluster.local,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 HTTPS_PROXY: "https://proxy.example.com:8080" HTTP_PROXY: "http://proxy.example.com:8080" https_proxy: "https://proxy.example.com:8080" http_proxy: "http://proxy.example.com:8080" ``` ### Apply custom configuration[​](#apply-custom-configuration "Direct link to Apply custom configuration") Values files can be passed to the Helm upgrade command. Modify the following with your specific values to generate a copyable command: RELEASE\_NAMEvcluster-platform RELEASE\_NAMESPACEvcluster-platform PLATFORM\_VERSION4.12.0 Deploy with custom values.yaml ``` helm upgrade vcluster-platform vcluster-platform \ --install \ --repo https://charts.loft.sh/ \ --namespace vcluster-platform \ --create-namespace \ --values values.yaml \ --version 4.12.0 ``` You can provide multiple values files if desired by specifying additional file names after the `--values` flag. This can be handy if you prefer to break up the values sections into different files. Helm has many additional flags and other ways to pass values. As always, it's a great idea to check out the Helm [docs](https://Helm.sh/docs/) to stay up to date on Helm. ## Login[​](#login "Direct link to Login") If the `loftHost` is not configured in the platform settings, a random domain is automatically provisioned. You can retrieve this domain from the `loft-router-domain` secret located in the installation namespace. This domain is essential for accessing the platform interface and configuring other services. info You can easily configure your own [custom domain](https://vcluster.com/docs/platform/configure/installation-options/domain.md). Follow these steps to retrieve the domain name from the secret: * Linux and WSL * macOS To retrieve the domain from the secret on **Linux**, run: Retrieve domain value. ``` # Set the namespace where the platform is installed PLATFORM_NAMESPACE=vcluster-platform echo "https://$(kubectl get secret loft-router-domain \ -n "$PLATFORM_NAMESPACE" \ -o jsonpath="{.data.domain}" \ | base64 --decode)" ``` To retrieve the domain from the secret on **macOS**, run: Retrieve domain value. ``` # Set the namespace where the platform is installed PLATFORM_NAMESPACE=vcluster-platform echo "https://$(kubectl get secret loft-router-domain \ -n "$PLATFORM_NAMESPACE" \ -o jsonpath="{.data.domain}" \ | base64 -D)" ``` note If you encounter an error or the secret is not found, ensure that the platform is properly installed and that you have the necessary permissions to access secrets in the specified namespace. ### Default credentials[​](#default-credentials "Direct link to Default credentials") If the username and password are not set in your `values.yaml`, the default credentials are: * Username: `admin` * Password: `my-password` info It is strongly recommended to change these default credentials for security reasons. You can [reset the administrator password](https://vcluster.com/docs/platform/administer/authentication/reset-admin-password.md). Navigate to the domain in your browser and log in with the default credentials or the credentials you set in `values.yaml` and complete the profile setup. ## Next steps[​](#next-steps "Direct link to Next steps") ### Create tenant clusters[​](#create-tenant-clusters "Direct link to Create tenant clusters") After logging into the UI, you'll be able to start creating virtual clusters immediately. You're automatically part of a [project](https://vcluster.com/docs/platform/understand/what-are-projects.md) called `Default Project`. Click **New Virtual Cluster** and **Create** to spin one up to try out. tip Find more information about creating virtual clusters in the [create virtual clusters](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-no-template.md) section. Otherwise, read more about some primary concepts: * [Projects](https://vcluster.com/docs/platform/understand/what-are-projects.md) - How resources can be grouped together into different projects. * [Tenant clusters](https://vcluster.com/docs/platform/introduction/vcluster_intro.md) - What tenant clusters are and how Platform manages them. * [Templates](https://vcluster.com/docs/platform/understand/what-are-templates.md) - How to use templates to control what type of resources that can be made. * [Connected clusters](https://vcluster.com/docs/platform/understand/what-are-clusters.md) - How to connect Control Plane Clusters to the platform. * [Auto sleep and wakeup](https://vcluster.com/docs/vcluster/manage/sleep-wakeup.md) - How to temporarily scale down unused virtual clusters and bring them back up. --- # Install with CLI The `vCluster CLI` is the preferred method for installing the platform to a Kubernetes cluster. It streamlines the installation process and ensures compatibility with your Kubernetes environment. If you later decide to manage your platform instance using GitOps, you can export the applied Kubernetes manifests or Helm chart configuration and store them in a Git repository. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Your current kube-context must have administrative privileges, which you can verify with `kubectl auth can-i create clusterrole -A` info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm` installed: Helm v3.10.0+ is required for deploying the platform. Refer to the [Helm Installation Guide](https://helm.sh/docs/intro/install/) if you need to install it. * `kubectl` installed: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. ### Download and install vCluster CLI[​](#download-and-install-vcluster-cli "Direct link to Download and install vCluster CLI") Use one of the following commands to download the vCluster CLI binary from GitHub: * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` tip After installation, verify the successful installation of the vCluster CLI by running `vcluster version` in your terminal. This command should display the installed version of the vCluster CLI. ### Resource requirements[​](#resource-requirements "Direct link to Resource requirements") * Allow egress traffic from vCluster Platform pods to \* (HTTPS, port 443) for license retrieval and usage reporting. If the license response names a different analytics endpoint, allow the platform pods to reach that endpoint too. Only the platform pods need this. Connected cluster agents and clusters don't contact the license server, so they need no licensing-specific egress. For environments without internet access, see the [air-gapped environment guide](https://vcluster.com/docs/platform/install/air-gapped.md). For the full model, see [How licensing works](https://vcluster.com/docs/platform/understand/licensing.md). - Ensure that the required network ports are open. info The platform requires open ports for API extensions, webhooks, and cost monitoring. In the infrastructure of private Kubernetes clusters, ensure firewall rules allow traffic from the Kubernetes master to platform components. * `8443`—API service extension for cluster communication (`v1.cluster.loft.sh`) * `9443`—Webhook validation and enforcement (`loft webhook`) * `9444`—Management API for platform administration (`v1.management.loft.sh`) * `9090`—Prometheus metrics proxy for cost monitoring For more information, see [Networking](https://vcluster.com/docs/platform/administer/clusters/advanced/networking.md) to learn about port and firewall rules. For more detailed information about networking requirements, see [Networking](https://vcluster.com/docs/platform/administer/clusters/advanced/networking.md). ## Deployment[​](#deployment "Direct link to Deployment") Before installing the platform, ensure the correct kube-context is active. The kube-context configures which Kubernetes cluster and namespace `kubectl` commands interact with. To confirm the current context, run: Get the current Kubernetes context ``` kubectl config current-context ``` Once you've confirmed the correct context, install the platform by running: Deploy the platform ``` vcluster platform start ``` This command installs the platform into the current cluster in the `vcluster-platform` namespace. The `vcluster-platform` namespace is created automatically and serves as a dedicated space for the platform components. The UI automatically opens in your browser and logs you in. You are asked for your user details to create the administrator user. info The installation process typically takes less than 1 minute, but can take up to 2 minutes depending on your cluster's resources and network speed. You should see output similar to the following: Platform installation output ``` ... ########################## LOGIN ############################ Username: admin Password: 27177595-21bc-4ff9-9f3a-51e0f722408b # Change via UI or via: vcluster platform reset password Login via UI: https://hth45c8.loft.host Login via CLI: vcluster platform login https://hth45c8.loft.host ################################################################# vCluster Platform was successfully installed and can now be reached at: https://hth45c8.loft.host Thanks for using vCluster Platform! 11:38:33 done You are successfully logged into vCluster Platform! - Use `vcluster platform create vcluster` to create a new virtual cluster - Use `vcluster platform add vcluster` to add an existing virtual cluster to a vCluster platform instance ``` After successful installation, the UI automatically opens in your default web browser. You are prompted to create an administrator user. Troubleshooting If your platform pod is not starting or the UI is not showing up, view the [Troubleshooting Guide](https://vcluster.com/docs/platform/use-platform/troubleshooting/.md) for assistance. ## Configuration[​](#configuration "Direct link to Configuration") The `vcluster CLI` offers various configuration options to customize the installation process. ### Non-default Kubernetes cluster[​](#non-default-kubernetes-cluster "Direct link to Non-default Kubernetes cluster") By default, the platform is installed to the current Kubernetes context. Get the current Kubernetes context ``` kubectl config current-context ``` This can be overridden by specifying the `--context` flag: Deploy the platform to a specific Kubernetes context ``` vcluster platform start --context my-cluster ``` ### Custom namespace[​](#custom-namespace "Direct link to Custom namespace") By default, the platform is installed to the `vcluster-platform` namespace. This can be changed by specifying the `--namespace` flag: Deploy the platform to a custom namespace ``` vcluster platform start --namespace my-namespace ``` info If the specified namespace does not exist, it is created automatically. ### Different platform version[​](#different-platform-version "Direct link to Different platform version") The `vCluster CLI` uses the latest version of the platform Helm chart by default. To use a specific version, specify the `--version` flag: tip To retrieve all available versions of the platform Helm chart, run the following command: List available platform versions ``` helm search repo loft/vcluster-platform ``` Deploy the platform using a specific version ``` vcluster platform start --version 4.0.0-alpha.0 ``` tip Provide version number without the `v` prefix. However if you do, the CLI automatically removes it. info Refer to the [vCluster CLI Reference](https://vcluster.com/docs/vcluster/cli/vcluster_platform) for more configuration options. See the [Platform CLI hub](https://vcluster.com/docs/platform/reference/platform-cli.md) for UI-to-CLI equivalents across common Platform tasks. ## Login[​](#login "Direct link to Login") While `vcluster platform start` logs you in automatically, you can also manually log in using the CLI: Platform login ``` vcluster platform login https://1rtjxak.loft.host # See `vCluster platform start` output for login credentials and your actual sub-domain name (auto-generated). ``` This command opens the browser for signing in using the login data displayed in the output of `vcluster platform start`. Manual login may be necessary if accessing the platform from a different machine or if the automatic login process is interrupted. ## Next steps[​](#next-steps "Direct link to Next steps") ### Create tenant clusters[​](#create-tenant-clusters "Direct link to Create tenant clusters") After logging into the UI, you'll be able to start creating virtual clusters immediately. You're automatically part of a [project](https://vcluster.com/docs/platform/understand/what-are-projects.md) called `Default Project`. Click **New Virtual Cluster** and **Create** to spin one up to try out. tip Find more information about creating virtual clusters in the [create virtual clusters](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-no-template.md) section. Otherwise, read more about some primary concepts: * [Projects](https://vcluster.com/docs/platform/understand/what-are-projects.md) - How resources can be grouped together into different projects. * [Tenant clusters](https://vcluster.com/docs/platform/introduction/vcluster_intro.md) - What tenant clusters are and how Platform manages them. * [Templates](https://vcluster.com/docs/platform/understand/what-are-templates.md) - How to use templates to control what type of resources that can be made. * [Connected clusters](https://vcluster.com/docs/platform/understand/what-are-clusters.md) - How to connect Control Plane Clusters to the platform. * [Auto sleep and wakeup](https://vcluster.com/docs/vcluster/manage/sleep-wakeup.md) - How to temporarily scale down unused virtual clusters and bring them back up. --- # Argo CD | | | Enterprise | | | | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Argo Integration](https://vcluster.com/docs/platform/integrations/argocd.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). The Argo CD integration connects vCluster Platform to an Argo CD or [Akuity](https://akuity.io) instance using declarative *connectors*. A connector is a Kubernetes Secret stored in the Platform namespace that holds the credentials and endpoint for your Argo CD server. Once a connector exists, Platform automatically registers tenant clusters and control plane clusters with Argo CD and manages Argo CD Applications on their behalf. This enables a fully GitOps-driven workflow with no manual cluster imports or Argo CD UI configuration: * **Connector-based cluster registration**: Clusters register with Argo CD when you reference a connector by name. No manual import is needed. * **Akuity support**: In addition to self-hosted Argo CD servers, the integration supports [Akuity](https://akuity.io). Platform fetches and applies the Akuity agent manifest to each registered cluster automatically. * **Declarative application deployment**: Define Argo CD Applications directly in your tenant cluster configuration. Platform creates and syncs them without requiring access to the Argo CD UI. * **Control plane cluster registration**: Register control plane clusters with Argo CD directly, and deploy shared infrastructure applications to them alongside tenant cluster applications. ## Key concepts[​](#key-concepts "Direct link to Key concepts") ### Connectors[​](#connectors "Direct link to Connectors") A connector is a Kubernetes Secret in the Platform namespace labeled `loft.sh/connector-type: argocd`. Platform discovers connectors by this label and makes them available by name to any cluster in the platform. The connector holds the Argo CD server URL, authentication credentials, and for Akuity the organization and instance identifiers. A single connector can be referenced by many clusters. You create it once and reference it by name from each cluster or tenant cluster configuration. caution Deleting a connector, or changing what it points to, doesn't delete any Argo CD Application it already created. Existing clusters and Stack tasks that reference it simply stop reconciling and report `InvalidConnector` until a connector with that name exists again. See [Reasons that can come from a blocked task](https://vcluster.com/docs/platform/troubleshoot/stacks.md#reasons-that-can-come-from-a-blocked-task) to diagnose the StackInstance. Pointing an existing connector at a different Argo CD server doesn't clean up applications left behind on the previous server either, since Platform tracks a connector by name, not by target. Check the previous server for orphaned applications after repointing a connector. ### Cluster registration[​](#cluster-registration "Direct link to Cluster registration") When a tenant cluster or control plane cluster references a connector, Platform registers that cluster as a destination in Argo CD. The registration uses the Platform proxy as the API server endpoint and a scoped access key as the bearer token. This means: * The cluster's Kubernetes API server does not need to be publicly reachable from Argo CD. * Access is scoped per cluster. Argo CD authenticates using a Platform-issued access key, not a long-lived kubeconfig. * When a tenant cluster is deleted, Platform deregisters it from Argo CD automatically. For Akuity, registration also provisions an agent inside the cluster (see [Connect to Akuity](https://vcluster.com/docs/platform/integrations/argocd/connect-akuity.md) for details). danger Disabling the Argo CD integration on a tenant cluster or control plane cluster deletes that cluster's Argo CD Applications from the connected Argo CD server, not just the Platform-side `ArgoCDApplication` objects. Confirm Argo CD no longer needs to manage that cluster's applications before disabling the integration. ### ArgoCDApplication and ArgoCDApplicationTemplate[​](#argocdapplication-and-argocdapplicationtemplate "Direct link to ArgoCDApplication and ArgoCDApplicationTemplate") The integration introduces two Platform-level resources: * **`ArgoCDApplication`**: Represents an Argo CD application targeted at either the tenant cluster (`vcluster`) or its control plane cluster (`host`). Platform creates the corresponding Argo CD Application object and keeps it in sync * **`ArgoCDApplicationTemplate`**: A reusable Argo CD ApplicationSpec with named parameters. Templates allow the same application definition to be referenced across many tenant clusters and control plane clusters without duplication. These resources live in the Platform project namespace and are managed by Platform, not by Argo CD directly. A Stack task can create an ArgoCDApplication from an inline definition or an ArgoCDApplicationTemplate reference. Use a [Stack](https://vcluster.com/docs/platform/understand/what-are-stacks.md) when separate applications need dependency ordering, health gates, or output handoff. The destination must have an enabled Argo CD connector, and the task requires the Argo CD Integration feature. Restoring a tenant cluster doesn't roll back what Argo CD manages The `ArgoCDApplication` object itself lives in the Platform project namespace, entirely outside a tenant cluster's own [snapshot and restore](https://vcluster.com/docs/vcluster/manage/backup-restore/restore.md) boundary, so a restore never touches it. The resources Argo CD deployed into the tenant cluster aren't exempt. They're part of the tenant cluster's own state, so a restore reverts them to their snapshotted versions along with everything else. Whether that reverted state persists depends on the application's `syncPolicy.automated` settings: * **`selfHeal: true`**: after Argo CD detects the drift between the reverted live state and Git, it reapplies the Git revision, undoing the restore for those resources. Git always wins. * **`prune: true`**: required alongside `selfHeal` to remove any resources the restore brought back that no longer exist in Git. Without pruning, Argo CD reapplies drifted fields but leaves stale resources in place. * **Automated sync enabled, `selfHeal: false`**: live drift alone doesn't trigger an immediate resync, but this isn't a durable rollback. The next Git change to the application still triggers a normal sync, which reapplies the full desired state and overwrites the restored resources. * **Automated sync disabled entirely**: the only case where the restored state persists until you trigger a manual sync yourself. The application shows as `OutOfSync` or `Missing` in Argo CD in the meantime. To use a snapshot as a durable rollback mechanism for resources Argo CD manages, disable automated sync, not just `selfHeal`, before restoring. ## Standard Argo CD vs Akuity[​](#standard-argo-cd-vs-akuity "Direct link to Standard Argo CD vs Akuity") Use the standard Argo CD integration if you run your own Argo CD server. Argo CD reaches registered clusters through the Platform proxy rather than a direct connection. Choose the Akuity integration if your Argo CD instance is managed by Akuity. Platform still uses its proxy to register the cluster with Akuity and to install and update the in-cluster agent. Once installed, that agent connects outward to Akuity on its own and reconciles Applications independently of the Platform proxy. | | Standard Argo CD | Akuity | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | | Argo CD hosting | Self-hosted | Akuity-managed | | Cluster connectivity | Routed through the Platform proxy | Agent installed inside the cluster | | Private clusters | Fully supported. Argo CD talks to the Platform proxy, and Platform reaches the cluster through its own outbound-only agent connection | Fully supported (outbound agent connection only) | | Agent management | Not applicable | Platform provisions and updates the agent automatically | ## Get started[​](#get-started "Direct link to Get started") * [Connect to Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md) — set up a standard Argo CD connector and register clusters * [Connect to Akuity](https://vcluster.com/docs/platform/integrations/argocd/connect-akuity.md) — set up an Akuity connector with automatic agent provisioning * [Deploy applications](https://vcluster.com/docs/platform/integrations/argocd/deploy-applications.md) — declare Argo CD Applications in your tenant cluster configuration * [Feature support and limitations](https://vcluster.com/docs/platform/integrations/argocd/considerations.md) — what's passed through, what's not modeled, and common patterns Legacy integration If you are using the project-level Argo CD integration from an earlier version of vCluster Platform, see [Legacy Argo CD integration](https://vcluster.com/docs/platform/integrations/argocd/legacy.md). That integration is deprecated and will be removed in a future release. --- # Connect to Akuity This guide connects vCluster Platform to an [Akuity](https://akuity.io)-managed Argo CD instance. At the end you will have a connector that registers tenant clusters and control plane clusters with Akuity and provisions the Akuity agent inside each cluster automatically. For a self-hosted Argo CD server, see [Connect to Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md) instead. ## How the Akuity agent works[​](#how-the-akuity-agent-works "Direct link to How the Akuity agent works") When you use an Akuity connector, Argo CD does not connect directly to your cluster's API server. Instead, Akuity uses a lightweight agent running inside each registered cluster that establishes an outbound connection to the Akuity control plane. The managed Akuity Argo CD instance communicates with the cluster through that persistent connection. This means the cluster's API server does not need to be publicly reachable — the agent runs inside the cluster and connects outward — so private clusters are fully supported. You do not install or manage the agent manually. When a cluster is registered with an Akuity connector, Platform: 1. Calls the Akuity API to register the cluster and retrieve the agent installation manifest. 2. Applies the manifest to the cluster. The agent is installed in its own namespace within the cluster. 3. Stores a hash of the applied manifest in a cluster annotation. If the manifest changes (for example, after an Akuity agent version update), Platform detects the drift and applies the updated manifest automatically. When the integration is disabled or the connector is changed, Platform removes the agent namespace from the cluster and deregisters it from Akuity. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * vCluster Platform running with admin access * An Akuity organization with an active Argo CD instance * An Akuity API key (see below) ### Akuity API key permissions[​](#akuity-api-key-permissions "Direct link to Akuity API key permissions") The Akuity API key must have a [custom role](https://docs.akuity.io/organizations/custom-roles/) with the following permissions on the target Argo CD instance. The built-in **Owner** role satisfies all of these. For least-privilege access, create a custom role with only the permissions below. | Resource | Permission | Why | | --------------- | ---------- | ---------------------------------------------------------------- | | Argo CD Cluster | `Get` | Fetch cluster registration state and agent manifests | | Argo CD Cluster | `Create` | Register new tenant clusters and control plane clusters | | Argo CD Cluster | `Update` | Update cluster registration when connector or access key changes | | Argo CD Cluster | `Delete` | Deregister clusters when the integration is disabled | See the [Akuity API key documentation](https://docs.akuity.io/organizations/api-keys) for instructions on creating an API key and attaching a custom role. ## Step 1: Create a connector[​](#step-1-create-a-connector "Direct link to Step 1: Create a connector") * Platform UI * YAML 1. Go to Infrastructure > Connectors and select the Argo CD tab. 2. Click Connect Argo CD. 3. In the Display name field, enter a human-readable name for the connector. The Argo CD Connector ID is auto-generated from the display name and is used to reference this connector from clusters. 4. Enable the Use Akuity API toggle, then fill in: * Org ID: your Akuity organization ID * Argo CD Instance: the Argo CD instance ID within your organization * API Key ID and API Key Secret: your Akuity API key credentials 5. In the Server URL field, enter the Akuity instance URL. The format is `https://.cd.akuity.cloud/`. 6. If Argo CD is installed in a namespace other than `argocd`, update the Argo CD Namespace field. 7. Select an authentication method and fill in the credentials: * Authenticate with Argo CD API token: paste your token in the Argo CD API Token field. * Authenticate with username / password: fill in Username and Password. 8. In the Agent Configuration section, select an Agent Size for the cluster workload. Optionally override Replicas and Memory for the `argocd-repo-server` (see [Agent sizing](#agent-sizing) below). 9. Click Connect Argo CD. Set `connectorType: "akuity"` to use Akuity. Akuity supports two authentication methods for the Argo CD API. If `token` is set, the controller uses bearer token authentication. Otherwise it falls back to `username` and `password`. Apply the Secret with `kubectl apply`. * Token * Basic auth ``` apiVersion: v1 kind: Secret metadata: name: akuity-prod namespace: loft labels: loft.sh/connector-type: argocd type: Opaque stringData: connectorType: "akuity" server: "https://.cd.akuity.cloud/" token: "" namespace: "argocd" insecure: "false" akuityOrgId: "" akuityInstanceId: "" akuityApiKeyId: "" akuityApiKeySecret: "" akuityAgentSize: "CLUSTER_SIZE_MEDIUM" akuityRepoServerReplicas: "1" akuityRepoServerMemory: "1Gi" ``` ``` apiVersion: v1 kind: Secret metadata: name: akuity-prod namespace: loft labels: loft.sh/connector-type: argocd type: Opaque stringData: connectorType: "akuity" server: "https://.cd.akuity.cloud/" username: "" password: "" namespace: "argocd" insecure: "false" akuityOrgId: "" akuityInstanceId: "" akuityApiKeyId: "" akuityApiKeySecret: "" akuityAgentSize: "CLUSTER_SIZE_MEDIUM" akuityRepoServerReplicas: "1" akuityRepoServerMemory: "1Gi" ``` | Field | Required | Description | | -------------------------- | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `connectorType` | Yes | Must be `"akuity"` | | `server` | Yes | Akuity instance URL. Format: `https://.cd.akuity.cloud/` | | `token` | If set, bearer token authentication is used | Argo CD API bearer token for the Akuity instance | | `username` | Required when `token` is not set | Argo CD username | | `password` | Required when `token` is not set | Argo CD password | | `namespace` | No | Namespace where Argo CD is installed. Defaults to `argocd` | | `insecure` | No | Set to `"true"` to skip TLS verification. Defaults to `"false"` | | `akuityOrgId` | Yes | Akuity organization ID | | `akuityInstanceId` | Yes | Argo CD instance ID within your Akuity organization | | `akuityApiKeyId` | Yes | Akuity API key ID, used to register and manage clusters using the Akuity API | | `akuityApiKeySecret` | Yes | Akuity API key secret | | `akuityAgentSize` | No | Agent size profile: `CLUSTER_SIZE_SMALL`, `CLUSTER_SIZE_MEDIUM` (default), or `CLUSTER_SIZE_LARGE`. See [Agent sizing](#agent-sizing). | | `akuityRepoServerReplicas` | No | Override the `argocd-repo-server` replica count. See [Agent sizing](#agent-sizing). | | `akuityRepoServerMemory` | No | Override `argocd-repo-server` memory. Sets both `requests` and `limits`. See [Agent sizing](#agent-sizing). | ## Agent sizing[​](#agent-sizing "Direct link to Agent sizing") Three fields in the connector Secret control how the Akuity agent is provisioned inside each registered cluster. ### `akuityAgentSize`[​](#akuityagentsize "Direct link to akuityagentsize") This field is sent to the Akuity API when the cluster is registered and determines the overall resource profile of the agent as defined by Akuity. It is an Akuity-level concept, not a direct Kubernetes resource value — Akuity translates this into the appropriate CPU and memory requests for the agent workloads on its end. | Value | Recommended for | | --------------------- | ----------------------------------------------------------------- | | `CLUSTER_SIZE_SMALL` | Development or low-traffic clusters with few Applications | | `CLUSTER_SIZE_MEDIUM` | General-purpose clusters (default) | | `CLUSTER_SIZE_LARGE` | High-throughput clusters with many Applications or frequent syncs | ### `akuityRepoServerReplicas` and `akuityRepoServerMemory`[​](#akuityreposerverreplicas-and-akuityreposervermemory "Direct link to akuityreposerverreplicas-and-akuityreposervermemory") These fields let you override specific resource settings on the `argocd-repo-server` Deployment inside the cluster, independently of the size profile. Platform applies a kustomization patch to the agent manifest before installing it: * `akuityRepoServerReplicas` sets `spec.replicas` on the `argocd-repo-server` Deployment. * `akuityRepoServerMemory` sets both `resources.requests.memory` and `resources.limits.memory` to the same value on the `argocd-repo-server` container. Accepts standard Kubernetes memory quantity strings, for example `"1Gi"` or `"512Mi"`. Both fields are optional and independent of `akuityAgentSize`. Use them when the size profile alone does not give you the control you need over repo server memory or replica count. ## Step 2: Enable the connector on a cluster[​](#step-2-enable-the-connector-on-a-cluster "Direct link to Step 2: Enable the connector on a cluster") ### On a tenant cluster[​](#on-a-tenant-cluster "Direct link to On a tenant cluster") Add the `integrations.argoCD` block to the tenant cluster's `vcluster.yaml`. The `connector` field references the Secret name from Step 1. ``` integrations: argoCD: enabled: true connector: akuity-prod ``` The connector can also be set directly in the VirtualClusterInstance manifest: ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterInstance metadata: name: app-dev namespace: p-team-a spec: template: metadata: name: vcluster spec: helmRelease: values: | integrations: argoCD: enabled: true connector: akuity-prod ``` When the VirtualClusterInstance reconciles, Platform registers the tenant cluster with Akuity, retrieves the agent manifest, and installs the agent into the cluster. ### On a control plane cluster[​](#on-a-control-plane-cluster "Direct link to On a control plane cluster") To register a control plane cluster with Akuity, add `spec.argoCD` to the Cluster object: ``` apiVersion: management.loft.sh/v1 kind: Cluster metadata: name: my-cluster spec: argoCD: enabled: true connector: akuity-prod ``` Disabling the integration Disabling the integration removes the cluster from Akuity, uninstalls the agent namespace from the cluster, and deletes every `ArgoCDApplication` object Platform generated for it from `deploy.argoCD.applications`, including ones with `target: host`. This is a cascading delete. Argo CD also removes the Kubernetes resources those Applications were managing, not just the Application registration. This applies whether the integration is configured using `vcluster.yaml`, a VirtualClusterInstance manifest, or a Cluster object. Standalone `ArgoCDApplication` objects you created directly, including ones using `spec.destination.cluster`, keep their CR when you disable the integration alone. Once the destination's connector stops resolving, Platform still deletes the standalone object's remote Argo CD Application the same way, cascading to the workloads it managed, and marks the CR's `Synced` condition `False` until the cluster registers again. Re-enabling the integration reconciles it again; you don't have to recreate it. Deleting the tenant cluster itself deletes every `ArgoCDApplication` targeting it, whether standalone or not, along with the workloads those applications managed. The Platform UI keeps these two settings in order. The guided `vcluster.yaml` editor only shows the Applications section once the integration is enabled with a connector, and disabling the integration through the UI prompts you to confirm removing any configured applications, stripping them from `deploy.argoCD.applications` as part of the disable. Editing `vcluster.yaml` directly doesn't enforce that order. Entries in `deploy.argoCD.applications` stay inert until the integration is enabled with a connector, and disabling directly doesn't remove them from the file. They just stop producing `ArgoCDApplication` objects until you re-enable the integration. ## Next step[​](#next-step "Direct link to Next step") With the connector enabled, you can declare Argo CD Applications in your tenant cluster or control plane cluster configuration. See [Deploy applications](https://vcluster.com/docs/platform/integrations/argocd/deploy-applications.md). --- # Connect to Argo CD This guide connects vCluster Platform to a self-hosted Argo CD server. At the end you will have a connector that registers tenant clusters and control plane clusters with Argo CD automatically. For Akuity-managed Argo CD, see [Connect to Akuity](https://vcluster.com/docs/platform/integrations/argocd/connect-akuity.md) instead. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * vCluster Platform running with admin access * A self-hosted Argo CD server reachable from the Platform namespace ### Argo CD API token permissions[​](#argo-cd-api-token-permissions "Direct link to Argo CD API token permissions") The API token you provide in the connector must have the following RBAC permissions. Create a dedicated role in your Argo CD `argocd-rbac-cm` ConfigMap and generate a token for it: ``` policy.csv: | p, role:vcluster-platform, clusters, get, *, allow p, role:vcluster-platform, clusters, create, *, allow p, role:vcluster-platform, clusters, update, *, allow p, role:vcluster-platform, clusters, delete, *, allow p, role:vcluster-platform, applications, get, */*, allow p, role:vcluster-platform, applications, create, */*, allow p, role:vcluster-platform, applications, update, */*, allow p, role:vcluster-platform, applications, delete, */*, allow p, role:vcluster-platform, applications, sync, */*, allow ``` | Resource | Actions | Why | | -------------- | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `clusters` | `get`, `create`, `update`, `delete` | Register and deregister tenant clusters and control plane clusters | | `applications` | `get`, `create`, `update`, `delete`, `sync` | Create and sync Argo CD Applications across all projects (`*/*`). `sync` is required to stop in-progress sync operations when deleting a tenant cluster with installed apps | ## Step 1: Create a connector[​](#step-1-create-a-connector "Direct link to Step 1: Create a connector") * Platform UI * YAML 1. Go to Infrastructure > Connectors and select the Argo CD tab. 2. Click Connect Argo CD. 3. In the Display name field, enter a human-readable name for the connector. The Argo CD Connector ID is auto-generated from the display name and is used to reference this connector from clusters. 4. In the Server URL field, enter the URL of your Argo CD server. 5. If Argo CD is installed in a namespace other than `argocd`, update the Argo CD Namespace field. 6. Select an authentication method and fill in the credentials: * Authenticate with Argo CD API token: paste your token in the Argo CD API Token field. * Authenticate with username / password: fill in Username and Password. 7. Click Connect Argo CD. Argo CD supports two authentication methods. If `token` is set, the controller uses bearer token authentication. Otherwise it falls back to `username` and `password`. Apply the Secret with `kubectl apply`. * Token * Basic auth ``` apiVersion: v1 kind: Secret metadata: name: argocd-main namespace: loft labels: loft.sh/connector-type: argocd type: Opaque stringData: server: "https://argocd.example.com" token: "" namespace: "argocd" insecure: "false" ``` ``` apiVersion: v1 kind: Secret metadata: name: argocd-main namespace: loft labels: loft.sh/connector-type: argocd type: Opaque stringData: server: "https://argocd.example.com" username: "" password: "" namespace: "argocd" insecure: "false" ``` | Field | Required | Description | | ----------- | ------------------------------------------- | --------------------------------------------------------------- | | `server` | Yes | Full URL of the Argo CD API server | | `token` | If set, bearer token authentication is used | Argo CD API bearer token | | `username` | Required when `token` is not set | Argo CD username | | `password` | Required when `token` is not set | Argo CD password | | `namespace` | No | Namespace where Argo CD is installed. Defaults to `argocd` | | `insecure` | No | Set to `"true"` to skip TLS verification. Defaults to `"false"` | | `caData` | No | Base64-encoded CA certificate for custom TLS verification | ## Step 2: Enable the connector on a cluster[​](#step-2-enable-the-connector-on-a-cluster "Direct link to Step 2: Enable the connector on a cluster") ### On a tenant cluster[​](#on-a-tenant-cluster "Direct link to On a tenant cluster") Add the `integrations.argoCD` block to the tenant cluster's `vcluster.yaml`. The `connector` field references the Secret name from [Step 1](#step-1-create-a-connector). ``` integrations: argoCD: enabled: true connector: argocd-main ``` The connector can also be set directly in the VirtualClusterInstance manifest: ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterInstance metadata: name: app-dev namespace: p-team-a spec: template: metadata: name: vcluster helmRelease: values: | integrations: argoCD: enabled: true connector: argocd-main ``` When the VirtualClusterInstance reconciles, Platform registers the tenant cluster with Argo CD using the Platform proxy as the API server endpoint and a scoped access key as the bearer token. ### On a control plane cluster[​](#on-a-control-plane-cluster "Direct link to On a control plane cluster") Connecting a cluster to Platform creates a `Cluster` object for it. Add `spec.argoCD` to that existing object instead of applying a new manifest. Find the object's name: ``` kubectl get clusters.management.loft.sh ``` When Platform runs in the same cluster it manages, that cluster's object is typically named `loft-cluster`. Patch it in place: ``` kubectl patch clusters.management.loft.sh --type merge \ -p '{"spec":{"argoCD":{"enabled":true,"connector":"argocd-main"}}}' ``` If you manage the object through GitOps, add `argoCD` alongside its existing fields instead: ``` apiVersion: management.loft.sh/v1 kind: Cluster metadata: name: my-cluster spec: # existing connection method: spec.local, spec.networkPeer, or spec.config argoCD: enabled: true connector: argocd-main ``` A `Cluster` manifest fails validation unless it targets an existing object or already sets one of `spec.local`, `spec.networkPeer`, or `spec.config`. Applying only `spec.argoCD` on its own returns `spec.config: Required value: spec.config or spec.local or spec.networkPeer is required`. Disabling the integration Disabling the integration removes the cluster from Argo CD and deletes every `ArgoCDApplication` object Platform generated for it from `deploy.argoCD.applications`, including ones with `target: host`. This is a cascading delete. Argo CD also removes the Kubernetes resources those Applications were managing, not just the Application registration. This applies whether the integration is configured using `vcluster.yaml`, a VirtualClusterInstance manifest, or a Cluster object. Standalone `ArgoCDApplication` objects you created directly, including ones using `spec.destination.cluster`, keep their CR when you disable the integration alone. Once the destination's connector stops resolving, Platform still deletes the standalone object's remote Argo CD Application the same way, cascading to the workloads it managed, and marks the CR's `Synced` condition `False` until the cluster registers again. Re-enabling the integration reconciles it again; you don't have to recreate it. Deleting the tenant cluster itself deletes every `ArgoCDApplication` targeting it, whether standalone or not, along with the workloads those applications managed. The Platform UI keeps these two settings in order. The guided `vcluster.yaml` editor only shows the Applications section once the integration is enabled with a connector, and disabling the integration through the UI prompts you to confirm removing any configured applications, stripping them from `deploy.argoCD.applications` as part of the disable. Editing `vcluster.yaml` directly doesn't enforce that order. Entries in `deploy.argoCD.applications` stay inert until the integration is enabled with a connector, and disabling directly doesn't remove them from the file. They just stop producing `ArgoCDApplication` objects until you re-enable the integration. ## Next step[​](#next-step "Direct link to Next step") With the connector enabled, you can declare Argo CD Applications in your tenant cluster or control plane cluster configuration. See [Deploy applications](https://vcluster.com/docs/platform/integrations/argocd/deploy-applications.md). --- # Feature support and limitations This page covers what the Argo CD integration passes through unmodified, what it doesn't model, operational limitations to plan around, and common patterns for using it at scale. ## Feature support[​](#feature-support "Direct link to Feature support") ### Passed through as a standard Argo CD ApplicationSpec[​](#passed-through-as-a-standard-argo-cd-applicationspec "Direct link to Passed through as a standard Argo CD ApplicationSpec") `ArgoCDApplication.spec.template.spec` and `ArgoCDApplicationTemplate.spec.template.spec` embed a vendored copy of Argo CD's own `ApplicationSpec` type, not a redefined subset. Fields that type supports pass through unmodified, including: * Helm, Kustomize, and directory/Jsonnet sources * Config management plugins * Multiple sources (`sources`) * `syncPolicy`, including `automated.selfHeal` and `automated.prune` * `ignoreDifferences` Platform only touches `spec.destination` (to route to the correct tenant cluster or control plane cluster) and defaults an empty `spec.project` to `default`. Everything else reaches Argo CD as written. Actual compatibility still depends on your connected Argo CD server. The vendored type is pinned to whatever Argo CD API version Platform was built against. A field that type doesn't yet have, or one your Argo CD server's version doesn't support, won't work even if it's otherwise valid syntax. Config management plugins also need matching plugin configuration on the Argo CD server itself. The integration doesn't set that up for you. See the [ArgoCDApplication](https://vcluster.com/docs/platform/api/resources/argocdapplication.md) and [ArgoCDApplicationTemplate](https://vcluster.com/docs/platform/api/resources/argocdapplicationtemplate.md) API references for the full field list. ### Not modeled by the integration[​](#not-modeled-by-the-integration "Direct link to Not modeled by the integration") * **ApplicationSets**: there's no ApplicationSet resource or generator support. Each `ArgoCDApplication` is a single Argo CD Application. [Fan out an application fleet with a tenant cluster template](#fan-out-an-application-fleet-with-a-tenant-cluster-template) below is the closest available substitute for fanning the same Application out across many clusters. * **Cluster-label targeting**: destinations reference a tenant cluster or control plane cluster by name only. There's no label-selector targeting like an ApplicationSet cluster generator provides. The same template-based fan-out is the closest substitute here too, since it groups clusters by the template that created them rather than by label. * **Per-application AppProject guardrails**: see [Limitations](#no-per-application-appproject-guardrails) below. **App-of-apps automation** isn't modeled or automated by the integration either. Platform overwrites `spec.destination.server` and `spec.destination.name` to the tenant cluster or control plane cluster you registered, but leaves `spec.destination.namespace` exactly as you set it. The usual blocker isn't the namespace, since you control that fully. It's the cluster. Argo CD only picks up child Application manifests it finds on the cluster hosting its own controller, and the registered destination is rarely that cluster. One narrow exception exists. First, the control plane cluster your self-hosted Argo CD instance itself runs on needs its own Argo CD connector enabled (see [Connect to Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md#on-a-control-plane-cluster)). With that in place, an `ArgoCDApplication` with `spec.destination.virtualCluster.target: host` and `spec.template.spec.destination.namespace` set to Argo CD's own watched namespace can let a parent Application deploy child Application manifests there. The integration doesn't set this up or manage it either way. ## Limitations[​](#limitations "Direct link to Limitations") ### Self-hosted Argo CD depends on the Platform proxy[​](#self-hosted-argo-cd-depends-on-the-platform-proxy "Direct link to Self-hosted Argo CD depends on the Platform proxy") When you register a cluster with a standard (self-hosted) Argo CD connector, Platform sets the Argo CD destination server to the Platform proxy endpoint, not the tenant cluster's or control plane cluster's real API server. There's no direct or air-gapped path around this for the standard connector. The Platform proxy is Argo CD's only path to that cluster's API. If Platform, or your connectivity to it, is unavailable, Argo CD loses all access to any cluster registered under that connector, not just sync and apply, but live-state comparison, health assessment, pruning, and resource hooks too. Existing workloads keep running during that window. Only Argo CD's view into and control over the cluster pauses, until Platform is reachable again. See [Platform communication and connectivity](https://vcluster.com/docs/platform/understand/platform_communication.md) and [What happens during control plane outages?](https://vcluster.com/docs/platform/understand/control-plane-outages.md) for the full blast-radius model. Akuity isn't affected by this at steady state, but only for cluster-side reconciliation. The Akuity agent installed in the destination cluster runs Argo CD's reconciliation locally, so live-state diffing, health checks, and applying resources happen inside the cluster against its own API server. The agent's outbound connection to Akuity's control plane carries agent-to-control-plane traffic, not the reconciliation traffic itself. Platform itself stays equally involved for both connector types at the application level. It upserts the Argo CD Application, processes refresh and sync requests, and copies status back onto the CR regardless of whether the connector is standard or Akuity. For Akuity specifically, Platform also installs the agent and reapplies its manifest whenever Akuity's registration data or the connector's agent size changes. ### Read `status.application` for real Argo CD state[​](#read-statusapplication-for-real-argo-cd-state "Direct link to read-statusapplication-for-real-argo-cd-state") The `Synced` condition on `ArgoCDApplication.status.conditions` only reflects whether Platform successfully reconciled the Application through the Argo CD API, whether or not that reconciliation actually changed anything. It can show `True` while the application itself is `OutOfSync` or unhealthy in Argo CD. For the application's actual state, read `status.application.sync.status` and `status.application.health.status` instead. These mirror Argo CD's own Application status verbatim. ArgoCDApplication status (abbreviated) ``` status: conditions: - type: Synced # Platform reconciled successfully. Not Argo CD's own sync state. status: "True" application: sync: status: OutOfSync # Argo CD's real sync state health: status: Degraded # Argo CD's real health state ``` ### No per-application AppProject guardrails[​](#no-per-application-appproject-guardrails "Direct link to No per-application AppProject guardrails") The integration doesn't create or manage Argo CD AppProjects on your behalf. Every `ArgoCDApplication` uses Argo CD's `default` project unless `spec.project` names a project you've already created directly in Argo CD. There's no built-in way to scope repo allow-lists, destination allow-lists, sync windows, or project-level RBAC per application. If you need per-project AppProject guardrails managed by Platform, see the [legacy Argo CD integration](https://vcluster.com/docs/platform/integrations/argocd/legacy.md), which creates an AppProject per vCluster Platform project. That integration is deprecated and covers project-level registration only, not individual `ArgoCDApplication` resources. ### The integration deletes managed apps when disabled[​](#the-integration-deletes-managed-apps-when-disabled "Direct link to The integration deletes managed apps when disabled") Disabling the Argo CD integration on a tenant cluster or control plane cluster removes it from Argo CD and deletes every `ArgoCDApplication` object Platform generated for it from `deploy.argoCD.applications`. This is a cascading delete. Argo CD also removes the Kubernetes resources those Applications were managing in the destination cluster, not just the Application registration. Plan for workload deletion, not just deregistration, when you disable the integration. Standalone `ArgoCDApplication` objects you created directly, for example applications targeting a control plane cluster, keep their CR when you disable the integration alone. Once the destination's connector stops resolving, Platform still deletes the standalone object's remote Argo CD Application the same way, cascading to the workloads it managed, and marks the CR's `Synced` condition `False` until the cluster registers again. Deleting the tenant cluster itself is different. That deletes every `ArgoCDApplication` targeting it, regardless of whether it's standalone, along with the workloads those applications managed. See [Disabling the integration](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md#step-2-enable-the-connector-on-a-cluster) for details. The Platform UI keeps enabling and `deploy.argoCD.applications` in order. The guided `vcluster.yaml` editor only shows the Applications section once the integration is enabled with a connector, and disabling the integration through the UI prompts you to confirm removing any configured applications, stripping them from `deploy.argoCD.applications` as part of the disable. Editing `vcluster.yaml` directly doesn't enforce that order. Entries in `deploy.argoCD.applications` stay inert until the integration is enabled with a connector, and disabling directly doesn't remove them from the file, it just stops them from producing `ArgoCDApplication` objects until you re-enable the integration. ## Common patterns[​](#common-patterns "Direct link to Common patterns") ### Fan out an application fleet with a tenant cluster template[​](#fan-out-an-application-fleet-with-a-tenant-cluster-template "Direct link to Fan out an application fleet with a tenant cluster template") Declare `deploy.argoCD.applications` and reference an `ArgoCDApplicationTemplate` from a tenant cluster template instead of an individual tenant cluster's `vcluster.yaml`. Every tenant cluster created from that template gets the same Argo CD Application declaration. Whether a template update then rolls out to existing tenant clusters depends on template versioning. Only tenant clusters using a wildcard version, for example `1.X.X`, pick up a new compatible version automatically. Tenant clusters pinned to an exact version, or created without versioning enabled, need a manual update. See [Deploy applications](https://vcluster.com/docs/platform/integrations/argocd/deploy-applications.md), [Templates](https://vcluster.com/docs/platform/administer/templates/overview.md), and [Versioning templates](https://vcluster.com/docs/platform/administer/templates/versioning.md#template-version-strings). ### Segment across Argo CD instances with multiple connectors[​](#segment-across-argo-cd-instances-with-multiple-connectors "Direct link to Segment across Argo CD instances with multiple connectors") A connector is just a named Secret, and clusters reference connectors by name. Create multiple connectors, for example one per environment tier, business unit, or region, and point different tenant cluster templates at different connectors. This keeps unrelated teams' Applications on separate Argo CD (or Akuity) instances without any additional integration logic. ### Progressive delivery with Argo Rollouts[​](#progressive-delivery-with-argo-rollouts "Direct link to Progressive delivery with Argo Rollouts") The integration doesn't configure progressive delivery for you, but nothing blocks it either. Since `source` is pass-through, point it at manifests or a Helm chart that install Argo Rollouts alongside `Rollout` and `AnalysisTemplate` objects. Rollout mechanics run entirely inside the tenant cluster and the Argo CD Application, independent of the integration itself. --- # Deploy applications This guide covers declaring Argo CD Applications in your Platform configuration after a connector is enabled. Platform reconciles these declarations into `ArgoCDApplication` objects and syncs them to your Argo CD instance. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * A connector is already configured and enabled on the target cluster. See [Connect to Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md) or [Connect to Akuity](https://vcluster.com/docs/platform/integrations/argocd/connect-akuity.md). ## Application templates[​](#application-templates "Direct link to Application templates") An `ArgoCDApplicationTemplate` is a reusable Argo CD ApplicationSpec. Create one when you want to deploy the same application definition across multiple tenant clusters without duplicating configuration. `ArgoCDApplicationTemplate` supports the same parameter definitions (types, validation, defaults) as tenant cluster templates. Both templated and inline `ArgoCDApplication` specs can use the built-in platform-specific values (project, tenant cluster name, and so on) described in [Parameters](https://vcluster.com/docs/platform/administer/templates/parameters.md#argo-cd-application-parameter-values). * Platform UI * YAML 1. Go to Management > Apps and, if shown, click the ArgoCD Apps tab. 2. Click Create Argo CD Template. 3. In the Basics section, enter a Display name for the template. The Argo CD Template ID is auto-generated from the display name and is the identifier you reference in your applications. Optionally add a Description. 4. In the Config Options section, fill in the YAML editor with the Argo CD ApplicationSpec. 5. Click Add Argo CD Template. ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplicationTemplate metadata: name: app-template spec: # Define values that each ArgoCDApplication can supply. parameters: - variable: imageTag label: Image tag type: string defaultValue: latest template: spec: source: repoURL: "https://github.com/acme/app-charts" targetRevision: "main" path: "app" helm: values: |- image: # Render the value supplied by ArgoCDApplication.spec.parameters. tag: "{{ .Values.imageTag }}" platform: # Platform supplies built-in loft values without parameter definitions. cluster: "{{ .Values.loft.cluster }}" project: default syncPolicy: automated: prune: true selfHeal: true ``` If an application fails to sync, see [Argo CD ApplicationTemplate sync failures](https://vcluster.com/docs/platform/troubleshoot/template-sync-failures.md#argo-cd-applicationtemplate-sync-failures). ## Application naming[​](#application-naming "Direct link to Application naming") Every application has two name fields: `name` and `displayName`. `name` is the Kubernetes resource name for the `ArgoCDApplication` object. It must be unique within the project namespace. Platform checks explicitly defined names against existing `ArgoCDApplication` resources in the same project namespace. A name already owned by another tenant cluster is rejected. `displayName` is the label shown in the Platform UI. `displayName` defined in `vcluster.yaml` must be unique across all the applications defined in the `vcluster.yaml`. ### Name modes[​](#name-modes "Direct link to Name modes") * **Set `name` explicitly:** Platform uses it as the Kubernetes resource name. If `displayName` is not set, it defaults to the value of `name`. * **Set `displayName` only:** Omit `name` and specify only `displayName`. Platform automatically generates a unique Kubernetes resource name. * **Set neither field:** When defined in `vcluster.yaml`, Platform rejects the configuration with a validation error. When creating an application through the UI or Management API, Platform automatically generates a resource name if one is not provided. Argo CD name uniqueness Argo CD requires application names to be unique across all projects. Platform satisfies this by appending the project name when registering the application in Argo CD. ## Deploy to a tenant cluster[​](#deploy-to-a-tenant-cluster "Direct link to Deploy to a tenant cluster") Applications deployed to a tenant cluster are declared in the cluster's `vcluster.yaml` under `deploy.argoCD.applications`. Each entry can target: * **`vcluster`** (default): the tenant cluster itself * **`host`**: the control plane cluster the tenant cluster runs on (see [Deploy to the control plane cluster](#deploy-to-the-control-plane-cluster)) - Platform UI - YAML * New tenant cluster * Existing tenant cluster 1. Select your project from the project selector at the top left. 2. Navigate to Tenant Clusters and click Create Tenant Cluster. 3. Click `Continue Without a Template`, select the wanted type of tenant cluster then click Continue to Configuration. Optionally select a cluster template, then configure it. The template must have been created to allow ArgoCD Applications configuration. 4. Select the Argo CD Integration tab in the cluster creation form. 5. Click on the slider `Enable Argo CD Integration` 6. Select a `Connector` in the dropdown list of connectors or click `Configure Argo CD Connectors` to configure a connector. 7. Click Add Argo CD Application. 8. In the Basics section, enter a Display Name for the application. The Argo CD Application ID is auto-generated and uniquely identifies the application in Argo CD. 9. Choose where to deploy: * Select Deploy to tenant cluster to deploy into the tenant cluster itself. * Select Deploy to control plane cluster to deploy into the control plane cluster the tenant cluster runs on. The control plane cluster must already have an Argo CD connector configured. 10. Select a Template or choose Inline application spec from the Argo CD Application Template dropdown. 11. If you choose a Template, fill the Template Parameters if needed. If you choose Inline application spec, fill the Argo CD Application Spec. You can make additional change directly in the YAML editor. 12. Click Save to add the ArgoCD Application to the tenant cluster. 13. Complete the remaining cluster configuration and click Create Tenant Cluster. 1) Select your project from the project selector at the top left. 2) Navigate to Tenant Clusters and open the tenant cluster where you want to deploy an application. 3) Select the ArgoCD Apps tab in the cluster detail view. 4) Click Add Argo CD Application. 5) In the Basics section, enter a Display Name for the application. The Argo CD Application ID is auto-generated and uniquely identifies the application in Argo CD. 6) Choose where to deploy: * Select Deploy to tenant cluster to deploy into the tenant cluster itself. * Select Deploy to control plane cluster to deploy into the control plane cluster the tenant cluster runs on. The control plane cluster must already have an Argo CD connector configured. 7) Select a Template or choose Inline application spec from the Argo CD Application Template dropdown. 8) If you choose a Template, fill the Template Parameters if needed. If you choose Inline application spec, fill the Argo CD Application Spec. You can make additional change directly in the YAML editor. 9) Click Save to add the ArgoCD Application to the tenant cluster. 10) Click Add Argo CD Application. Add the `deploy.argoCD.applications` block to the tenant cluster's `vcluster.yaml`. Set `target: vcluster` to deploy into the tenant cluster (this is the default) or `target: host` to deploy into the control plane cluster it runs on. * Template reference * Inline spec ``` deploy: argoCD: applications: - name: myapp # omit this to generate the name or ensure that the defined name is unique across the project displayName: "My Application" target: vcluster # or "host" to target the control plane cluster template: name: app-template # references an ArgoCDApplicationTemplate ``` ``` deploy: argoCD: applications: - name: grafana # omit this to generate the name or ensure that the defined name is unique across the project displayName: "Grafana" target: vcluster # or "host" to target the control plane cluster inline: source: repoURL: "https://github.com/acme/charts" targetRevision: HEAD path: grafana project: default syncPolicy: automated: prune: true selfHeal: true destination: namespace: `default destination` ``` ### Deploy to the control plane cluster[​](#deploy-to-the-control-plane-cluster "Direct link to Deploy to the control plane cluster") Setting `target: host` deploys the application onto the control plane cluster rather than inside the tenant cluster. Use this for shared infrastructure — monitoring agents, logging collectors, or networking components — that belongs on the underlying cluster rather than inside a tenant cluster. Understand the access boundary before using `target: host` `target: host` allows configuration inside a tenant cluster to affect workloads on the control plane cluster. Before enabling this, consider: * **Who controls the tenant cluster configuration?** If tenants can edit `vcluster.yaml` directly, they can deploy arbitrary workloads onto the control plane cluster. Restrict this to platform admins or use cluster templates to limit what values tenants can set. * **The control plane cluster must have a connector.** If the control plane cluster does not have an Argo CD connector configured, applications with `target: host` will fail to deploy. * **Applications deploy into a specific namespace.** The `destination.namespace` in the argoCDApplication field determines where the application lands on the control plane cluster. Ensure that deploying in that namespace wouldn't cause any disruption on the control plane cluster. This is a platform-admin-level capability. Do not expose it to end users without proper guardrails. Prerequisite for `target: host` The control plane cluster the tenant cluster runs on must already be registered with an Argo CD connector. Complete the connector setup for the control plane cluster first (see [Connect to Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md#on-a-control-plane-cluster) or [Connect to Akuity](https://vcluster.com/docs/platform/integrations/argocd/connect-akuity.md#on-a-control-plane-cluster)). ## Deploy to a control plane cluster[​](#deploy-to-a-control-plane-cluster "Direct link to Deploy to a control plane cluster") Applications deployed directly onto a control plane cluster are created as `ArgoCDApplication` objects in the project namespace. This is the recommended pattern for shared infrastructure that should live on the underlying cluster rather than inside a tenant cluster. * Platform UI * YAML 1. Go to Infrastructure > Control Plane Clusters. 2. Click the name of the control plane cluster to open its detail page. 3. Select the ArgoCD Apps tab in the cluster header. 4. Click Add Argo CD Application. 5. In the Basics section, enter a Display Name for the application. The Argo CD Application ID is auto-generated from the display name. 6. Select a Template or choose Inline application spec from the Argo CD Application Template dropdown. 7. If you choose a Template, fill the Template Parameters if needed. If you choose Inline application spec, fill the Argo CD Application Spec. You can make additional change directly in the YAML editor. 8. Click Create Argo CD Application. Create an `ArgoCDApplication` object in the project namespace. Use `spec.destination.cluster` to target the control plane cluster by name. * Template reference * Inline spec ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplication metadata: name: shared-infra namespace: p-my-project # replace with your project namespace spec: displayName: "Shared Infrastructure" destination: cluster: name: loft-cluster # replace with your control plane cluster name templateRef: name: app-template # references the ArgoCDApplicationTemplate defined above parameters: imageTag: "1.2.3" # supplies the template's imageTag parameter ``` ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplication metadata: name: cluster-monitoring namespace: p-my-project # replace with your project namespace spec: displayName: "Cluster Monitoring" destination: cluster: name: loft-cluster # replace with your control plane cluster name template: spec: source: repoURL: "https://github.com/acme/charts" targetRevision: HEAD path: monitoring project: default syncPolicy: automated: prune: true selfHeal: true ``` ## Delete applications[​](#delete-applications "Direct link to Delete applications") Deleting an `ArgoCDApplication` object directly always cascades. Platform deletes the corresponding Argo CD `Application` with cascading delete enabled, which prunes the workloads it manages from the target cluster. This isn't currently configurable. Deleting a tenant cluster follows a separate cleanup path. Platform always deletes the `ArgoCDApplication` object, but whether the underlying Argo CD `Application` cascades depends on the connector and destination: * With the Akuity connector, Platform always requests cascading deletion, regardless of destination. * With a self-hosted Argo CD connector and `target: vcluster` (the default), Platform skips cascading delete. Argo CD doesn't prune the workloads, so they're orphaned in the tenant cluster unless its own teardown removes them first. * With a self-hosted Argo CD connector and `target: host`, Platform requests cascading deletion, since the control plane cluster, and the workloads on it, outlives the tenant cluster. The generated Argo CD `Application` doesn't carry the `resources-finalizer.argocd.argoproj.io` finalizer, and there's no way to add it. `spec.template.metadata` on `ArgoCDApplication` and `ArgoCDApplicationTemplate`, the field that populates the generated `Application` object's metadata, only supports labels and annotations. Deleting an Application outside Platform orphans workloads Because the generated `Application` has no finalizer, Argo CD doesn't block on cleanup if it's removed outside Platform. This can happen through external automation or a non-cascade delete from the Argo CD UI. The workloads it deployed are orphaned in the target cluster instead of being pruned. Delete the `ArgoCDApplication` object through Platform instead so cleanup runs as expected. Disabling the integration Disabling `integrations.argoCD.enabled` on a tenant cluster, or removing `spec.argoCD` from a control plane cluster, removes the cluster from Argo CD and deletes every `ArgoCDApplication` object Platform generated for it from `deploy.argoCD.applications`, including ones with `target: host`. This is a cascading delete. Argo CD also removes the Kubernetes resources those Applications were managing, not just the Application registration. Standalone `ArgoCDApplication` objects you created directly, like the one in [Deploy to a control plane cluster](#deploy-to-a-control-plane-cluster) above, keep their CR when you disable the integration alone. Once the destination's connector stops resolving, Platform still deletes the standalone object's remote Argo CD Application the same way, cascading to the workloads it managed, and marks the CR's `Synced` condition `False` until the cluster registers again. Re-enabling the integration reconciles it again; you don't have to recreate it. Deleting the tenant cluster itself deletes every `ArgoCDApplication` targeting it, whether standalone or not, along with the workloads those applications managed. The Platform UI keeps these two settings in order. The guided `vcluster.yaml` editor only shows the Applications section once the integration is enabled with a connector, and disabling the integration through the UI prompts you to confirm removing any configured applications, stripping them from `deploy.argoCD.applications` as part of the disable. Editing `vcluster.yaml` directly doesn't enforce that order. Entries in `deploy.argoCD.applications` stay inert until the integration is enabled with a connector, and disabling directly doesn't remove them from the file. They just stop producing `ArgoCDApplication` objects until you re-enable the integration. --- # Argo CD (legacy) Deprecated This is the legacy project-level Argo CD integration. It is deprecated and will be removed in a future release. For new installations, use the connector-based integration instead: * [Connect to Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md) * [Connect to Akuity](https://vcluster.com/docs/platform/integrations/argocd/connect-akuity.md) | | | Enterprise | | | | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Argo Integration](https://vcluster.com/docs/platform/integrations/argocd.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Multi-region platforms On a [multi-region platform](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md), register the platform as an Argo CD cluster against the platform HTTPS endpoint with a platform access key. The host EKS endpoint isn't a valid target on multi-region deployments. See [Access the management API](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md#access-the-management-api) for a working Argo CD cluster secret example. vCluster Platform provides several points of integration with [Argo CD](https://argo-cd.readthedocs.io/en/stable/), a popular GitOps tool for Kubernetes. The Argo CD integration is designed to help users take advantage of the speed and ease of creating tenant clusters, as well as the power of Argo CD for managing and deploying applications within those tenant clusters. Argo CD Integration has three major components or levels, with each level adding additional integration capabilities. 1. Sync tenant clusters to Argo CD 2. Enable SSO integration (sign in to Argo CD using vCluster Platform) 3. Create Argo CD [App Project](https://argo-cd.readthedocs.io/en/stable/user-guide/projects/) per vCluster Platform Project The simplest level of integration is syncing tenant clusters into Argo CD as available clusters for deployments. In this deployment model, tenant clusters that are part of an Argo enabled project, and have the `loft.sh/import-argocd` label set to `true` will be automatically "synced" to Argo CD. These tenant clusters will show up as clusters that are available for deployments in the Argo CD UI. When the tenant clusters are deleted, vCluster Platform will automatically remove the deleted cluster from Argo CD. The second integration level is enabling SSO integration. This is a project level setting that will configure Argo CD to allow users to authenticate using vCluster Platform. After enabling this setting users who browse to the Argo CD instance will see a button to log in using vCluster Platform. All members of the project will be able to log in using vCluster Platform and gain access to Argo CD. The final integration level is the App Project/Argo CD Project integration. This tier configures an App Project in Argo CD that corresponds to the vCluster Platform project. This option allows for configuring the Argo CD Project's source repositories (repositories that can be deployed to clusters in the given project), as well as RBAC roles to apply to the project. Configuration of each of these integration modes all starts at the vCluster Platform project object. Argo CD can be enabled on a per-project basis, where multiple projects can be configured to integrate with a single Argo CD instance, or with different Argo CD instances depending on your requirements. ## Project Integration[​](#project-integration "Direct link to Project Integration") ### Enable Argo CD integration per project[​](#enable-argo-cd-integration-per-project "Direct link to Enable Argo CD integration per project") Integration Errors If there are any errors establishing Argo CD integration, an `Argo CD Error` message will appear under the 'Project Settings' button. * During Project Creation * On Existing Project 1. Select Create New Project from the project selector dropdown (top left). 2. Create your project as normal, then click the Argo CD configuration tab. 3. Slide the Enable Argo CD Integration slider to enabled. 4. In the new configurations that appeared under the Enable Argo CD Integration slider, select either Clusters or Tenant Clusters from the dropdown box in the Where is Argo CD running dropdown. This tells vCluster Platform if your Argo CD instance lives in a connected physical cluster, or if it is running inside of a tenant cluster. 5. In the next dropdown to the right, select the name of the cluster or tenant cluster that your Argo CD instance is deployed in. 6. If your Argo CD instance is deployed in a namespace other than `argocd`, enter the name of the namespace in the Argo CD Namespace box. 7. Finish configuring anything else you'd like on your project, then click Save Changes. 1) Click Settings for the appropriate project in the project selector dropdown (top left). 2) Click the Argo CD configuration tab. 3) Slide the Enable Argo CD Integration slider to enabled. 4) In the new configurations that appeared under the Enable Argo CD Integration slider, select either Clusters or Tenant Clusters from the dropdown box in the Where is Argo CD running dropdown. This tells vCluster Platform if your Argo CD instance lives in a connected physical cluster, or if it is running inside of a virtual cluster. 5) In the next dropdown to the right, select the name of the cluster or tenant cluster that your Argo CD instance is deployed in. 6) If your Argo CD instance is deployed in a namespace other than `argocd`, enter the name of the namespace in the Argo CD Namespace box. 7) Click the Save Changes button. ### Enable Argo CD SSO integration[​](#enable-argo-cd-sso-integration "Direct link to Enable Argo CD SSO integration") Projects may also enable SSO integration with Argo CD, which allows all members of the vCluster Platform project to log in to Argo CD using vCluster Platform. Pre-Requisites The `loftHost` vCluster Platform configuration must be set to enable this integration. The `loftHost` can be configured using Helm values, or in the `Platform` > `Platform Config` section of the UI on the `Config` pane. * During Project Creation * On Existing Project 1. Select Create New Project from the project selector dropdown (top left). 2. Create your project as normal, then click the Argo CD configuration tab. 3. Slide the Enable Argo CD Integration slider to enabled. 4. In the new configurations that appeared under the Enable Argo CD Integration slider, select either Clusters or Tenant Clusters from the dropdown box in the Where is Argo CD running dropdown. This tells vCluster Platform if your Argo CD instance lives in a connected physical cluster, or if it is running inside of a virtual cluster. 5. In the next dropdown to the right, select the name of the cluster or tenant cluster that your Argo CD instance is deployed in. 6. If your Argo CD instance is deployed in a namespace other than `argocd`, enter the name of the namespace in the Argo CD Namespace box. 7. Slide the Enable SSO Integration slider to enabled. 8. In the `ArgoCD Url` field, enter the resolvable and reachable (from vCluster Platform) URL of your Argo CD instance. 9. **\[Optional]** select the desired RBAC role to assign to project members who log into Argo CD via vCluster Platform. 10. Finish configuring anything else you'd like on your project, then click the Save Changes button. 1) Click Settings for the appropriate project in the project selector dropdown (top left). 2) Click the Argo CD configuration tab. 3) Ensure that standard project Argo CD integration is configured. 4) Slide the Enable SSO Integration slider to enabled. 5) In the `Argo CD Url` field, enter the resolvable and reachable (from vCluster Platform) URL of your Argo CD instance. 6) **\[Optional]** select the desired RBAC role to assign to project members who log into Argo CD via vCluster Platform. 7) Finish configuring anything else you'd like on your project, then click the Save Changes button. ### Enable Argo CD App Project integration[​](#enable-argo-cd-app-project-integration "Direct link to Enable Argo CD App Project integration") App Project integration, can be enabled by editing a project manifest and setting the `argoCD.project.enabled` field of the project spec to 'true'. Additionally, users may provide metadata fields to apply the Argo CD App Project object, Argo CD RBAC roles to apply, and an array of permissible source repositories that may be accessed within the project. Moving tenant clusters between projects Migrating a tenant cluster between projects while Argo CD integration is enabled is not supported. Because Argo CD applications are project-scoped, this operation deletes all Argo CD applications managed by Platform for that cluster and removes their workloads. Applications are not recreated in the destination project. The tenant cluster may also get stuck in `Terminating`. Avoid this migration path when Argo CD integration is active. See [Move a vCluster to another project](https://vcluster.com/docs/platform/api/resources/project/migratevirtualclusterinstance.md) for details. ## Tenant cluster integration[​](#tenant-cluster-integration "Direct link to Tenant cluster integration") ### Enable tenant cluster import[​](#enable-tenant-cluster-import "Direct link to Enable tenant cluster import") Once a project has Argo CD enabled, tenant clusters within that project are eligible to be synced to the Argo instance as available clusters. You can enable tenant cluster import into Argo CD during the tenant cluster creation step, or by enabling this feature on existing tenant clusters. Make Sure Your Project Has Argo CD Enabled. If the project your tenant cluster resides in does *not* have Argo CD integration enabled, you will not see the Argo CD import option! * During Tenant Cluster Creation * On Existing Tenant Cluster 1. Select the project you intend to enable Argo CD from from the projects dropdown at the top of the left navigation bar. 2. Click on Tenant Clusters and the New Tenant Cluster button. 3. In the popup, optionally select the tenant cluster template, then click the Create button. 4. Find your desired tenant cluster in the tenant cluster list. Slide the ArgoCD slider to enabled. 1) Select the project you intend to enable Argo CD from from the projects dropdown at the top of the left navigation bar. 2) Find your desired tenant cluster in the tenant cluster list. Slide the ArgoCD slider to enabled. Integration Errors If there are any errors adding your tenant cluster to Argo CD, the tenant cluster status will show a warning, clicking on the warning will give you a message indicating the cause of the issues. ## Disable the Argo CD Integration[​](#disable-the-argo-cd-integration "Direct link to Disable the Argo CD Integration") The Argo CD integration can be disabled on a per tenant cluster or per project level by toggling the same sliders used to enable it. Disabling the integration at the tenant cluster level simply removes it as a registered cluster in Argo CD. Disabling the integration at the project level removes all tenant clusters from Argo CD, so be careful when disabled Argo at this level. Disabling at the project level deletes all managed applications When Argo CD integration is disabled at the project level, all Argo CD applications managed by Platform for tenant clusters in that project are deleted. The workloads deployed by those applications are also removed. --- # Certified Stacks Certified Stacks are tested [StackTemplates](https://vcluster.com/docs/platform/understand/what-are-stacks.md) and supporting resources bundled with vCluster Platform. They package multi-application integrations that need dependency ordering, health gates, and shared configuration. Bundled StackTemplates carry the certified marker and are read-only. They use the same StackTemplate and StackInstance APIs as custom Stacks. ## Understand the certified catalog[​](#understand-the-certified-catalog "Direct link to Understand the certified catalog") Platform 4.12 includes the NVIDIA Run:ai certified bundle. It contains four StackTemplates, two VirtualClusterTemplates, and the Apps those templates reference. | Integration | Deployment models | Components | | ----------------------------------------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------- | | [NVIDIA Run:ai](https://vcluster.com/docs/platform/integrations/certified-stacks/runai.md) | Dedicated control plane and central control plane | vCluster, NVIDIA GPU components, and NVIDIA Run:ai 2.26 | The bundle is pinned in each Platform release. The upstream [vcluster-stacks repository](https://github.com/loft-sh/vcluster-stacks) contains its source manifests, tests, and exhaustive implementation guidance. The same repository also hosts Community Stacks from customers, partners, and the community. Platform doesn't bundle them. ## Use a certified stack[​](#use-a-certified-stack "Direct link to Use a certified stack") Choose a deployment model on the integration page. Then use the administrator guide to validate prerequisites and follow the supported lifecycle: * [Choose an NVIDIA Run:ai model](https://vcluster.com/docs/platform/integrations/certified-stacks/runai.md). * [Manage Certified Stacks](https://vcluster.com/docs/platform/administer/templates/certified-stacks.md). * [Troubleshoot Stacks](https://vcluster.com/docs/platform/troubleshoot/stacks.md). Certified Stack orchestration doesn't have a separate license feature. Each task uses the feature for the resource it creates. The bundled NVIDIA Run:ai templates contain App tasks, and [Apps](https://vcluster.com/docs/platform/understand/what-are-apps.md) are available on every plan. ## Customize a certified stack[​](#customize-a-certified-stack "Direct link to Customize a certified stack") Certified StackTemplates are read-only. To change one, create your own StackTemplate from the certified stack's YAML. Give it a different name, and remove the `vcluster.com/certified` annotation. Your template is a separate resource and doesn't receive changes from later certified bundles. Keep the bundled object unchanged so its source and support status remain clear. Review task dependencies, fixed names, chart versions, and output namespaces together when modifying your template. For the steps, see [Customize a certified stack](https://vcluster.com/docs/platform/administer/templates/certified-stacks.md#customize-a-certified-stack). ## Understand the previous workflow[​](#understand-the-previous-workflow "Direct link to Understand the previous workflow") Earlier Certified Stacks are repository-based reference implementations. They can include Terraform, Helm, and other orchestration outside the Stack API. Those implementations remain available in the repository for users who rely on them. The native Platform 4.12 bundle is a separate delivery model. It seeds StackTemplates directly into Platform and lets users install StackInstances through the UI or management API. --- # NVIDIA Run:ai [NVIDIA Run:ai](https://www.run.ai/) schedules and manages artificial intelligence and machine learning workloads across Kubernetes clusters. vCluster provides isolated Kubernetes APIs and flexible GPU-node models for NVIDIA Run:ai tenants. Platform 4.12 bundles native StackTemplates for NVIDIA Run:ai. The templates coordinate registry setup, ingress, TLS, NVIDIA GPU components, the NVIDIA Run:ai control plane, tenant registration, and cluster agents. ## Review verified versions[​](#review-verified-versions "Direct link to Review verified versions") | Component | Version | | ----------------- | ------- | | vCluster Platform | 4.12 | | vCluster | 0.37 | | Kubernetes | 1.34 | | NVIDIA Run:ai | 2.26 | The bundled Apps pin their chart versions. Review the certified bundle for the exact component versions in a Platform patch release. ## Choose a deployment model[​](#choose-a-deployment-model "Direct link to Choose a deployment model") | Model | Control plane and GPU components | Tenant lifecycle | Use case | | ----------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------- | | Dedicated control plane | Each tenant cluster runs its own NVIDIA Run:ai control plane and GPU Operator. | Deleting the tenant removes its dedicated StackInstance. | Independent NVIDIA Run:ai and GPU Operator lifecycles per tenant. | | Central control plane | One control plane cluster runs a shared NVIDIA Run:ai control plane and GPU Operator. | Each tenant has a separate registration and in-cluster agent Stack. | Trusted internal tenants sharing nodes and one control-plane lifecycle. | Shared nodes are for trusted tenants Shared nodes are a supported, common model for trusted tenants. They give each tenant control-plane, API, and namespace isolation, but tenant workloads share the same kernel and physical nodes. They aren't a security boundary for untrusted tenants with Kubernetes access or arbitrary workload execution. Good fits are development, testing, CI/CD, and internal engineering teams. Use [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for external, resale, regulated, or otherwise untrusted tenant offerings, optionally with [vNode](https://www.vnode.com/docs/) for runtime isolation. NetworkPolicy is an added isolation layer worth enabling even for trusted tenants. vCluster can create the policies for you through `policies.networkPolicy`, and your control plane cluster's CNI enforces them. Confirm your CNI supports enforcement, since some accept NetworkPolicy resources without acting on them. See the [security baseline](https://vcluster.com/docs/vcluster/security/.md). If you accept shared nodes for a trusted-tenant use case, follow the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) as the required next step. Both bundled tenant templates use shared control plane cluster nodes by default. The dedicated model separates the NVIDIA Run:ai and GPU Operator lifecycle, but doesn't create private worker nodes. The central model uses disjoint node labels to control which GPU nodes each tenant can see. Labels are scheduling policy, not a runtime security boundary. ## Understand dedicated control-plane tenancy[​](#understand-dedicated-control-plane-tenancy "Direct link to Understand dedicated control-plane tenancy") Use the `runai-tenant` VirtualClusterTemplate. It creates a tenant cluster and declares a [`deploy.stacks`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/deploy.md#platform-stacks) entry named `runai`. This entry references the `run-ai-dedicated-control-plane` StackTemplate. Platform generates the StackInstance name from the tenant cluster name and Stack configuration name. This model installs ingress-nginx, the NVIDIA GPU Operator, the NVIDIA Run:ai control plane, and the tenant components inside one tenant-cluster lifecycle. The Stack captures the ingress address and registration credentials as task outputs. Choose this model when each tenant needs an independent NVIDIA Run:ai and GPU Operator lifecycle. It consumes more resources because every tenant runs control-plane and GPU-management components. ## Understand central control-plane tenancy[​](#understand-central-control-plane-tenancy "Direct link to Understand central control-plane tenancy") The central model separates three lifecycles: 1. `run-ai-central-control-plane-host` installs shared infrastructure once on the control plane cluster. 2. `run-ai-central-control-plane-registration` registers and deregisters one tenant. 3. `run-ai-central-control-plane` installs the agent and tenant components inside that tenant cluster. The `runai-tenant-central-control-plane` VirtualClusterTemplate creates the registration and tenant StackInstances. Secrets carry registration facts, registry credentials, and certificate trust into the tenant cluster. Each tenant needs a unique, persistent node-label value. A tenant with no matching nodes sees no GPU capacity. Overlapping tenant labels can let independent schedulers allocate the same GPU. The central model doesn't provide per-tenant GPU utilization views. DCGM Exporter runs on the control plane cluster, outside each tenant API. GPU allocation still uses the capacity on synced Node objects. ## Plan TLS and ingress[​](#plan-tls-and-ingress "Direct link to Plan TLS and ingress") The dedicated template can derive a `nip.io` hostname after its ingress LoadBalancer receives an IPv4 address. The central host template reuses an existing ingress-nginx LoadBalancer. Both models default to a self-signed certificate authority. Use a certificate for a domain you control in production. The central host and tenant Stack must agree about whether a private certificate authority is required. ## Get started[​](#get-started "Direct link to Get started") * [Manage Certified Stacks](https://vcluster.com/docs/platform/administer/templates/certified-stacks.md) for prerequisites, installation, verification, and removal. * [Troubleshoot Stacks](https://vcluster.com/docs/platform/troubleshoot/stacks.md#troubleshoot-nvidia-runai-certified-stacks) for registration, GPU-node, TLS, and cleanup failures. * Read the upstream [dedicated control-plane](https://github.com/loft-sh/vcluster-stacks/blob/main/certified-stacks/run-ai/dedicated-control-plane/README.md) or [central control-plane](https://github.com/loft-sh/vcluster-stacks/blob/main/certified-stacks/run-ai/central-control-plane/README.md) implementation guide. --- # GitLab CI When using vCluster Platform with GitLab you can use the official image `ghcr.io/loft-sh/vcluster-cli` as either a base image or directly. If additional tooling is needed for your CI/CD process, a custom image can be created. ## Tenant clusters for merge requests[​](#tenant-clusters-for-merge-requests "Direct link to Tenant clusters for merge requests") This example shows how to create and delete a Tenant Cluster for running end-to-end tests for the default branch and merge requests. It assumes you have configured CI/CD variables `VCLUSTER_PRO_URL`, `VCLUSTER_PRO_ACCESS_KEY`, and `VCLUSTER_PRO_PROJECT`. note The `script:` section below demonstrates what usage with a custom image would look like, as the base image does not contain the `./kubernetes` directory with resources to apply. Those would be provided by you to accommodate you particular testing requirements. ``` image: ghcr.io/loft-sh/vcluster-cli stages: - test e2e: rules: - if: $CI_MERGE_REQUEST_IID - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH stage: test before_script: # Add --namespace to create a space for the e2e tests, see the tip below. - vcluster platform login $VCLUSTER_PRO_URL --access-key $VCLUSTER_PRO_ACCESS_KEY - vcluster platform create vcluster "${CI_PROJECT_NAME}-${CI_COMMIT_SHORT_SHA}-${CI_PIPELINE_ID}" --project $VCLUSTER_PRO_PROJECT - apk add go make script: - kubectl apply -Rf ./kubernetes - kubectl rollout status deployments/my-app - make e2e after_script: - vcluster delete "${CI_PROJECT_NAME}-${CI_COMMIT_SHORT_SHA}-${CI_PIPELINE_ID}" --delete-namespace --project $VCLUSTER_PRO_PROJECT ``` tip To create a space (that is, a vCluster Platform-managed namespace) for your e2e test, add `vcluster platform create namespace ` to the `before_script` section, or add the `--namespace` flag to the `vcluster platform create` command. **Explanation:** 1. The `ghcr.io/loft-sh/vcluster-cli` image is used for all pipeline jobs and provides the `vcluster` CLI, the `devspace` CLI, `helm`, and `kubectl`. 2. The `before_script` first logs in to vCluster Platform using the `$VCLUSTER_PRO_URL` and `$VCLUSTER_PRO_ACCESS_KEY` variables that you defined in GitLab. [See the GitLab docs for more information](https://docs.gitlab.com/ee/ci/variables/) 3. The `before_script` then creates a tenant cluster using [predefined GitLab variables](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html) to create a unique name. Optionally, the `--upgrade` flag can be used to reuse an existing tenant cluster instead of creating a new one. 4. Next `before_script` installs some additional tooling needed to run the end-to-end tests. For more complex scenarios creating a custom image is recommended. 5. Then the `script` section uses `kubectl` to deploy the application to the space and waits for the `my-app` deployment to become ready. `make` is then used to run an end-to-end test suite. 6. Finally `after_script` deletes the tenant cluster, and passes `--delete-namespace` to ensure the corresponding namespace for the tenant cluster is deleted. By using `after_script` we can ensure the tenant cluster is deleted even if the tests fail. You may wish to skip this step if reusing the tenant cluster with the `--upgrade` flag when creating the tenant cluster. --- # HashiCorp Vault Secret Sync | | | Enterprise | | | | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [HashiCorp Vault Integration](https://vcluster.com/docs/platform/integrations/hashicorp-vault.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). vCluster Platform supports syncing secrets from a HashiCorp Vault instance into both project secrets and global secrets. This feature allows you to use secrets from Vault in your projects, virtual clusters and global vCluster Platform tenant without copying them manually or using third-party tools. This guide will walk you through setting up global and project-based Vault integrations with vCluster Platform, syncing Vault secrets to global and project secrets, and storing the Vault token in a Kubernetes secret for access by vCluster Platform. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you begin, make sure you have the following: * A running vCluster Platform instance * A running HashiCorp Vault instance * Vault token with appropriate permissions * Optional: admin access to the vCluster Platform instance, if you intend to set up a global secret sync ### Optional: Store the Vault token in a Kubernetes secret[​](#optional-store-the-vault-token-in-a-kubernetes-secret "Direct link to Optional: Store the Vault token in a Kubernetes secret") To store the Vault token in a Kubernetes secret and make it accessible for vCluster Platform, create a Secret resource: ``` apiVersion: v1 kind: Secret metadata: name: secret-containing-token namespace: vcluster-platform type: Opaque data: VAULT_TOKEN: ``` Secret namespace Line 4 of this secret requires one to either specify the namespace of the vCluster Platform installation, if the Vault token will be used for global secrets syncing, or the namespace of the project you want to use the Vault Token in. ## Setup[​](#setup "Direct link to Setup") ### Global Vault integration[​](#global-vault-integration "Direct link to Global Vault integration") You must update the [vCluster Platform configuration](https://vcluster.com/docs/platform/api/resources/config.md) with the Vault object to set up a global Vault integration. 1. Log in to the vCluster Platform UI as admin and navigate to Platform > Platform Config. 2. Edit the vCluster Platform configuration by adding the Vault object with the following properties: * Vault Token from Secret * Inline Vault Token ``` vault: enabled: true address: https://your-vault-instance:8200 auth: tokenSecretRef: name: secret-containing-token key: VAULT_TOKEN ``` ``` vault: enabled: true address: https://your-vault-instance:8200 auth: token: your-vault-token ``` Don't forget to change default values Replace `https://your-vault-instance:8200` with the address of your Vault instance. Replace `secret-containing-token` with the name of the secret containing the Vault token in the loft namespace. For further configuration options, see the [vCluster Platform configuration reference](https://vcluster.com/docs/platform/api/resources/config.md) and the [Vault configuration options](#configuration-options). 3. Save the changes to the configuration using the Apply button and wait for vCluster Platform to restart. ### Project-based Vault integration[​](#project-based-vault-integration "Direct link to Project-based Vault integration") You need to update the [project specification](https://vcluster.com/docs/platform/api/resources/project/.md) with the Vault object to set up a project-based Vault integration. 1. Navigate to Projects and then select the desired project. 2. Edit the project specification by adding the Vault object within the spec section: * Vault Token from Secret * Inline Vault Token ``` vault: enabled: true address: https://your-vault-instance:8200 auth: tokenSecretRef: key: VAULT_TOKEN name: secret-containing-token ``` ``` vault: enabled: true address: https://your-vault-instance:8200 auth: token: your-vault-token ``` Don't forget to change default values Replace `https://your-vault-instance:8200` with the address of your Vault instance. Replace `secret-containing-token` with the name of the secret containing the Vault token in the project's namespace. For further configuration options, see the [Project reference](https://vcluster.com/docs/platform/api/resources/project/.md) and the [Vault configuration options](#configuration-options). 3. Save the changes to the project settings by pressing the Save Changes button. ## Sync Vault secrets[​](#sync-vault-secrets "Direct link to Sync Vault secrets") Once you have set up the Vault integration, you can sync Vault secrets to global and project secrets. For the Vault sync to work, one has to add three annotations to the Global Secret or Project Secret of one's choice: * `loft.sh/vault-kv-version`: Defines the KV version of the Vault secret. Can be either `kv1` or `kv2`. * `loft.sh/vault-kv-mount-path`: Defines the mount path of the Vault secret. * `loft.sh/vault-kv-secret-path`: Defines the path of the Vault secret. ### Global Secrets[​](#global-secrets "Direct link to Global Secrets") To sync Vault secrets to global secrets, one can create a SharedSecret resource using the following example specification: ``` apiVersion: management.loft.sh/v1 kind: SharedSecret metadata: name: my-secret namespace: vcluster-platform annotations: loft.sh/vault-kv-version: kv2 loft.sh/vault-kv-mount-path: secrets loft.sh/vault-kv-secret-path: global_secret spec: access: - users: - admin verbs: - "*" subresources: - "*" ``` note Lines 7 to 9 specify the Vault secret sync annotations, which are required for the sync to work. Adjust the annotations to your Vault secret accordingly. ### Project Secrets[​](#project-secrets "Direct link to Project Secrets") To sync Vault secrets to project secrets, one can create a ProjectSecret resource using the following example specification: ``` apiVersion: management.loft.sh/v1 kind: ProjectSecret metadata: name: my-secret annotations: loft.sh/vault-kv-version: kv2 loft.sh/vault-kv-mount-path: secrets loft.sh/vault-kv-secret-path: my_secret spec: {} ``` note Lines 6 to 8 specify the Vault secret sync annotations, which are required for the sync to work. Adjust the annotations to your Vault secret accordingly. Apply the appropriate YAML configuration using `kubectl` to create or update the secrets. ## Configuration options[​](#configuration-options "Direct link to Configuration options") ## `enabled` boolean [​](#enabled "Direct link to enabled")Enabled indicates if the Vault Integration is enabled for the project -- this knob only enables the syncing of secrets to or from Vault, but does not setup Kubernetes authentication methods or Kubernetes secrets engines for vclusters. ## `address` string [​](#address "Direct link to address")Address defines the address of the Vault instance to use for this project. Will default to the `VAULT_ADDR` environment variable if not provided. ## `skipTLSVerify` boolean [​](#skipTLSVerify "Direct link to skipTLSVerify")SkipTLSVerify defines if TLS verification should be skipped when connecting to Vault. ## `namespace` string [​](#namespace "Direct link to namespace")Namespace defines the namespace to use when storing secrets in Vault. ## `auth` object [​](#auth "Direct link to auth")Auth defines the authentication method to use for this project. ### `token` string [​](#auth-token "Direct link to auth-token")Token defines the token to use for authentication. ### `tokenSecretRef` object [​](#auth-tokenSecretRef "Direct link to auth-tokenSecretRef")TokenSecretRef defines the Kubernetes secret to use for token authentication. Will be used if `token` is not provided.Secret data should contain the `token` key. #### `name` string [​](#auth-tokenSecretRef-name "Direct link to auth-tokenSecretRef-name")Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: TODO: Drop `kubebuilder:default` when controller-gen doesn't need it . #### `key` required string [​](#auth-tokenSecretRef-key "Direct link to auth-tokenSecretRef-key")The key of the secret to select from. Must be a valid secret key. #### `optional` boolean [​](#auth-tokenSecretRef-optional "Direct link to auth-tokenSecretRef-optional")Specify whether the Secret or its key must be defined ## `syncInterval` string [​](#syncInterval "Direct link to syncInterval")SyncInterval defines the interval at which to sync secrets from Vault. Defaults to `1m.` See for supported formats. --- # Using Karpenter with vCluster Platform vCluster is compatible with Karpenter and vCluster's features can be used in conjuntion with Karpenter to further reduce infrastructure costs. This article will explain how to [Create Designated Node Pool for a vCluster's Workload Pods](#create-designated-node-pool-for-a-vclusters-workload-pods), [Create Designated Node Pool for a Project](#create-designated-node-pool-for-a-project), and [Target Node Pool Based on vCluster Pod Labels](#target-node-pool-based-on-vcluster-pod-labels). All three of which can be powerful cost saving tools. This article assumes you are using a control plane cluster provisioned on one of the following managed Kubernetes services: EKS (Elastic Kubernetes Service) or AKS (Azure Kubernetes Service). The pattern `` repeatedly occurs in this documentation and is meant to imply that the user should replace the value, angular brackets included, with their own. Much of this article borrows and builds upon the [Getting Started with Karpenter](https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/) documentation and the [Karpenter Provider Azure README.md](https://github.com/Azure/karpenter-provider-azure). Most if not all scripts and yaml manifests were written by the others of those articles or articles of other AKS and Azure documentation. Some of those works were then modified while others remain the same. ### Pre-requisites[​](#pre-requisites "Direct link to Pre-requisites") 1. Either an [EKS Kubernetes cluster](https://karpenter.sh/docs/getting-started/getting-started-with-karpenter) or an [AKS Kubernetes cluster](https://github.com/Azure/karpenter-provider-azure). 2. vCluster Platform installed on control plane cluster. Read [Deploy the vCluster Platform docs](https://www.vcluster.com/docs/platform/install/quick-start-guide#2-deploy-the-vcluster-platform). 3. EKS specific requirements for vCluster Platform compatibility: 1. EBS driver installed. [Get the Amazon EBS CSI driver add-on](https://docs.aws.amazon.com/eks/latest/userguide/managing-ebs-csi.html#adding-ebs-csi-eks-add-on). 2. Delete the gp2 StorageClass with `kubectl delete storageclass gp2` and replace with gp3 by applying following yaml: ``` apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gp3 annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: ebs.csi.aws.com volumeBindingMode: WaitForFirstConsumer parameters: type: gp3 allowVolumeExpansion: true ``` ## Install Karpenter[​](#install-karpenter "Direct link to Install Karpenter") * EKS RBAC Requirements * AKS RBAC Requirements 1. Run helm install for karpenter: ``` helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ --set "settings.clusterName=" \ --set "settings.interruptionQueue=" \ --set controller.resources.requests.cpu=1 \ --set controller.resources.requests.memory=1Gi \ --set controller.resources.limits.cpu=1 \ --set controller.resources.limits.memory=1Gi \ --wait ``` 1. Enable OIDC issuer for your cluster if it is not enabled yet. ``` az aks update \ --resource-group \ --name \ --enable-oidc-issuer ``` 2. Enable OIDC workload identity for your cluster if it is not enabled yet. ``` az aks update \ --resource-group \ --name \ --enable-workload-identity ``` 3. Set local AKS credentials. ``` az aks get-credentials --name "" --resource-group "resource-group-name" --overwrite-existing ``` 4. Create managed identity for karpenter. ``` KMSI_JSON=$(az identity create --name karpentermsi --resource-group "" --location "") ``` 5. Get the OIDC issuer URL. ``` export ISSUER_URL=$(az aks show --name --resource-group --query "oidcIssuerProfile.issuerUrl" -o tsv) ``` 6. You will then need to create a credential linked to the identity and a service account using the OIDC issuer URL. ``` az identity federated-credential create --name KARPENTER_FID --identity-name --resource-group "" \ --issuer "${ISSUER_URL}" \ --subject system:serviceaccount:kube-system:karpenter-sa \ --audience api://AzureADTokenExchange ``` 7. Create the role assignments that Karpenter needs. ``` KARPENTER_USER_ASSIGNED_CLIENT_ID=$(jq -r '.principalId' <<< "$KMSI_JSON") RG_MC=$(jq -r ".nodeResourceGroup" <<< "$AKS_JSON") RG_MC_RES=$(az group show --name "${RG_MC}" --query "id" -otsv) for role in "Virtual Machine Contributor" "Network Contributor" "Managed Identity Operator"; do az role assignment create --assignee "${KARPENTER_USER_ASSIGNED_CLIENT_ID}" --scope "${RG_MC_RES}" --role "$role" done ``` 8. Generate values file needed to deploy Karpenter in AKS cluster. ``` curl -sO https://raw.githubusercontent.com/Azure/karpenter-provider-azure/main/hack/deploy/configure-values.sh chmod +x ./configure-values.sh && ./configure-values.sh karpenter-sa karpentermsi ``` 9. Finally, deploy Karpenter in your AKS cluster using helm upgrade: ``` export KARPENTER_NAMESPACE=kube-system export KARPENTER_VERSION=0-f83fadf2c99ffc2b7429cb40a316fcefc0c4752a helm upgrade --install karpenter oci://ksnap.azurecr.io/karpenter/snapshot/karpenter \ --version "${KARPENTER_VERSION}" \ --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ --values configure-values.yaml \ --set controller.resources.requests.cpu=1 \ --set controller.resources.requests.memory=1Gi \ --set controller.resources.limits.cpu=1 \ --set controller.resources.limits.memory=1Gi \ --wait kubectl logs -f -n "kube-system" -l app.kubernetes.io/name=karpenter -c controller ``` ## Configure control plane cluster Kubernetes RBAC permissions[​](#configure-control-plane-cluster-kubernetes-rbac-permissions "Direct link to Configure control plane cluster Kubernetes RBAC permissions") To deploy the manifests presented in the remainder of this article you will need certain permissions in the Kubernetes control plane cluster. The following Kubernetes permissions will be required in the control plane cluster for the user: * EKS RBAC Requirements * AKS RBAC Requirements - Create `NodePools.karpenter.sh/v1beta1` in control plane cluster. - Create `EC2NodeClass.karpenter.sh/v1beta1` in control plane cluster. * Create `NodePools.karpenter.sh/v1beta1` in control plane cluster. * Create `AKSNodeClass.karpenter.sh/v1beta1` in control plane cluster. ## Create Node Pools[​](#create-node-pools "Direct link to Create Node Pools") ### Create Designated Node Pool for a vCluster's Workload Pods[​](#create-designated-node-pool-for-a-vclusters-workload-pods "Direct link to Create Designated Node Pool for a vCluster's Workload Pods") #### vCluster Platform Permissions The following permissions will be required in vCluster Platform: * Create tenant clusters #### Create Node Pool This section utilizes taints and tolerations to guide tenant cluster pods to the desired node. This works best if every node pool or node utilizes a taint. Otherwise, the pods may be scheduled on non-tainted nodes. 1. Select "Create Tenant Cluster" 2. Select "Configure" and add a toleration to tenant cluster pods using the yaml editor. If the below yaml is out of date then see the [applying tolerations docs](https://www.vcluster.com/docs/v0.19/architecture/scheduling#automatically-applying-tolerations-to-all-pods-synced-by-vcluster). ``` sync: toHost: pods: enforceTolerations: - vclusterID=:NoSchedule ``` 3. Create a node pool with a taint matching the toleration and create node class for it to use. * Provision Node Pool for EKS * Provision Node Pool for AKS ``` cat < spec: template: spec: taints: - key: vclusterID value: effect: NoSchedule requirements: - key: kubernetes.io/arch operator: In values: ["amd64"] - key: kubernetes.io/os operator: In values: ["linux"] - key: karpenter.sh/capacity-type operator: In values: ["spot"] - key: karpenter.k8s.aws/instance-category operator: In values: ["c", "m", "r"] - key: karpenter.k8s.aws/instance-generation operator: Gt values: ["2"] nodeClassRef: apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass name: limits: cpu: 100 disruption: consolidationPolicy: WhenUnderutilized expireAfter: Never --- apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass metadata: name: spec: amiFamily: AL2 # Amazon Linux 2 role: "KarpenterNodeRole-" subnetSelectorTerms: - tags: karpenter.sh/discovery: "" securityGroupSelectorTerms: - tags: karpenter.sh/discovery: "" amiSelectorTerms: - id: "" - id: "" EOF ``` ``` cat < spec: template: spec: taints: - key: vclusterID value: effect: NoSchedule requirements: - key: kubernetes.io/arch operator: In values: ["amd64"] - key: kubernetes.io/os operator: In values: ["linux"] - key: karpenter.sh/capacity-type operator: In values: ["on-demand"] - key: karpenter.azure.com/sku-family operator: In values: [D] nodeClassRef: apiVersion: karpenter.k8s.aws/v1beta1 kind: AKSNodeClass name: limits: cpu: 100 disruption: consolidationPolicy: WhenUnderutilized expireAfter: Never --- apiVersion: karpenter.azure.com/v1alpha2 kind: AKSNodeClass metadata: name: spec: imageFamily: Ubuntu2204 EOF ``` 4. Now we will confirm the node pool is working. Run `kubectl get nodes --watch`. Open a separate terminal tab. Connect to your vCluster by running `vcluster connect `. Then run: ``` cat < Tenant Clusters -> Add Tenant Cluster Template. 2. In the yaml editor box, which contains the vcluster.yaml, add the following: ``` sync: toHost: pods: enforceTolerations: - vclusterID=:NoSchedule ``` 3. Create a node pool with the taint corresponding to the toleration in step 2, e.g.: * Provision NodePool for EKS * Provision NodePool for AKS ``` cat < spec: template: spec: taints: - key: value: "true" effect: NoSchedule requirements: - key: kubernetes.io/arch operator: In values: ["amd64"] - key: kubernetes.io/os operator: In values: ["linux"] - key: karpenter.sh/capacity-type operator: In values: ["spot"] - key: karpenter.k8s.aws/instance-category operator: In values: ["c", "m", "r"] - key: karpenter.k8s.aws/instance-generation operator: Gt values: ["2"] nodeClassRef: apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass name: limits: cpu: 100 disruption: consolidationPolicy: WhenUnderutilized expireAfter: Never --- apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass metadata: name: spec: amiFamily: AL2 # Amazon Linux 2 role: "" # replace with your cluster name subnetSelectorTerms: - tags: karpenter.sh/discovery: "" securityGroupSelectorTerms: - tags: karpenter.sh/discovery: "" amiSelectorTerms: - id: "" - id: "" EOF ``` ``` cat < spec: template: spec: taints: - key: value: "true" effect: NoSchedule requirements: - key: kubernetes.io/arch operator: In values: ["amd64"] - key: kubernetes.io/os operator: In values: ["linux"] - key: karpenter.sh/capacity-type operator: In values: ["on-demand"] - key: karpenter.azure.com/sku-family operator: In values: [D] nodeClassRef: apiVersion: karpenter.k8s.aws/v1beta1 kind: AKSNodeClass name: limits: cpu: 100 disruption: consolidationPolicy: WhenUnderutilized expireAfter: Never --- apiVersion: karpenter.azure.com/v1alpha2 kind: AKSNodeClass metadata: name: spec: imageFamily: Ubuntu2204 EOF ``` 4. Create a new project that only allows templates with set tolerations. Go to project dropdown and create a new project. In the Allowed Templates section, remove the All Templates option. Select the templates that include tolerations for the node pool created in step 3. Now, all pods created in tenant clusters belonging to this project will deploy to the designated node pools. ### Target Node Pool Based on vCluster Pod Labels[​](#target-node-pool-based-on-vcluster-pod-labels "Direct link to Target Node Pool Based on vCluster Pod Labels") #### vCluster Platform Permissions The following permissions will be required in vCluster Platform: * Create tenant clusters #### Create Node Pool 1. Create a node pool with custom labels e.g. `my.org/environ`. * Provision NodePool for EKS * Provision NodePool for AKS ``` cat < limits: cpu: 100 disruption: consolidationPolicy: WhenUnderutilized expireAfter: Never --- apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass metadata: name: spec: amiFamily: AL2 # Amazon Linux 2 role: "" # replace with your cluster name subnetSelectorTerms: - tags: karpenter.sh/discovery: "" securityGroupSelectorTerms: - tags: karpenter.sh/discovery: "" amiSelectorTerms: - id: "" - id: "" EOF ``` ``` cat < limits: cpu: 100 disruption: consolidationPolicy: WhenUnderutilized expireAfter: Never --- apiVersion: karpenter.azure.com/v1alpha2 kind: AKSNodeClass metadata: name: spec: imageFamily: Ubuntu2204 EOF ``` 2. Create a tenant cluster. 3. Add nodeSelector to deployments in your tenant cluster using a nodeSelector that targets the label from step 1 with the desired value. ``` cat < terminationGracePeriodSeconds: 0 containers: - name: inflate image: public.ecr.aws/eks-distro/kubernetes/pause:3.7 resources: requests: cpu: 1 EOF kubectl scale deployment inflate --replicas 5 ``` If you make a deployment with a key/value pair that matches an existing node it can be scheduled to that node. Otherwise, a new node will be provisioned. Since you are adding the key to the node pool but not the value, using different values necessitates different nodes. You can read more about user-defined labels in the [Karpenter's User-Defined Labels docs](https://karpenter.sh/docs/concepts/scheduling/#user-defined-labels). --- # Migrate to the vCluster Rancher operator Built-in Rancher integration removed The built-in Rancher integration was removed in vCluster Platform v4.9.0. This migration guide walks you through moving your existing tenant clusters from the built-in Rancher integration to the [vCluster Rancher operator](https://vcluster.com/docs/vcluster/third-party-integrations/rancher.md), the supported replacement. If you've upgraded to v4.9.0 or later and had tenant clusters with the Rancher integration enabled, those clusters are stuck in `Terminating` when you try to delete them. See the [upgrade guide](https://vcluster.com/docs/platform/integrations/rancher/upgrade-guide.md) to resolve this before proceeding. On vCluster Platform v4.8 or lower? If you're on v4.8 or lower, use the [v4.8 migration guide](https://vcluster.com/docs/platform/integrations/rancher/migration-guide.md), which includes additional steps for disabling the integration at the platform level before cleaning up individual tenant clusters. Alternatively, upgrade to v4.9.0 first and then follow this guide. ## Before you begin[​](#before-you-begin "Direct link to Before you begin") This guide requires `kubectl` access to the control plane cluster with vCluster Platform installed. Step 1 connects to individual tenant clusters using `vcluster connect`. You don't need a direct kubeconfig for each tenant cluster. Work through the steps in order. Step 1 cleans up each tenant cluster. Step 2 removes the corresponding cluster entries from Rancher. Step 3 installs the replacement operator on the control plane cluster. If you upgraded from v4.8 or lower, complete the cleanup section at the end before using the new operator. ## Step 1: Clean up each tenant cluster[​](#step-1-clean-up-each-tenant-cluster "Direct link to Step 1: Clean up each tenant cluster") Complete both sub-steps on one tenant cluster before moving to the next. Repeat for every tenant cluster that had the integration enabled. 1. Connect to the tenant cluster and delete the `cattle-system` namespace. This removes the Rancher cluster agent and all Rancher-related workloads that were injected into the tenant cluster. Modify the following with your specific values to generate a copyable command: NAMEmy-tenant-cluster NAMESPACEp-my-project ``` vcluster connect my-tenant-cluster -n p-my-project kubectl delete namespace cattle-system --ignore-not-found ``` 2. Remove the cluster-level RBAC resources that Rancher created: ``` # List what will be removed — verify before deleting kubectl get clusterrole,clusterrolebinding | grep cattle # Remove them kubectl delete clusterrole,clusterrolebinding \ $(kubectl get clusterrole,clusterrolebinding \ --no-headers -o custom-columns=":metadata.name" \ | grep cattle) \ --ignore-not-found ``` ## Step 2: Delete the Rancher cluster entries[​](#step-2-delete-the-rancher-cluster-entries "Direct link to Step 2: Delete the Rancher cluster entries") Each tenant cluster imported into Rancher has a corresponding cluster entry. Before deleting, verify that the cluster shows as **Unavailable** or **Disconnected** in the Rancher UI. This confirms the agent has been removed and Rancher has lost contact with the cluster. Don't delete the entry if it still shows as **Active**. This may indicate the `cattle-system` namespace wasn't fully removed in the previous step. Once the cluster shows as **Disconnected**, go to **Cluster Management** in the Rancher UI, find each tenant cluster entry, and delete it. If a deletion hangs, Rancher force-removes the entry once it confirms the agent is gone. ## Step 3: Install the vCluster Rancher operator[​](#step-3-install-the-vcluster-rancher-operator "Direct link to Step 3: Install the vCluster Rancher operator") With the old integration fully removed, install the new vCluster Rancher operator on the control plane cluster. The operator runs independently of vCluster Platform and manages tenant cluster registration with Rancher directly. Follow the [installation guide](https://vcluster.com/docs/vcluster/third-party-integrations/rancher.md) for full setup instructions. ## Clean up stale Platform state[​](#clean-up-stale-platform-state "Direct link to Clean up stale Platform state") If you upgraded from v4.8 or lower, your tenant clusters and projects carry stale metadata from the old integration. Complete this cleanup before using the new operator. **Remove the import label from tenant clusters:** ``` kubectl get virtualclusterinstances -A \ -o json | jq -r ' .items[] | select(.metadata.labels["loft.sh/import-rancher"] == "true") | "\(.metadata.namespace) \(.metadata.name)" ' | while read ns name; do kubectl label virtualclusterinstance "$name" -n "$ns" loft.sh/import-rancher- done ``` **Remove stale Rancher conditions from projects:** ``` kubectl get projects.management.loft.sh -o json | jq -r ' .items[] | select( .status.conditions[]? | select(.type == "RancherIntegrationSynced") ) | .metadata.name ' | while read project; do idx=$(kubectl get project.management.loft.sh "$project" -o json | jq ' .status.conditions | to_entries[] | select(.value.type == "RancherIntegrationSynced") | .key') kubectl patch project.management.loft.sh "$project" \ --type=json --subresource=status \ -p="[{\"op\":\"remove\",\"path\":\"/status/conditions/${idx}\"}]" done ``` --- # Upgrade from legacy Rancher integration Legacy integration removed The built-in Rancher integration was removed in vCluster Platform v4.9.0. This page explains what happens when you upgrade an existing deployment and how to recover tenant clusters that become stuck. For the current Rancher integration, see [Install the Rancher integration for vCluster](https://vcluster.com/docs/vcluster/third-party-integrations/rancher.md). ## What changes on upgrade[​](#what-changes-on-upgrade "Direct link to What changes on upgrade") When you upgrade a vCluster Platform instance with the legacy Rancher integration to v4.9.0 or later, the following happens automatically: * **Running tenant clusters are unaffected.** Existing tenant clusters continue running normally. * **Rancher sync no longer runs.** vCluster Platform stops syncing members with Rancher. * **Rancher auth is removed.** Users can no longer log in to vCluster Platform using Rancher SSO. * **Rancher clusters stay active.** The Rancher cluster agent running inside each tenant cluster maintains its own direct connection to Rancher, independent of vCluster Platform. Rancher clusters remain in `active` state after the upgrade. Want to upgrade your tenant clusters to use the new Rancher integration? If you want to use the new operator-based Rancher integration, which lets you create new tenant clusters integrated with Rancher, see the [migration guide](https://vcluster.com/docs/platform/integrations/rancher/migration-guide.md). ## Stuck tenant clusters after deletion[​](#stuck-tenant-clusters-after-deletion "Direct link to Stuck tenant clusters after deletion") Tenant clusters that had the Rancher integration enabled carry a `loft.sh/cleanup-rancher` finalizer on their VirtualClusterInstance object. After upgrading to v4.9.0 or later, nothing processes this finalizer. The controller that handled it no longer exists. This means **any deletion attempt on a Rancher-integrated tenant cluster blocks indefinitely**. The tenant cluster appears in `Terminating` state in the Platform UI. The corresponding Rancher cluster entry shows as disconnected or unavailable. ### Identify affected tenant clusters[​](#identify-affected-tenant-clusters "Direct link to Identify affected tenant clusters") ``` kubectl get virtualclusterinstances -A -o json | jq ' .items[] | select(.metadata.finalizers[]? == "loft.sh/cleanup-rancher") | {name: .metadata.name, namespace: .metadata.namespace, phase: .status.phase} ' ``` ### Unblock a stuck tenant cluster[​](#unblock-a-stuck-tenant-cluster "Direct link to Unblock a stuck tenant cluster") Remove the finalizer manually to allow the deletion to complete: Modify the following with your specific values to generate a copyable command: NAMEmy-tenant-cluster NAMESPACEp-my-project ``` kubectl patch virtualclusterinstance my-tenant-cluster -n p-my-project \ --type=json \ -p='[{"op":"remove","path":"/metadata/finalizers/0"}]' ``` The tenant cluster completes deletion once the finalizer is removed. After the tenant cluster is deleted, the corresponding Rancher cluster entry isn't automatically removed. vCluster Platform no longer communicates with Rancher. Verify the cluster shows as **Unavailable** or **Disconnected** in Rancher before deleting it. After confirming the status in Rancher, go to **Cluster Management** in the Rancher UI, find the corresponding cluster entry, and delete it. If the deletion hangs, Rancher force-removes the entry once it confirms the agent is gone. ## Next steps[​](#next-steps "Direct link to Next steps") If you want to keep your tenant clusters registered in Rancher using the new operator-based integration, see the [migration guide](https://vcluster.com/docs/platform/integrations/rancher/migration-guide.md). --- # Architecture vCluster Platform is the management plane for your tenant cluster fleet. It runs in Kubernetes and exposes UI, CLI, and API entry points. It reconciles Platform resources into connected control plane clusters. Platform is where admins define projects, specify who can create tenant clusters, and set which templates and policies apply. Platform controllers reconcile those decisions into connected clusters, where vCluster runs each tenant cluster control plane. That control plane delivers managed Kubernetes directly, or serves as the foundation for other cluster types like Slurm, Ray, and inference. Platform connects admins and automation to connected clusters and tenant clusters ## Management plane[​](#management-plane "Direct link to Management plane") The management plane runs as the `vcluster-platform` Helm chart, deployed into the `vcluster-platform` namespace. It is the source of truth for all Platform decisions. * **API gateway.** Authenticates all inbound requests and routes them to the right backend. * **Web UI.** Browser interface for the Platform API, running as part of the management API process. * **Management API.** Serves Platform resources: projects, clusters, VirtualClusterInstances, templates, users, and access keys. * **Platform manager.** Controllers that reconcile Platform resources into connected clusters. * **Operational services.** Audit, metrics, cost data, snapshots, integrations, and sleep/wake behavior. Platform uses Kubernetes custom resources for all management state. The default single-replica deployment uses an embedded database. High-availability deployments can connect to an [external relational database](https://vcluster.com/docs/platform/administer/clusters/advanced/external-database/.md) through Kine, which provides an etcd-compatible API over MySQL or PostgreSQL. ## Connected clusters and agents[​](#connected-clusters-and-agents "Direct link to Connected clusters and agents") A connected cluster is a Kubernetes cluster registered with Platform. Tenant cluster control planes and project workloads run there. Connecting a cluster installs the Platform agent, which maintains an outbound connection to Platform and keeps cluster-side state in sync. Platform automatically registers the cluster where it is installed as the local cluster. Smaller deployments can use it for both the management plane and tenant workloads. Platform management plane and connected clusters [Dive deeper into connected clusters →](https://vcluster.com/docs/platform/understand/what-are-clusters.md) ## Projects[​](#projects "Direct link to Projects") A project is the primary policy boundary in Platform. It groups tenant clusters, spaces, templates, secrets, quotas, and integrations, and defines the users and teams who can access them. Each project specifies which connected clusters its members can deploy into, and which templates they can use. [Dive deeper into projects →](https://vcluster.com/docs/platform/understand/what-are-projects.md) ## Access[​](#access "Direct link to Access") Most users enter through the UI or CLI. All requests pass through the API gateway, which authenticates each request and routes it to the right backend. * **Web UI.** Browser interface for managing projects, tenant clusters, users, templates, and connected clusters. * **CLI.** Calls the Platform API for login, create, connect, inspect, and lifecycle commands. * **Automation.** Integrations, GitOps workflows, and scripts call the Platform API directly. The gateway uses Kubernetes impersonation to forward requests. It presents the caller's verified identity to the target API without issuing direct cluster credentials. Platform API gateway request paths [Dive deeper into entry points and proxy paths →](https://vcluster.com/docs/platform/introduction/how-platform-works.md#entry-points) ## Further reading[​](#further-reading "Direct link to Further reading") * [How Platform works](https://vcluster.com/docs/platform/introduction/how-platform-works.md) * [Connected clusters](https://vcluster.com/docs/platform/understand/what-are-clusters.md) * [Projects](https://vcluster.com/docs/platform/understand/what-are-projects.md) * [vCluster architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md) --- # How Platform works Platform follows the Kubernetes controller model. Users and automation create or update API resources expressing desired state. Controllers in the management plane compare that state against connected clusters and reconcile the difference. The connected cluster agent applies changes locally and reports status back to Platform. vCluster then runs the tenant cluster control plane within the connected cluster. That control plane delivers managed Kubernetes directly, or serves as the foundation for other cluster types like Slurm, Ray, and inference running on top. For the structural components, see [Platform architecture](https://vcluster.com/docs/platform/introduction/architecture.md). ## Entry points[​](#entry-points "Direct link to Entry points") All requests enter through the API gateway, which authenticates each request, checks access, applies impersonation, and forwards to the appropriate backend. Users don't receive credentials for connected clusters directly. | Entry point | Used by | Purpose | | ------------------------- | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | Web UI | Browser users | Manage projects, tenant clusters, users, templates, connected clusters, and settings. | | Platform API | UI, CLI, integrations, automation | Manage Platform resources such as projects, clusters, templates, users, and tenant cluster instances. | | CLI | Operators and automation | Calls the Platform API for login, create, connect, inspect, and lifecycle commands. | | Management Kubernetes API | CLI and Kubernetes clients | Proxies authenticated requests to Platform management resources under `/kubernetes/management`. | | Connected cluster proxy | CLI and UI workflows | Proxies authenticated requests to a connected cluster under `/kubernetes/cluster/`. | | Project resource proxy | Tenant cluster and space access | Proxies project-scoped requests under `/kubernetes/project//virtualcluster/` or `/kubernetes/project//space/`. | Platform API gateway request paths ## Resource lifecycle[​](#resource-lifecycle "Direct link to Resource lifecycle") When a user or automation creates or updates a Platform resource, it moves through a consistent path from request to deployed state: 1. A user, CLI command, integration, or GitOps workflow submits a request. Platform creates or updates the API resource expressing desired state. 2. Platform validates the resource against project policy, allowed clusters, allowed templates, and quotas. 3. Platform controllers reconcile the desired state. The connected cluster agent applies changes locally, including namespaces, RBAC, and Helm releases. 4. The connected cluster reports status back to Platform. The resource status is updated and reflected in the UI and API. Tenant resource lifecycle For tenant clusters, the central resource is VirtualClusterInstance. A Platform-managed tenant cluster uses this resource as its desired state. It covers provisioning, configuration updates, upgrades, access, sleep behavior, wake behavior, Argo CD registration, and cleanup. An [externally deployed tenant cluster](https://vcluster.com/docs/platform/understand/externally-vs-platform-deployed.md) can be registered with Platform without transferring lifecycle ownership to Platform. Platform manages supporting resources within a project. NodeProvider and NodeClaim resources handle private node provisioning and lifecycle. These resources follow the same lifecycle pattern as other project resources. ## Tenant cluster runtime[​](#tenant-cluster-runtime "Direct link to Tenant cluster runtime") Platform decides where and how a tenant cluster should be deployed, but vCluster runs the tenant cluster itself. During creation, Platform combines project policy, the selected VirtualClusterTemplate, user-provided parameters, and the selected connected cluster. It then installs or updates the vCluster Helm release in the target namespace. After that point, the vCluster runtime owns the Kubernetes control plane behavior. For the API server, syncer, data store, CoreDNS, networking, and worker node models, see the [vCluster architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md). ## Network paths[​](#network-paths "Direct link to Network paths") Platform supports multiple network paths depending on what is being accessed and how your environment is connected. Network paths to connected cluster and tenant cluster APIs ### Regional and direct access endpoints[​](#regional-and-direct-access-endpoints "Direct link to Regional and direct access endpoints") Platform supports two approaches for bypassing the central API gateway. **Regional cluster endpoints** route requests through the connected cluster agent, avoiding the central Platform gateway while still applying authentication, authorization, and audit locally. **Direct access points** expose the tenant cluster API server endpoint directly in the user's kubeconfig. Platform still controls identity, access, and lifecycle in both cases. Use these paths when round-trip latency through the central gateway is a concern, or when users need a kubeconfig that points directly to their tenant cluster. See [regional cluster endpoints](https://vcluster.com/docs/platform/administer/clusters/advanced/regional-cluster-endpoints.md) for setup. ### Platform proxy[​](#platform-proxy "Direct link to Platform proxy") Platform can proxy Kubernetes API requests through its API gateway. Users authenticate to Platform. Platform checks project and tenant cluster access. The gateway forwards the request to the connected cluster or tenant cluster API server. Use the proxy path when users should not receive connected cluster credentials or when the target cluster is only reachable from Platform or the agent. It centralizes access checks and audit records. ### Tailscale network path[​](#tailscale-network-path "Direct link to Tailscale network path") Platform uses a Tailscale network path for environments where direct connectivity is unavailable or unreliable. Platform coordinates network peers and can relay traffic through DERP when peers can't connect directly. This path is useful when connected clusters, tenant clusters, or users are behind NAT, firewall rules, or private networks. The agent and Platform establish the network relationship. DERP provides a relay path when peer-to-peer traffic can't be established. For connectivity requirements and troubleshooting, see [Platform communication](https://vcluster.com/docs/platform/understand/platform_communication.md) and [tsnet connectivity](https://vcluster.com/docs/platform/troubleshoot/tsnet-connectivity.md). ## Failure behavior[​](#failure-behavior "Direct link to Failure behavior") Platform, connected cluster agents, and tenant cluster control planes fail independently of each other. Existing workloads continue running in connected clusters even when management connectivity is lost. For the full breakdown by layer, see [What happens during control plane outages?](https://vcluster.com/docs/platform/understand/control-plane-outages.md). ## Further reading[​](#further-reading "Direct link to Further reading") * [Platform architecture](https://vcluster.com/docs/platform/introduction/architecture.md) * [Connected clusters](https://vcluster.com/docs/platform/understand/what-are-clusters.md) * [Projects](https://vcluster.com/docs/platform/understand/what-are-projects.md) * [vCluster architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md) * [What happens during control plane outages?](https://vcluster.com/docs/platform/understand/control-plane-outages.md) --- # What is vCluster? vCluster helps AI cloud providers deliver highly scalable, resilient compute clusters as self-service tenant products, including managed Kubernetes, Slurm, inference, Ray, and other cluster types. Each tenant gets a dedicated tenant cluster on your existing infrastructure, with its own API server, RBAC, and CRDs. ## What a tenant cluster looks like[​](#what-a-tenant-cluster-looks-like "Direct link to What a tenant cluster looks like") From the perspective of a user or workload, a tenant cluster is a full Kubernetes cluster. They receive a `kubeconfig` scoped to their cluster and interact with it using tools built for the standard Kubernetes API, such as `kubectl`, Helm, Argo CD, and Crossplane. They cannot see other tenant clusters or the underlying control plane cluster through the Kubernetes API. Network-level isolation between tenants depends on the worker node model you choose. That same Kubernetes foundation is what higher-level cluster types like Slurm, Ray, and inference-serving stacks run on as workloads. From the platform operator's perspective, the tenant cluster control plane runs as a pod in a namespace on the control plane cluster. Creating one requires no additional infrastructure. ## Isolation model[​](#isolation-model "Direct link to Isolation model") Each tenant cluster enforces isolation at the API level. Tenants get their own API server, their own etcd, and their own resource namespace. A resource created inside one tenant cluster has no visibility into another. CRDs, admission webhooks, and RBAC policies scope fully to each tenant and cannot affect other tenants or the host. For workloads that require node-level isolation (GPU infrastructure, regulated industries, or paying external customers), tenant clusters support [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) where each tenant's workloads run on dedicated compute with no cross-tenant visibility at the infrastructure level. ## How Platform manages tenant clusters[​](#how-platform-manages-tenant-clusters "Direct link to How Platform manages tenant clusters") vCluster Platform is the management layer for your tenant cluster fleet. It handles: * **Provisioning** — create tenant clusters from the UI, CLI, or API, with optional templates that enforce configuration baselines * **Access control** — RBAC at the platform, project, and individual cluster level; SSO integration with any OIDC provider * **Lifecycle** — sleep, wake, auto-delete, and version upgrades coordinated across the fleet * **Multi-cluster** — manage tenant clusters across multiple control plane clusters from a single Platform instance [Learn how Platform works →](https://vcluster.com/docs/platform/.md) ## Learn more[​](#learn-more "Direct link to Learn more") For the full architecture, deployment options, and feature reference, see the [vCluster documentation](https://vcluster.com/docs/vcluster/introduction/what-is-vcluster.md). --- # Backup and restore vCluster Platform This guide describes how to backup and restore the vCluster Platform management plane. To back up the connected clusters, reuse a regular Kubernetes backup solution such as [Velero](https://velero.io/). Preserve your license When performing disaster recovery or cluster migrations, ensure you preserve your platform license. See [Preserve license during disaster recovery](https://vcluster.com/docs/platform/maintenance/backup-restore/preserve-license.md) for detailed instructions. See the [Platform CLI hub](https://vcluster.com/docs/platform/reference/platform-cli.md) for UI-to-CLI equivalents across common Platform tasks, including backup and destroy commands. The vCluster Platform saves its state as Kubernetes custom resource (CR) objects. You can back up this state using [Velero](https://velero.io) or the vCluster CLI command we created for convenience. This command saves all relevant Kubernetes resources to a single file. You can run the vCluster CLI to back up these components at any time. The backup includes the following resources: * `ClusterRoleTemplates` * `ClusterAccesses` * `SpaceConstraints` * `Users` * `Teams` * `SharedSecrets` * `AccessKeys` * `Apps` * `SpaceTemplates` * `VirtualClusterTemplates` * `Clusters` and associated Kubernetes Secret objects * `Projects` * `VirtualClusterInstances` * `SpaceInstances` * `ProjectSecrets` The platform backup includes the custom resources listed previously and their associated Kubernetes Secret objects. It does not include other resources used by the platform deployment or connected clusters, such as persistent volumes that store cost data. Use a Kubernetes backup solution such as [Velero](https://velero.io) to back up these resources separately. Back up persistent volumes separately The `vcluster platform backup` command backs up the custom resources and associated Secrets listed previously. You must separately back up the following PersistentVolumeClaims (PVCs): * `platform-db` - stores platform state and configuration * `storage-volume-global-prometheus-server-0` - stores cost metrics data Use [Velero](https://velero.io) or your preferred Kubernetes backup solution to include these PVCs in your backup strategy. To create a backup using the default settings, run the following command. It saves the manifests for all of the components listed previously to a file named `backup.yaml` in the current directory: ``` vcluster platform backup management ``` Alternatively, you cab specify the name for the backup file by running the command as: ``` vcluster platform backup management --filename vcluster-platform-backup.yaml ``` In some cases, the vCluster Platform might have been installed in a different namespace instead of the default `vcluster-platform`. If so, you can specify the correct namespace as shown in the following command: ``` vcluster platform backup management --namespace my-namespace ``` You can also exclude specific components from the backup by passing their names as a comma-separated list. The following example excludes `users` and `teams` from the backup: ``` vcluster platform backup management --skip users,teams ``` ### Restore vCluster Platform from backup[​](#restore-vcluster-platform-from-backup "Direct link to Restore vCluster Platform from backup") The backup file contains the resource manifests for the vCluster Platform management plane. You can apply it like any standard Kubernetes manifest. To restore, apply the backup to your cluster using the following command: note The backup contains vCluster Platform custom resources and associated Secrets, not the Custom Resource Definitions (CRDs) that define them. Install vCluster Platform on the target cluster before restoring. Its Helm chart installs the required CRDs automatically. Restoring onto a cluster without vCluster Platform already installed fails, because the CRDs for those resources don't exist yet. ``` kubectl apply -f .yaml ``` You might see warning messages similar to the following: ``` Warning: resource apps/argocd is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation is patched automatically. ``` These are just warnings and should not affect the restoration task. Multiple Kubernetes clusters with the same vCluster Platform config Do not install vCluster Platform (or its agents) into a cluster where it is already running. Running multiple vCluster Platform instances in the same cluster can cause reconciliation conflicts and lead to unexpected issues. --- # Restore license When performing disaster recovery or migrating the platform between clusters, it's critical to preserve your license instance ID. The platform generates a unique instance ID tied to a TLS certificate stored in the `loft-cert` Kubernetes secret. If this secret is lost during cluster reinstallation, the platform generates a new instance ID, which can invalidate your existing license. ## Understand the license instance ID[​](#understand-license-instance-id "Direct link to Understand the license instance ID") The platform's license system uses a unique instance ID that: * Ties to the TLS certificate stored in the `loft-cert` secret * Validates your platform license features * Persists across platform upgrades and restarts * Must remain consistent to maintain license validity ## Preservation strategies[​](#preservation-strategies "Direct link to Preservation strategies") You have two main approaches to preserve the license instance ID: ### Kubernetes backup solutions (recommended)[​](#kubernetes-backup-solutions "Direct link to Kubernetes backup solutions (recommended)") If you use a Kubernetes backup and restore solution like [Velero](https://velero.io/), [Kasten K10](https://www.kasten.io/), or similar tools, the `loft-cert` secret is automatically included in regular cluster backups. When you restore your cluster from these backups, the license instance ID is preserved automatically. This approach is recommended for: * Standard disaster recovery scenarios * Regular backup and restore workflows * Environments with existing backup infrastructure Automatic backup with Kubernetes tools If you use a Kubernetes backup and restore mechanism, you don't need to manually back up the `loft-cert` secret separately. Your regular backup process handles this automatically. The manual approach documented below is primarily relevant for migrations from one control plane cluster to another, rather than standard backup/restore or disaster recovery scenarios. ### Manual backup and restore[​](#manual-backup-restore "Direct link to Manual backup and restore") For cluster-to-cluster migrations or when not using a full Kubernetes backup solution, manually back up and restore the `loft-cert` secret. ## Manual backup procedure[​](#manual-backup-procedure "Direct link to Manual backup procedure") Before uninstalling the platform or wiping your cluster, back up the certificate secret: Back up the loft-cert secret ``` kubectl get secret loft-cert -n vcluster-platform -o yaml > loft-cert-backup.yaml ``` This command saves the `loft-cert` secret to a local file. Store this file securely as it contains sensitive certificate data. Store backups securely The backup file contains sensitive TLS certificate data. Store it in a secure location with appropriate access controls, such as: * Encrypted storage systems * Secret management tools * Version control systems with encryption at rest ## Manual restore procedure[​](#manual-restore-procedure "Direct link to Manual restore procedure") After reinstalling the platform, restore the certificate before the platform starts. Understand platform backup first Before performing a restore, familiarize yourself with the overall [platform backup and restore process](https://vcluster.com/docs/platform/maintenance/backup-restore/backup-restore-platform.md) to understand how license preservation fits into the complete disaster recovery workflow. ### Step 1: Install the platform without starting it[​](#install-without-starting "Direct link to Step 1: Install the platform without starting it") Install the platform using Helm with replica count set to 0: Install platform without starting ``` helm upgrade --install vcluster-platform vcluster-platform \ --repo https://charts.loft.sh \ --namespace vcluster-platform \ --create-namespace \ --set replicaCount=0 \ --version ``` warning vCluster Platform exclusively supports the default `secret` backend for storing Helm releases. Alternative `HELM_DRIVER` configurations (such as `configmap` or `sql`) are not supported. ### Step 2: Restore the certificate[​](#restore-certificate "Direct link to Step 2: Restore the certificate") Apply the backed-up certificate secret: Restore the loft-cert secret ``` kubectl apply -f loft-cert-backup.yaml ``` ### Step 3: Start the platform[​](#start-platform "Direct link to Step 3: Start the platform") Scale the platform deployment to start it with the restored certificate: Start the platform ``` helm upgrade vcluster-platform vcluster-platform \ --repo https://charts.loft.sh \ --namespace vcluster-platform \ --reuse-values \ --set replicaCount=1 ``` The platform now starts with your original instance ID, preserving your license validity. ## Verification[​](#verification "Direct link to Verification") After restoration, verify that your license is valid: 1. Access the platform UI 2. Navigate to **Admin > Platform Config > License and Billing** 3. Check the license status to ensure it shows as valid 4. Verify that your licensed features are available ## Important notes[​](#important-notes "Direct link to Important notes") ### Secrets not required for license preservation[​](#secrets-not-required "Direct link to Secrets not required for license preservation") The following secrets are **not** required for license preservation: * `loft-license-cache` * `loft-apiservice-agent-cert` * `loft-webhook-*` certificates * Agent certificates * User access keys These secrets are regenerated automatically and don't affect your license. The `loft-license-cache` secret is worth calling out because its name suggests otherwise. It holds a cached copy of the last license the platform retrieved, so the platform can keep running while the license server is unreachable. It carries no identity. The platform rebuilds it on the next successful retrieval. Only `loft-cert` determines the instance ID. See [How licensing works](https://vcluster.com/docs/platform/understand/licensing.md) for the full retrieval and caching model. ### Consider timing[​](#consider-timing "Direct link to Consider timing") * Always restore the `loft-cert` secret **before** the platform starts * If the platform starts without the secret, it generates a new instance ID * Once a new instance ID is generated, you need to contact support to update your license ### Multiple clusters[​](#multiple-clusters "Direct link to Multiple clusters") One instance per license Each platform license is valid for a single instance ID. Running multiple platform instances with the same `loft-cert` secret violates license terms and can cause synchronization issues. ## Troubleshoot issues[​](#troubleshoot-issues "Direct link to Troubleshoot issues") ### License shows as invalid after restoration[​](#license-invalid "Direct link to License shows as invalid after restoration") If your license shows as invalid after restoration: 1. Verify the `loft-cert` secret was restored correctly: ``` kubectl get secret loft-cert -n vcluster-platform ``` 2. Check the secret contains the expected data: ``` kubectl get secret loft-cert -n vcluster-platform -o jsonpath='{.data}' | jq 'keys' ``` Expected output should include `tls.crt` and `tls.key`. 3. Ensure the platform pods are using the restored certificate: ``` kubectl delete pods -n vcluster-platform -l app=loft ``` ### New instance ID generated[​](#new-instance-id "Direct link to New instance ID generated") If the platform generated a new instance ID: 1. Check if the `loft-cert` secret was present before the platform started 2. Review the platform logs for certificate generation messages 3. If needed, request a license update for the new instance ID ## Next steps[​](#next-steps "Direct link to Next steps") * Learn about [backing up the entire platform](https://vcluster.com/docs/platform/maintenance/backup-restore/backup-restore-platform.md) * Configure high availability for improved disaster recovery * Set up regular [Kubernetes backups](https://velero.io/) for comprehensive protection --- # Central HostPath Mapper | | | Enterprise | | | | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Central HostPath Mapper](https://vcluster.com/docs/platform/maintenance/monitoring/central-hostpath-mapper) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). You can install a common logging solution such as Fluentd or the ELK stack on the control plane cluster and use it within the tenant cluster. However, in some scenarios, installing these tools directly on the tenant cluster is preferred. These logging solutions access pod and container logs by mounting node host paths and using specific filename formats to associate logs with the correct pods. This approach doesn't work in a tenant cluster because vCluster rewrites pod names to a vCluster-specific format when syncing them to the control plane cluster. The HostPath Mapper collects logs from tenant clusters. It works by matching virtual pod and container names to their physical names on the host through symbolic links. To enable HostPath Mapper for a tenant cluster, install a DaemonSet in the same namespace. This setup works well for small environments with few nodes and tenant clusters. As the number of nodes and tenant clusters grows, the number of required DaemonSet pods also grows. Each tenant cluster needs one DaemonSet per node, which increases resource usage. The Central HostPath Mapper solves this problem. It installs a single DaemonSet on the control plane cluster. This DaemonSet manages host path remapping for all tenant clusters on the platform. It reduces overhead and simplifies cluster management. Prerequisites To use this feature, install the Central HostPath Mapper on the control plane cluster. The component is available as a default **vCluster Platform App** in the **Apps** section. Install it before enabling the feature to ensure full capabilities are available. Templates This feature is only configurable from the UI for tenant clusters created without a template. When using a custom template that has the feature enabled, the UI doesn't display the configuration in **Advanced Options** as described below. In this case, configure the feature using the CLI or by modifying the manifests. * During tenant cluster creation * On an existing tenant cluster 1. From the project drop-down menu, click the project where you want to create the tenant cluster. 2. Click the New Tenant Cluster button above the list of tenant clusters. 3. **\[Optional]** Click the cluster where you want to create the tenant cluster. 4. Leave the Select a template section blank and click the Configure to continue. 5. Click the Advanced Options configuration tab and expand the **Host Path Mapper** section. 6. Toggle the slider Enable. This adds the `loft.sh/hpm-enabled` annotation to the template metadata, which syncs automatically to the tenant cluster on creation. It also sets `enabled: true` and `central: true` under the [controlPlane.hostPathMapper](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/host-path-mapper.md) field in the vcluster.yaml. 7. Finish configuring anything else on your tenant cluster, then click Create Tenant Cluster. 1) From the project drop-down menu, click the project that contains the tenant cluster where you want to enable Central HostPath Mapper. 2) Hover over the tenant cluster where you want to enable Central HostPath Mapper to reveal the Edit button. Click Edit. 3) Click the Advanced Options configuration tab and expand the **Host Path Mapper** section. 4) Toggle the slider Enable. This adds the `loft.sh/hpm-enabled` annotation to the template metadata, which syncs automatically to the tenant cluster on creation. It also sets `enabled: true` and `central: true` under the [controlPlane.hostPathMapper](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/host-path-mapper.md) field in the vcluster.yaml. 5) Finish configuring anything else on your tenant cluster, then click Save Changes. --- # Logging vCluster Platform surfaces three distinct types of logs, each serving a different purpose and requiring different configuration. | Type | What it captures | Configure | | ----------------------------------------------- | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | [Platform audit logs](#platform-audit-logs) | API requests that pass through the platform gateway, including who acted and when | [Audit Logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md) | | [Platform process logs](#platform-process-logs) | Operational output from the platform pod, including startup, controller activity, and errors | [Platform Process Logging](https://vcluster.com/docs/platform/maintenance/logging/platform-logging.md) | | [Workload logs](#workload-logs) | Output from applications running inside tenant clusters | [Central HostPath Mapper](https://vcluster.com/docs/platform/maintenance/logging/central-hostpath-mapper.md) | These three types are independent. Enabling audit logging does not affect process log verbosity. A separate infrastructure component handles workload log collection. ## Platform audit logs[​](#platform-audit-logs "Direct link to Platform audit logs") The platform's API gateway records every request that passes through it. This covers requests from the UI, the CLI, and any API call routed through the platform proxy. Each log entry captures: * The requesting user (or service account), their groups, and source IP * The resource being accessed (type, name, namespace, cluster) * The HTTP verb (`get`, `list`, `create`, `update`, `delete`, `patch`) * Whether impersonation was in use * Timestamps and a unique audit ID ### What gets logged[​](#what-gets-logged "Direct link to What gets logged") Four audit levels control the volume and detail of what the platform captures. | Level | Writes (create/update/delete/patch) | Reads (get/list/watch) | Request body | Response body | | ----- | ---------------------------------------------------- | -------------------------------- | ------------ | ------------- | | 1 | Metadata only, but creates also log the request body | Not logged | Creates only | | | 2 | Metadata only, but creates also log the request body | Metadata only | Creates only | | | 3 | Logged with request body | Logged with request body | | | | 4 | Logged with request and response | Logged with request and response | | | At all levels, a fixed set of built-in rules run before the level's own rules. Some requests are never logged, including internal access-review and token requests, and kubeconfig retrieval for a tenant cluster. Sensitive `create` requests, such as access keys and cluster registration, stay at metadata level even when the configured level would otherwise capture the full payload. Long-running streaming requests, such as `pods/log` and `pods/exec`, also stay at metadata level regardless of the configured level. See [Default policy exclusions](https://vcluster.com/docs/platform/configure/platform-configs/audit.md#default-policy-exclusions) for the full list of affected resources and why. ### Configure audit logging[​](#configure-audit-logging "Direct link to Configure audit logging") See [Audit Logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md) for how to enable audit logging, choose a level, define a custom policy, and persist logs to a PVC or external database. ## Platform process logs[​](#platform-process-logs "Direct link to Platform process logs") The platform pod writes operational logs to stdout. These cover initialization, controller reconciliation, API request handling, and runtime errors. The platform supports three log levels. | Level | What it captures | When to use | | ------- | ---------------------------------------------------------------- | ---------------------------------------------------------- | | `info` | Startup sequences, controller events, handled requests | Default for all environments | | `debug` | Full operation context for every request and reconciliation loop | Temporarily, when diagnosing an issue | | `error` | Failures and warnings only | Stable production environments with low noise requirements | By default, logs use a human-readable console format. Switch to JSON format when feeding logs into an aggregation pipeline such as Elasticsearch or Grafana Loki. Access platform logs with: ``` kubectl logs -n vcluster-platform -l app=loft -f ``` See [Platform Process Logging](https://vcluster.com/docs/platform/maintenance/logging/platform-logging.md) for how to change log level and output format. ## Workload logs[​](#workload-logs "Direct link to Workload logs") Standard log collectors such as Fluentd, the ELK stack, and Grafana Loki read container logs from the host filesystem at paths like: ``` /var/log/pods/__// ``` Tenant clusters introduce a naming problem. When vCluster syncs pods to the control plane cluster, it rewrites their names. A pod named `web-6d4f9b` inside a tenant cluster appears as `web-6d4f9b-x-default-x-my-vcluster` on the control plane cluster. Log collectors see only the physical name and cannot correlate entries back to the originating tenant cluster workload. The HostPath Mapper solves this by creating symlinks on each node that map physical pod paths to their virtual names. Log collectors follow the symlinks and see the names they expect. vCluster Platform offers two deployment options: * **Central HostPath Mapper**: A single DaemonSet on the control plane cluster handles path remapping for all tenant clusters. Recommended for fleet deployments. * **Per-cluster HostPath Mapper**: A DaemonSet installed in each tenant cluster's namespace. Better suited to small environments or where per-cluster isolation is needed. See [Central HostPath Mapper](https://vcluster.com/docs/platform/maintenance/logging/central-hostpath-mapper.md) for setup instructions for both options. --- # Platform Process Logging vCluster Platform writes operational logs to `stdout` from the platform pod. These logs cover initialization, controller reconciliation, API request handling, and runtime errors. Check these logs first when diagnosing unexpected platform behavior. By default, logs use a human-readable console format at `info` level. ## Log levels[​](#log-levels "Direct link to Log levels") Platform logging supports three log levels: * `info` * `debug` * `error` For a description of each level, see [Platform process logs](https://vcluster.com/docs/platform/maintenance/logging/overview.md#platform-process-logs). warning Debug logging significantly increases log volume. Enable it only while reproducing an issue, then return to `info`. To change the log level, upgrade the Helm release with the `logging.level` value: ``` helm upgrade vcluster-platform vcluster-platform \ --repo https://charts.loft.sh \ --namespace vcluster-platform \ --reuse-values \ --set logging.level=debug ``` ## Log format[​](#log-format "Direct link to Log format") vCluster Platform supports two output formats. * Console * JSON The default console format produces human-readable output suited to manual log review: ``` 2023-07-11 09:20:56 INFO controller-runtime.metrics metrics/listener.go:44 Metrics server is starting to listen {"component": "loft", "addr": "127.0.0.1:12000"} 2023-07-11 09:20:56 INFO initialize/context.go:68 Initialize... {"component": "loft"} 2023-07-11 09:20:56 INFO initialize/context.go:72 Ensure certificates... {"component": "loft"} ``` The JSON format produces structured output suited to log aggregation systems such as Elasticsearch, Splunk, or Grafana Loki: ``` {"level":"info","ts":1689067388.209614,"logger":"controller-runtime.metrics","caller":"metrics/listener.go:44","msg":"Metrics server is starting to listen","component":"loft","addr":"127.0.0.1:12000"} {"level":"info","ts":1689067388.2099726,"caller":"initialize/context.go:68","msg":"Initialize...","component":"loft"} {"level":"info","ts":1689067388.2100549,"caller":"initialize/context.go:72","msg":"Ensure certificates...","component":"loft"} ``` Each entry contains at minimum: | Field | Type | Description | | ----------- | ------ | -------------------------------------------- | | `level` | string | Log level (`info`, `warn`, `error`, `debug`) | | `ts` | number | Unix timestamp (seconds since epoch) | | `caller` | string | Source file and line number | | `msg` | string | Log message | | `component` | string | Platform component that emitted the log | Additional fields vary by subsystem and operation type. For example, controller events may include `namespace` and `name` while API handler logs may include `method` and `status`. To switch to JSON format: ``` helm upgrade vcluster-platform vcluster-platform \ --repo https://charts.loft.sh \ --namespace vcluster-platform \ --reuse-values \ --set logging.encoding=json ``` ## View logs[​](#view-logs "Direct link to View logs") ``` kubectl logs -n vcluster-platform -l app=loft -f ``` In high-availability deployments, multiple platform pods run concurrently. Pipe through a log aggregator, or target a specific pod, to avoid interleaved output from multiple replicas. ## File logging[​](#file-logging "Direct link to File logging") Besides `stdout`, the platform pod can also write logs to a file. This is useful for tailing logs from a sidecar container instead of going through the container runtime's log driver. Enable file logging with the `logging.file.enabled` Helm value: ``` helm upgrade vcluster-platform vcluster-platform \ --repo https://charts.loft.sh \ --namespace vcluster-platform \ --reuse-values \ --set logging.file.enabled=true ``` The log path is fixed at `/var/log/vcluster-platform/platform.log` and isn't configurable. Enabling this setting adds a `platform-logs` `emptyDir` volume (1Gi size limit) to the pod and mounts it into the `manager` container at `/var/log/vcluster-platform`. The chart reserves both the `platform-logs` volume name and the `manager`/`audit` container names, so a custom `volumes`, `volumeMounts`, or `sidecarContainers` entry can't reuse them. Enabling `logging.file.enabled` only writes the file. Nothing reads it automatically, so add a sidecar container that mounts the reserved `platform-logs` volume to actually read it: values.yaml ``` logging: file: enabled: true sidecarContainers: - name: log-reader image: busybox command: ["sh", "-c", "tail -F /var/log/vcluster-platform/platform.log"] volumeMounts: - name: platform-logs mountPath: /var/log/vcluster-platform readOnly: true ``` View the sidecar's output the same way as the `stdout` logs above, targeting the sidecar's container name instead: ``` kubectl logs -n vcluster-platform -l app=loft -c log-reader -f ``` Distinct from tenant cluster file logging This is the platform's own process logging. vCluster charts v0.37 and later have a separate, unrelated `logging.file.enabled` option for tenant cluster logs, configured in `vcluster.yaml` rather than the platform's Helm values. --- # Aggregating Metrics Reference architecture This guide predates Fleet Observability. It's an operator-maintained architecture example, not a Platform-managed monitoring feature. For new deployments, start with [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md). This guide's identity labels are added by the collector itself and aren't authenticated by vCluster Platform. They're a different label namespace than Fleet Observability's `vcluster_platform_*` labels. Dashboards, alerts, and recording rules built for one scheme return empty results against the other. For architecture and migration details, see [Fleet metrics architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md) and the [Fleet Observability labels reference](https://vcluster.com/docs/platform/maintenance/observability/labels-reference.md). This guide explains how to configure OpenTelemetry with Prometheus to collect workload metrics from across multiple tenant clusters and aggregate by Project, vCluster, and Space. This approach uses shared OpenTelemetry DaemonSets on the control plane cluster without requiring individual collector installations per tenant cluster. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you begin, decide where to deploy Prometheus and Grafana. This guide uses the Platform's connected `local-cluster` with both services in an `observability` namespace. You can adapt this to external clusters or different namespaces based on your requirements. Don't use kube-prometheus-stack Do not use the `kube-prometheus-stack` chart from the Platform Apps page for this setup. Instead, use the standalone `prometheus` and `grafana` charts as described below, which are optimized to work with the OpenTelemetry Collector. ## Deploy Prometheus[​](#deploy-prometheus "Direct link to Deploy Prometheus") Deploy Prometheus with OTLP receiver support using the Platform Apps UI. 1. Go to Infrastructure > Control Plane Clusters. 2. Click the cluster where you want to deploy Prometheus (for example, `local-cluster`). 3. Navigate to the Apps tab. 4. Click Install App and configure a Helm chart with the following settings. | Setting | Value | | -------------------- | ---------------------------------------------------- | | Chart Repository URL | `https://prometheus-community.github.io/helm-charts` | | Chart Name | `prometheus` | | Namespace | `observability` | | Release Name | `prometheus` | Use the following chart values to enable the OTLP receiver and required features: Prometheus Helm values ``` # Enable OTLP receiver for OpenTelemetry metrics ingestion # Enable delta-to-cumulative conversion for OTLP metrics # Enable lifecycle API for configuration reloads server: extraFlags: - web.enable-otlp-receiver - web.enable-lifecycle extraArgs: enable-feature: otlp-deltatocumulative # Skip RBAC creation if prometheus-server ClusterRole already exists rbac: create: false # Disable components not needed for this setup prometheus-node-exporter: enabled: false alertmanager: enabled: false kube-state-metrics: enabled: false prometheus-pushgateway: enabled: false ``` Existing Prometheus installations The `rbac.create: false` setting skips ClusterRole creation, which prevents conflicts if a `prometheus-server` ClusterRole already exists in your cluster from vCluster Platform or other components. If you don't have an existing ClusterRole, either remove this setting or use `server.clusterRoleNameOverride: "otel-prometheus-server"` to create one with a unique name. Click Install to deploy Prometheus. ## Deploy OpenTelemetry Collector[​](#deploy-opentelemetry-collector "Direct link to Deploy OpenTelemetry Collector") Deploy the built-in OpenTelemetry App on each connected cluster. This OpenTelemetry App accepts the Prometheus connection information and deploys [opentelemetry-collector](https://opentelemetry.io/docs/collector/) as a DaemonSet using Helm. The OpenTelemetry Collector Agent on each node pushes metrics about the workloads running on that node to the Prometheus instance. The metrics include vCluster, vCluster Platform, and Kubernetes metadata as labels. 1. Go to Infrastructure > Control Plane Clusters. 2. Click the cluster where you are installing the OpenTelemetry Collector App. 3. Navigate to the Apps tab. 4. Find and click the **OpenTelemetry Collector** App. 5. Enter the Prometheus connection endpoint: `http://prometheus-server.observability.svc.cluster.local:80` 6. Click on the Install button to finish. Prometheus endpoint If you deployed Prometheus in a different namespace or cluster, adjust the endpoint URL accordingly. The format is `http://-server..svc.cluster.local:80`. ## Deploy Grafana[​](#deploy-grafana "Direct link to Deploy Grafana") Deploy Grafana with a pre-configured Prometheus datasource and dashboard using the Platform Apps UI. 1. Go to Infrastructure > Control Plane Clusters. 2. Click the cluster where you deployed Prometheus. 3. Navigate to the Apps tab. 4. Click Install App and configure a Helm chart with the following settings. | Setting | Value | | -------------------- | --------------------------------------- | | Chart Repository URL | `https://grafana.github.io/helm-charts` | | Chart Name | `grafana` | | Namespace | `observability` | | Release Name | `grafana` | Use the following chart values to configure the Prometheus datasource and include a pre-built dashboard: Grafana Helm values ``` # Configure Prometheus as the default datasource datasources: datasources.yaml: apiVersion: 1 datasources: - name: Prometheus type: prometheus url: http://prometheus-server access: proxy isDefault: true # Configure dashboard provisioning dashboardProviders: dashboardproviders.yaml: apiVersion: 1 providers: - name: 'default' orgId: 1 folder: '' type: file disableDeletion: false editable: true options: path: /var/lib/grafana/dashboards/default # Include a pre-built dashboard for vCluster Platform metrics dashboards: default: vcluster-platform-metrics: json: | {"title":"CPU and Memory usage by Project, Space, Virtual Cluster","panels":[{"gridPos":{"h":8,"w":12,"x":0,"y":0},"id":3,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true}},"targets":[{"disableTextWrap":false,"editorMode":"code","expr":"sum by(loft_virtualcluster_name) (rate(k8s_pod_cpu_time_seconds_total{loft_virtualcluster_name=~\".+\"}[5m]))","fullMetaSearch":false,"includeNullMetadata":true,"instant":false,"legendFormat":"__auto","range":true,"refId":"A","useBackend":false}],"title":"CPU Usage by Virtual Cluster","type":"timeseries","fieldConfig":{"defaults":{"unit":"short"},"overrides":[]}},{"gridPos":{"h":8,"w":12,"x":12,"y":0},"id":4,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true}},"targets":[{"disableTextWrap":false,"editorMode":"code","expr":"sum by (loft_virtualcluster_name) (k8s_pod_memory_usage_bytes{loft_virtualcluster_name=~\".+\"})\n/1024/1024","fullMetaSearch":false,"includeNullMetadata":true,"instant":false,"legendFormat":"__auto","range":true,"refId":"A","useBackend":false}],"title":"Memory Usage (MiB) by Virtual Cluster","type":"timeseries","fieldConfig":{"defaults":{"unit":"decmbytes"},"overrides":[]}},{"gridPos":{"h":8,"w":12,"x":0,"y":8},"id":2,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true}},"targets":[{"disableTextWrap":false,"editorMode":"builder","expr":"sum by(loft_project_name) (rate(k8s_pod_cpu_time_seconds_total{loft_project_name=~\".+\"}[5m]))","fullMetaSearch":false,"includeNullMetadata":true,"instant":false,"interval":"","legendFormat":"__auto","range":true,"refId":"A","useBackend":false}],"title":"CPU Usage by Project","type":"timeseries","fieldConfig":{"defaults":{"unit":"short"},"overrides":[]}},{"gridPos":{"h":8,"w":12,"x":12,"y":8},"id":1,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true}},"targets":[{"disableTextWrap":false,"editorMode":"code","expr":"sum by (loft_project_name) (k8s_pod_memory_usage_bytes{loft_project_name=~\".+\"})\n/1024/1024","fullMetaSearch":false,"includeNullMetadata":true,"instant":false,"legendFormat":"__auto","range":true,"refId":"A","useBackend":false}],"title":"Memory Usage (MiB) by Project","type":"timeseries","fieldConfig":{"defaults":{"unit":"decmbytes"},"overrides":[]}},{"gridPos":{"h":8,"w":12,"x":0,"y":16},"id":6,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true}},"targets":[{"disableTextWrap":false,"editorMode":"code","expr":"sum by(loft_space_name) (rate(k8s_pod_cpu_time_seconds_total{loft_space_name=~\".+\"}[5m]))","fullMetaSearch":false,"includeNullMetadata":true,"instant":false,"legendFormat":"__auto","range":true,"refId":"A","useBackend":false}],"title":"CPU Usage by Space","type":"timeseries","fieldConfig":{"defaults":{"unit":"short"},"overrides":[]}},{"gridPos":{"h":8,"w":12,"x":12,"y":16},"id":5,"options":{"legend":{"calcs":[],"displayMode":"list","placement":"bottom","showLegend":true}},"targets":[{"disableTextWrap":false,"editorMode":"code","expr":"sum by (loft_space_name) (k8s_pod_memory_usage_bytes{loft_space_name=~\".+\"})\n/1024/1024","fullMetaSearch":false,"includeNullMetadata":true,"instant":false,"legendFormat":"__auto","range":true,"refId":"A","useBackend":false}],"title":"Memory Usage (MiB) by Space","type":"timeseries","fieldConfig":{"defaults":{"unit":"decmbytes"},"overrides":[]}}],"schemaVersion":38} ``` Click Install to deploy Grafana. ## Access Grafana[​](#access-grafana "Direct link to Access Grafana") After deploying Grafana, retrieve the admin password and access the dashboard. ### Get the Grafana password[​](#get-the-grafana-password "Direct link to Get the Grafana password") Run the following command to retrieve the Grafana admin password: ``` kubectl get secret --namespace observability grafana \ -o jsonpath="{.data.admin-password}" | base64 --decode; echo ``` ### Option 1: Port forward[​](#option-1-port-forward "Direct link to Option 1: Port forward") Use port forwarding to access Grafana locally: ``` kubectl port-forward -n observability service/grafana 8080:80 ``` Then open your browser and navigate to `http://localhost:8080`. Log in with username `admin` and the password from the previous step. ### Option 2: Ingress[​](#option-2-ingress "Direct link to Option 2: Ingress") Create an Ingress resource to expose Grafana externally. The following example uses the nginx ingress controller (deprecated): grafana-ingress.yaml ``` apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/rewrite-target: / # Uncomment the following for TLS with cert-manager: # cert-manager.io/cluster-issuer: letsencrypt-prod # nginx.ingress.kubernetes.io/ssl-redirect: "true" name: grafana namespace: observability spec: ingressClassName: nginx rules: - host: grafana.example.com # Replace with your hostname http: paths: - backend: service: name: grafana port: number: 80 path: / pathType: Prefix # Uncomment the following for TLS: # tls: # - hosts: # - grafana.example.com # secretName: grafana-ingress-tls ``` Apply the Ingress: ``` kubectl apply -f grafana-ingress.yaml ``` Navigate to your configured hostname and log in with the admin credentials. ## Create Prometheus queries[​](#create-prometheus-queries "Direct link to Create Prometheus queries") After collecting data with OpenTelemetry, you can aggregate the metrics using the associated labels like the vCluster name. `k8s_pod_cpu_time_seconds_total` is a cumulative counter, so wrap it in `rate()` to get CPU cores used rather than an ever-increasing total. Here's an example Prometheus query showing the CPU usage aggregated by the vCluster name: ``` sum by(loft_virtualcluster_name) (rate(k8s_pod_cpu_time_seconds_total{loft_virtualcluster_name=~".+"}[5m])) ``` ### Available labels[​](#available-labels "Direct link to Available labels") The OpenTelemetry Collector adds the following vCluster Platform labels to metrics: | Label | Description | | -------------------------- | ---------------------------------- | | `loft_project_name` | The vCluster Platform project name | | `loft_virtualcluster_name` | The tenant cluster name | | `loft_space_name` | The space name | | `cluster` | The connected cluster name | Space label only applies to Space-type tenants `loft_space_name` is only populated for namespaces backing a Space-type tenant. Environments with only vCluster-type tenants won't have any series for this label, so queries and dashboard panels aggregated `by(loft_space_name)` show no data. ### Example queries[​](#example-queries "Direct link to Example queries") **Memory usage by project:** ``` sum by (loft_project_name) (k8s_pod_memory_usage_bytes{loft_project_name=~".+"}) / 1024 / 1024 ``` **CPU usage by space:** ``` sum by(loft_space_name) (rate(k8s_pod_cpu_time_seconds_total{loft_space_name=~".+"}[5m])) ``` ## Example dashboard[​](#example-dashboard "Direct link to Example dashboard") The Grafana deployment includes a pre-built dashboard that visualizes CPU and Memory usage aggregated by vCluster, Project, and Space. ![Grafana dashboard showing CPU and Memory usage by Project, Space, and Tenant Cluster](/docs/assets/images/grafana-example-192dd1db76ec81fb1c03c3110cbf17ff.png) The **by Space** panels only show data if your environment has Space-type tenants. In a vCluster-only environment, they stay empty. You can also [import](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/import-dashboards/) additional dashboards or create custom visualizations using the available labels. --- # Fleet metrics architectures Use [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md) for new fleet-wide metrics deployments. It's the Platform-managed observability feature. The other pages in this section are operator-maintained reference architectures that predate Fleet Observability. The architectures don't share a label contract. Dashboards, alerts, and recording rules built for one architecture don't automatically work with another. ## Compare the architectures[​](#compare-the-architectures "Direct link to Compare the architectures") | | Fleet Observability | Fleet monitoring with OpenTelemetry | Aggregating Metrics | | ------------------------------------------------ | ------------------------------------------------------------- | -------------------------------------------------------- | --------------------------------------------------- | | Status | Recommended Platform feature | Reference architecture | Reference architecture | | Identity labels | `vcluster_platform_*`, authenticated | `vcluster_*`/`cluster`, self-reported | `loft_*`, self-reported | | Read access control | Scoped per tenant or cluster, Platform-authorized | None, any reader sees all data | None, any reader sees all data | | GPU observability | DCGM, NVSentinel | Not covered | Not covered | | Requires Argo CD for the bundled deploy path | Yes (manual config available for Standalone) | No | No | | Runs on embedded Kubernetes platform deployments | No | Yes | Yes | | Included examples | Query examples | PromQL library and two Grafana dashboards | Prometheus and Grafana example | | Operational ownership | Platform gateway and access policy, plus your metrics backend | You maintain and validate the collector Apps and backend | You maintain and validate the collector and backend | ## When a reference architecture is necessary[​](#when-a-reference-architecture-is-necessary "Direct link to When a reference architecture is necessary") Fleet Observability's bundled Argo CD deployment doesn't cover vCluster Standalone. You can still configure a collector manually. Embedded Kubernetes deployments can't run the observability gateway. If either constraint applies, the older guides can help you design a direct collector path: * Use [Fleet monitoring with OpenTelemetry](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-monitoring-otel.md) as a detailed example for Shared Nodes and Private Nodes deployments. * Use [Aggregating Metrics](https://vcluster.com/docs/platform/maintenance/monitoring/aggregating-metrics.md) as a smaller example built around the OpenTelemetry Collector App. These guides aren't maintained as turnkey production configurations. Validate chart versions, security settings, resource sizing, and backend compatibility for your environment. ## Next steps[​](#next-steps "Direct link to Next steps") * Start a supported deployment with the [Fleet Observability quickstart](https://vcluster.com/docs/platform/maintenance/observability/quickstart.md). * See the [labels reference](https://vcluster.com/docs/platform/maintenance/observability/labels-reference.md) before migrating dashboards or alerts from an older architecture. --- # Fleet monitoring with OpenTelemetry Reference architecture This guide predates Fleet Observability. It's an operator-maintained architecture example, not a Platform-managed monitoring feature. For new deployments, start with [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md). This guide's identity labels are added by the collector itself and aren't authenticated by vCluster Platform. They're a different label namespace than Fleet Observability's `vcluster_platform_*` labels. Dashboards, alerts, and recording rules built for one scheme return empty results against the other. For architecture and migration details, see [Fleet metrics architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md) and the [Fleet Observability labels reference](https://vcluster.com/docs/platform/maintenance/observability/labels-reference.md). When managing many tenant clusters across control plane clusters, you need per-tenant visibility into resource consumption and API health without deploying monitoring inside each tenant cluster. This guide explains how to configure OpenTelemetry Collectors to collect workload and control plane metrics from across multiple tenant clusters. All metrics are enriched with vCluster identity labels at enrichment time and pushed to a central Prometheus using `remote_write`. For a simpler setup using the built-in OpenTelemetry DaemonSet App, see [Aggregating Metrics](https://vcluster.com/docs/platform/maintenance/monitoring/aggregating-metrics.md) instead. This guide covers advanced fleet monitoring with `remote_write`, Target Allocator, and per-tenancy-model collectors. This architecture supports both the Shared Nodes and Private Nodes tenancy models. Each model uses a different collector configuration deployed as a vCluster Platform App. warning This guide isn't a production-ready monitoring solution that you can copy directly to your infrastructure. Observability is highly specialized to the underlying architecture. The goal is to lay out general capabilities and show what's possible along with a stripped-down example architecture. Apply these patterns with modifications to your actual use cases. ## Architecture[​](#architecture "Direct link to Architecture") The architecture comprises the following: * Cluster architecture: * A local cluster that hosts vCluster Platform. * Two tenant clusters running on the local cluster: * One tenant cluster sharing the nodes of the local cluster (Shared Nodes tenancy model). * One tenant cluster with private nodes (Private Nodes tenancy model). * An external cluster connected to vCluster Platform. * Two tenant clusters running on the connected cluster with the same configuration. * Collector architecture: * A central Prometheus with the remote write receiver enabled. * One OTel Collector Deployment with Target Allocator per control plane cluster (scrapes shared-nodes tenant clusters and their control planes using ServiceMonitors). * One OTel Collector DaemonSet per private-nodes tenant cluster (scrapes local kubelet, cAdvisor, and [API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) metrics from inside the tenant cluster). ## How it works[​](#how-it-works "Direct link to How it works") ### Shared nodes[​](#shared-nodes "Direct link to Shared nodes") note For background on why direct Prometheus scraping fails in Shared Nodes mode and how to configure it, see [Prometheus node metrics on shared nodes](https://vcluster.com/docs/platform/maintenance/monitoring/prometheus-shared-nodes.md). The shared-nodes collector runs on the control plane cluster as a Deployment with 2 replicas. A Target Allocator discovers vCluster ServiceMonitors and distributes cAdvisor and ServiceMonitor scrape targets across replicas using consistent-hashing. **Metrics pipeline:** ``` prometheus receiver → memory_limiter → groupbyattrs (split cAdvisor batch into per-pod resource scopes) → transform/pre_enrich (copy namespace/pod/node to k8s.* resource attributes) → k8sattributes (resolve pod/namespace metadata, add vCluster labels) → filter/vcluster_only (drop metrics without vCluster identity) → resource/add_cluster (add cluster label from Platform variable) → transform (copy resource attributes to datapoint attributes) → batch → prometheusremotewrite ``` The `groupbyattrs` processor is required because the Prometheus receiver batches all cAdvisor metrics from a single node into one resource scope. Without it, the `k8sattributes` processor matches one pod and applies its metadata to all metrics in the batch, causing cross-contamination between tenant clusters on the same node. The `groupbyattrs` processor splits the batch into per-pod resource scopes (by `namespace`, `pod`, `node`) so each pod is matched correctly. The `k8sattributes` processor resolves vCluster identity from Platform-managed namespace labels (`loft.sh/project`, `loft.sh/vcluster-instance-name`, and so on) and pod labels/annotations set by the vCluster [syncer](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync) (`vcluster.loft.sh/namespace`, `vcluster.loft.sh/name`). The `filter/vcluster_only` processor drops any metrics where `vcluster.name` is `nil` after enrichment. This means only vCluster workload metrics pass through, which also prevents duplicate series with any existing Prometheus scrapes. ### Private nodes[​](#private-nodes "Direct link to Private nodes") The private-nodes collector runs inside each tenant cluster as a DaemonSet with one pod per node. Each pod scrapes only its local node's kubelet `/metrics`, cAdvisor `/metrics/cadvisor`, and API server `/metrics` endpoints. **Metrics pipeline:** ``` prometheus receiver (kubelet, cAdvisor, API server) → memory_limiter → k8sattributes (resolve pod metadata) → transform (copy resource attributes to datapoint attributes) → batch → prometheusremotewrite + external_labels (cluster, vcluster_name, project, user) + metric_relabel (namespace → vcluster_virtual_namespace, pod → vcluster_virtual_pod) ``` Since the collector runs inside the tenant cluster, it can't access Control Plane Cluster namespace labels. Instead, the Platform injects `{{ .Values.loft.* }}` template variables at deploy time, which are set as `external_labels` on the `prometheusremotewrite` exporter. These are static per-tenant-cluster values applied to all exported metrics. The `metric_relabel_configs` copy `namespace` to `vcluster_virtual_namespace` and `pod` to `vcluster_virtual_pod`. Inside a private-nodes tenant cluster, the `namespace` and `pod` labels already represent virtual names, so this copy ensures dashboard compatibility with the shared-nodes collector. ## Metric labels[​](#metric-labels "Direct link to Metric labels") All metrics from both apps carry a consistent set of identity labels: | Label | Shared nodes source | Private nodes source | | ---------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------ | | `cluster` | `resource/add_cluster` processor using `{{ .Values.loft.cluster }}` | `external_labels` using `{{ .Values.loft.cluster }}` | | `vcluster_name` | `k8sattributes` from namespace label `loft.sh/vcluster-instance-name` | `external_labels` using `{{ .Values.loft.name }}` | | `vcluster_project` | `k8sattributes` from namespace label `loft.sh/project` | `external_labels` using `{{ .Values.loft.project }}` | | `vcluster_user` | `k8sattributes` from namespace label `loft.sh/user` | `external_labels` using `{{ .Values.loft.user.name }}` | | `vcluster_project_namespace` | `k8sattributes` from namespace label `loft.sh/vcluster-instance-namespace` | `external_labels` using `{{ .Values.loft.space }}` | | `vcluster_virtual_namespace` | `k8sattributes` from pod label `vcluster.loft.sh/namespace` | `metric_relabel_configs` copying `namespace` label | | `vcluster_virtual_pod` | `k8sattributes` from pod annotation `vcluster.loft.sh/name` | `metric_relabel_configs` copying `pod` label | info `vcluster_virtual_namespace` and `vcluster_virtual_pod` are missing on some metrics. These are vCluster system pods (syncer, CoreDNS) that don't have the syncer labels and annotations because they aren't user workloads synced from inside the tenant cluster. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") The central Prometheus must be configured as a remote write receiver. The following Helm values enable this: ``` server: extraFlags: - web.enable-remote-write-receiver ``` ### Shared nodes prerequisites[​](#shared-nodes-prerequisites "Direct link to Shared nodes prerequisites") * Prometheus Operator CRDs installed on the control plane cluster (`ServiceMonitor`, `PodMonitor`). * Tenant clusters deployed with a ServiceMonitor enabled. This allows scraping their API server and controller metrics. Enable this in your `vcluster.yaml`: ``` controlPlane: serviceMonitor: enabled: true ``` * kubelet scraping disabled in any existing kube-prometheus-stack to avoid duplicate cAdvisor series (`kubelet.enabled: false`). * Platform namespace labels present (added automatically by vCluster Platform). ### Private nodes prerequisites[​](#private-nodes-prerequisites "Direct link to Private nodes prerequisites") * Tenant clusters with dedicated/private nodes. * Node-to-node vCluster VPN enabled: ``` privateNodes: enabled: true vpn: enabled: true nodeToNode: enabled: true ``` ## Deploy the shared nodes collector[​](#deploy-the-shared-nodes-collector "Direct link to Deploy the shared nodes collector") Deploy one shared-nodes collector per control plane cluster. First, register the App manifest with the Platform, then deploy it to each cluster through the UI. ### App manifest[​](#app-manifest "Direct link to App manifest") The shared-nodes app deploys the [`opentelemetry-kube-stack`](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-kube-stack) Helm chart (v0.14.4) with the following configuration: otel-collector-shared-nodes-app.yaml otel-collector-shared-nodes-app.yaml ``` apiVersion: management.loft.sh/v1 kind: App metadata: name: otel-collector-shared-nodes spec: access: - users: - '*' verbs: - get config: chart: name: opentelemetry-kube-stack repoURL: https://open-telemetry.github.io/opentelemetry-helm-charts version: 0.14.4 values: | # yaml --- clusterName: "{{ .Values.loft.cluster }}" crds: installPrometheus: false opentelemetry-operator: enabled: true manager: collectorImage: repository: otel/opentelemetry-collector-contrib featureGatesMap: operator.targetallocator.mtls: true admissionWebhooks: certManager: enabled: false autoGenerateCert: enabled: true recreate: true # Post-install job: works around an OpenTelemetry Operator bug where DELETE # validation webhooks block app uninstallation. This block can be removed # once an upstream fix is released. extraObjects: - apiVersion: v1 kind: ServiceAccount metadata: name: patch-webhook-sa annotations: "helm.sh/hook": post-install,post-upgrade "helm.sh/hook-weight": "1" "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otel-collector-shared-nodes-patch-webhook annotations: "helm.sh/hook": post-install,post-upgrade "helm.sh/hook-weight": "1" "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded rules: - apiGroups: ["admissionregistration.k8s.io"] resources: ["validatingwebhookconfigurations"] verbs: ["get", "patch"] - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otel-collector-shared-nodes-patch-webhook annotations: "helm.sh/hook": post-install,post-upgrade "helm.sh/hook-weight": "1" "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded subjects: - kind: ServiceAccount name: patch-webhook-sa # Keep this namespace aligned with spec.defaultNamespace below. namespace: monitoring roleRef: kind: ClusterRole name: otel-collector-shared-nodes-patch-webhook apiGroup: rbac.authorization.k8s.io - apiVersion: batch/v1 kind: Job metadata: name: patch-webhook annotations: "helm.sh/hook": post-install,post-upgrade "helm.sh/hook-weight": "10" "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: template: spec: restartPolicy: Never serviceAccountName: patch-webhook-sa containers: - name: patch image: "bitnami/kubectl:latest" command: ["bash", "-c"] args: - | WH="otel-collector-shared-nodes-opentelemetry-operator-validation" for i in $(seq 1 30); do kubectl get validatingwebhookconfiguration "$WH" >/dev/null 2>&1 && break; sleep 2; done # Build JSON patch to remove webhooks with "delete" in name (reverse order to preserve indices) PATCH=$(kubectl get validatingwebhookconfiguration "$WH" -o jsonpath='{range .webhooks[*]}{.name}{"\n"}{end}' \ | awk '/delete/{print NR-1}' | sort -rn \ | awk 'BEGIN{printf "["} NR>1{printf ","} {printf "{\"op\":\"remove\",\"path\":\"/webhooks/%d\"}",$1} END{printf "]"}') [ "$PATCH" = "[]" ] && exit 0 echo "Removing DELETE webhooks: $PATCH" kubectl patch validatingwebhookconfiguration "$WH" --type=json -p="$PATCH" collectors: # Disable the default DaemonSet collector daemon: enabled: false # Deployment-mode collector with Target Allocator cluster: enabled: true suffix: cluster mode: deployment replicas: 2 resources: limits: memory: 1Gi requests: cpu: 250m memory: 512Mi livenessProbe: initialDelaySeconds: 15 periodSeconds: 10 failureThreshold: 5 presets: kubernetesAttributes: enabled: true targetAllocator: enabled: true allocationStrategy: consistent-hashing prometheusCR: enabled: true serviceMonitorSelector: matchLabels: app: vcluster podMonitorSelector: {} config: receivers: prometheus: config: scrape_configs: - job_name: 'kubelet-cadvisor' scrape_interval: 60s kubernetes_sd_configs: - role: node scheme: https tls_config: insecure_skip_verify: true authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token metrics_path: /metrics/cadvisor relabel_configs: - source_labels: [__meta_kubernetes_node_address_InternalIP] target_label: __address__ replacement: '$$1:10250' - action: labelmap regex: __meta_kubernetes_node_label_(.+) - source_labels: [__meta_kubernetes_node_name] target_label: node processors: groupbyattrs: keys: - namespace - pod - node transform/pre_enrich: error_mode: ignore metric_statements: - context: resource statements: - 'set(attributes["k8s.namespace.name"], attributes["namespace"]) where attributes["namespace"] != nil' - 'set(attributes["k8s.pod.name"], attributes["pod"]) where attributes["pod"] != nil' - 'set(attributes["k8s.node.name"], attributes["node"]) where attributes["node"] != nil' k8sattributes: auth_type: serviceAccount passthrough: false extract: metadata: - k8s.namespace.name - k8s.pod.name - k8s.pod.start_time - k8s.pod.uid - k8s.deployment.name - k8s.node.name - k8s.container.name labels: # Pod labels - vcluster syncer adds these to synced pods - tag_name: vcluster.virtual.namespace key: vcluster.loft.sh/namespace from: pod # Namespace labels - platform adds these to vcluster namespaces - tag_name: vcluster.project key: loft.sh/project from: namespace - tag_name: vcluster.project.namespace key: loft.sh/vcluster-instance-namespace from: namespace - tag_name: vcluster.user key: loft.sh/user from: namespace - tag_name: vcluster.name key: loft.sh/vcluster-instance-name from: namespace annotations: # Pod annotations - identifies the virtual pod name - tag_name: vcluster.virtual.pod key: vcluster.loft.sh/name from: pod transform: error_mode: ignore metric_statements: - context: datapoint statements: - 'set(attributes["k8s.node.name"], resource.attributes["k8s.node.name"])' - 'set(attributes["k8s.pod.name"], resource.attributes["k8s.pod.name"])' - 'set(attributes["k8s.namespace.name"], resource.attributes["k8s.namespace.name"])' - 'set(attributes["vcluster.virtual.pod"], resource.attributes["vcluster.virtual.pod"])' - 'set(attributes["vcluster.virtual.namespace"], resource.attributes["vcluster.virtual.namespace"])' - 'set(attributes["vcluster.project"], resource.attributes["vcluster.project"])' - 'set(attributes["vcluster.project.namespace"], resource.attributes["vcluster.project.namespace"])' - 'set(attributes["vcluster.user"], resource.attributes["vcluster.user"])' - 'set(attributes["vcluster.name"], resource.attributes["vcluster.name"])' filter/vcluster_only: metrics: datapoint: - 'resource.attributes["vcluster.name"] == nil' resource/add_cluster: attributes: - action: upsert key: cluster value: "{{ .Values.loft.cluster }}" memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 15 batch: send_batch_size: 10000 send_batch_max_size: 10000 timeout: 10s exporters: prometheusremotewrite: endpoint: '{{ .Values.prometheus.endpoint }}/api/v1/write' {{- if and .Values.prometheus.username .Values.prometheus.password }} auth: authenticator: basicauth/prw {{- end }} tls: insecure_skip_verify: {{ .Values.prometheus.insecure }} resource_to_telemetry_conversion: enabled: true extensions: health_check: endpoint: 0.0.0.0:13133 {{- if and .Values.prometheus.username .Values.prometheus.password }} basicauth/prw: client_auth: username: "{{ .Values.prometheus.username }}" password: "{{ .Values.prometheus.password }}" {{- end }} service: extensions: - health_check {{- if and .Values.prometheus.username .Values.prometheus.password }} - basicauth/prw {{- end }} pipelines: metrics: receivers: - prometheus processors: - memory_limiter - groupbyattrs - transform/pre_enrich - k8sattributes - filter/vcluster_only - resource/add_cluster - transform - batch exporters: - prometheusremotewrite defaultNamespace: monitoring displayName: OTEL Collector - Shared Nodes icon: https://opentelemetry.io/img/logos/opentelemetry-logo-nav.png parameters: - description: The Prometheus remote write endpoint (without /api/v1/write suffix) label: Prometheus Endpoint required: true variable: prometheus.endpoint - description: Username for basic auth (optional) label: Prometheus Username variable: prometheus.username - description: Password for basic auth (optional) label: Prometheus Password type: password variable: prometheus.password - description: Skip TLS verification for the connection to Prometheus label: Prometheus Skip TLS Verification type: boolean variable: prometheus.insecure recommendedApp: - cluster ``` Key configuration details * **Deployment mode with Target Allocator**: A Deployment with 2 replicas and `consistent-hashing` allocation is more resource-efficient than a DaemonSet. Since the `prometheus` receiver is used (not `kubeletstats`), there's no need for local-node scraping. * **`serviceMonitorSelector: app: vcluster`**: Without filtering, the Target Allocator discovers all ServiceMonitors in the cluster, overwhelming collectors with memory pressure. * **`operator.targetallocator.mtls: true`**: Each tenant cluster exposes its API server metrics over mTLS. Without this feature gate, the Target Allocator redacts TLS private keys when passing scrape configs to collectors. * **`otel/opentelemetry-collector-contrib` image**: The default image doesn't include the `prometheusremotewrite` exporter. ### Register the app[​](#register-the-app "Direct link to Register the app") Apply the App manifest to the management API so that it becomes available in the Platform UI: ``` kubectl apply -f otel-collector-shared-nodes-app.yaml ``` ### Deploy to a cluster[​](#deploy-to-a-cluster "Direct link to Deploy to a cluster") 1. Go to Infrastructure > Control Plane Clusters. 2. Click the cluster where you want to deploy the collector. 3. Navigate to the Apps tab. 4. Click Install App and click the **OTEL Collector - Shared Nodes** app. 5. Configure the following parameters and click Install. | Parameter | Required | Description | | -------------------------------- | -------- | --------------------------------------------------- | | Prometheus Endpoint | Yes | Remote write URL (without `/api/v1/write` suffix) | | Prometheus Username | No | Basic auth username | | Prometheus Password | No | Basic auth password | | Prometheus Skip TLS Verification | No | Skip TLS verification for the Prometheus connection | info Repeat these steps for each control plane cluster. ## Deploy the private nodes collector[​](#deploy-the-private-nodes-collector "Direct link to Deploy the private nodes collector") Deploy one private-nodes collector into each private-nodes tenant cluster. All vCluster identity labels are injected automatically by the Platform using `{{ .Values.loft.* }}`. ### App manifest[​](#app-manifest-1 "Direct link to App manifest") The private-nodes app deploys the [`opentelemetry-collector`](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector) Helm chart (v0.144.0) with the following configuration: otel-collector-private-nodes-app.yaml otel-collector-private-nodes-app.yaml ``` apiVersion: management.loft.sh/v1 kind: App metadata: name: otel-collector-private-nodes spec: access: - users: - '*' verbs: - get config: chart: name: opentelemetry-collector repoURL: https://open-telemetry.github.io/opentelemetry-helm-charts version: 0.144.0 values: | # yaml --- mode: daemonset image: repository: otel/opentelemetry-collector-contrib presets: kubeletMetrics: enabled: false kubernetesAttributes: enabled: true service: enabled: true # Explicitly inject node name for local-only scraping extraEnvs: - name: K8S_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName clusterRole: rules: - apiGroups: [""] resources: ["nodes", "nodes/metrics", "nodes/proxy", "services", "endpoints", "pods", "namespaces"] verbs: ["get", "list", "watch"] - apiGroups: ["apps"] resources: ["replicasets"] verbs: ["get", "list", "watch"] - apiGroups: ["extensions"] resources: ["replicasets"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["nodes/stats"] verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics", "/metrics/cadvisor"] verbs: ["get"] config: {{- if and .Values.prometheus.username .Values.prometheus.password }} extensions: basicauth/prw: client_auth: username: "{{ .Values.prometheus.username }}" password: "{{ .Values.prometheus.password }}" {{- end }} receivers: prometheus: config: scrape_configs: - job_name: 'kubelet' scrape_interval: 60s kubernetes_sd_configs: - role: node scheme: https tls_config: insecure_skip_verify: {{ .Values.kubelet.insecure }} authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: # Only scrape the node this pod is running on - source_labels: [__meta_kubernetes_node_name] regex: '${env:K8S_NODE_NAME}' action: keep - source_labels: [__meta_kubernetes_node_address_InternalIP] target_label: __address__ replacement: '$$1:10250' - action: labelmap regex: __meta_kubernetes_node_label_(.+) - source_labels: [__meta_kubernetes_node_name] target_label: node metric_relabel_configs: - source_labels: [namespace] target_label: vcluster_virtual_namespace - source_labels: [pod] target_label: vcluster_virtual_pod - job_name: 'kubelet-cadvisor' scrape_interval: 60s kubernetes_sd_configs: - role: node scheme: https tls_config: insecure_skip_verify: {{ .Values.kubelet.insecure }} authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token metrics_path: /metrics/cadvisor relabel_configs: # Only scrape the node this pod is running on - source_labels: [__meta_kubernetes_node_name] regex: '${env:K8S_NODE_NAME}' action: keep - source_labels: [__meta_kubernetes_node_address_InternalIP] target_label: __address__ replacement: '$$1:10250' - action: labelmap regex: __meta_kubernetes_node_label_(.+) - source_labels: [__meta_kubernetes_node_name] target_label: node metric_relabel_configs: - source_labels: [namespace] target_label: vcluster_virtual_namespace - source_labels: [pod] target_label: vcluster_virtual_pod - job_name: 'apiserver' scrape_interval: 60s kubernetes_sd_configs: - role: endpoints namespaces: names: ['default'] scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt authorization: credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: kubernetes;https processors: k8sattributes: auth_type: 'serviceAccount' extract: metadata: - k8s.namespace.name - k8s.pod.name - k8s.pod.start_time - k8s.pod.uid - k8s.deployment.name - k8s.node.name transform: error_mode: ignore metric_statements: - context: datapoint statements: - 'set(attributes["k8s.node.name"], resource.attributes["k8s.node.name"])' - 'set(attributes["k8s.pod.name"], resource.attributes["k8s.pod.name"])' - 'set(attributes["k8s.namespace.name"], resource.attributes["k8s.namespace.name"])' memory_limiter: check_interval: 1s limit_percentage: 75 spike_limit_percentage: 15 batch: send_batch_size: 10000 timeout: 10s exporters: prometheusremotewrite: endpoint: '{{ .Values.prometheus.endpoint }}/api/v1/write' {{- if and .Values.prometheus.username .Values.prometheus.password }} auth: authenticator: basicauth/prw {{- end }} tls: insecure_skip_verify: {{ .Values.prometheus.insecure }} # Use external_labels for loft labels (simpler than attributes processor for remote write) external_labels: cluster: "{{ .Values.loft.cluster }}" vcluster_name: "{{ .Values.loft.name }}" vcluster_project: "{{ .Values.loft.project }}" vcluster_project_namespace: "{{ .Values.loft.space }}" vcluster_user: "{{ .Values.loft.user.name }}" # Disable resource to telemetry conversion to avoid duplicate labels resource_to_telemetry_conversion: enabled: false service: extensions: - health_check {{- if and .Values.prometheus.username .Values.prometheus.password }} - basicauth/prw {{- end }} pipelines: metrics: receivers: - prometheus processors: - memory_limiter - k8sattributes - transform - batch exporters: - prometheusremotewrite defaultNamespace: monitoring description: | OpenTelemetry Collector for private/dedicated node vCluster monitoring. Deploys a DaemonSet-mode collector inside the vCluster, scraping kubelet, cAdvisor, and API server metrics from each node. displayName: OTEL Collector - Private Nodes icon: https://opentelemetry.io/img/logos/opentelemetry-logo-nav.png parameters: - description: The Prometheus endpoint to push metrics to label: Prometheus Endpoint required: true variable: prometheus.endpoint - description: The Prometheus username label: Prometheus Username variable: prometheus.username - description: The password to access Prometheus label: Prometheus Password type: password variable: prometheus.password - description: Skip TLS verification for the connection to Prometheus label: Prometheus Skip TLS Verification type: boolean variable: prometheus.insecure - defaultValue: true description: Skip kubelet certificate verification for clusters whose serving certificates use a separate CA or lack IP SANs label: Skip kubelet TLS Verification type: boolean variable: kubelet.insecure recommendedApp: - virtualcluster ``` Key configuration details * **DaemonSet mode**: Private-nodes tenant clusters have dedicated nodes. A DaemonSet ensures one collector per node, scraping only the local kubelet and cAdvisor using `${env:K8S_NODE_NAME}` filtering. No Target Allocator is needed. * **`external_labels` instead of `k8sattributes` for identity**: The collector runs inside the tenant cluster, so it can't access control plane cluster namespace labels. The Platform injects `{{ .Values.loft.* }}` template variables at deploy time. * **`resource_to_telemetry_conversion: false`**: Setting this to `true` causes duplicate labels that break Grafana dashboards. * **`metric_relabel_configs` for virtual namespace/pod**: Since `resource_to_telemetry_conversion` is disabled, OTel transform processor attributes don't reach the exported Prometheus labels. The relabel configs copy `namespace` and `pod` to `vcluster_virtual_namespace` and `vcluster_virtual_pod` at scrape time. ### Register the app[​](#register-the-app-1 "Direct link to Register the app") Apply the App manifest to the management API so that it becomes available in the Platform UI: ``` kubectl apply -f otel-collector-private-nodes-app.yaml ``` ### Deploy to a tenant cluster[​](#deploy-to-a-tenant-cluster "Direct link to Deploy to a tenant cluster") 1. Go to the Projects section using the menu on the left. 2. Click the project containing your private-nodes tenant cluster. 3. Click on the tenant cluster, then navigate to the Config tab. 4. Scroll down to the **Apps & Objects** section and add the **OTEL Collector - Private Nodes** app. 5. Configure the following parameters and click Save. | Parameter | Required | Description | | -------------------------------- | -------- | --------------------------------------------------------------------------------- | | Prometheus Endpoint | Yes | Remote write URL (without `/api/v1/write` suffix) | | Prometheus Username | No | Basic auth username | | Prometheus Password | No | Basic auth password | | Prometheus Skip TLS Verification | No | Skip TLS verification for the Prometheus connection | | Skip kubelet TLS Verification | No | Skip kubelet certificate verification for kubelet scrapes. The default is `true`. | The default skips kubelet certificate verification because kubelet serving certificates commonly use a separate certificate authority or omit node IP subject alternative names. Set **Skip kubelet TLS Verification** to `false` only when the collector can validate the kubelet serving certificate for every node address it scrapes. info Repeat these steps for each private-nodes tenant cluster. ## Golden signals queries[​](#golden-signals-queries "Direct link to Golden signals queries") With the collectors deployed and forwarding metrics to the central Prometheus, you can query the aggregated data. This section provides PromQL queries organized around the [Four Golden Signals](https://sre.google/sre-book/monitoring-distributed-systems/) of monitoring: latency, traffic, errors, and saturation. Because identity labels are enriched at collection time, every query can filter and aggregate by `cluster`, `vcluster_project`, and `vcluster_name` directly. ### Latency[​](#latency "Direct link to Latency") #### kube-apiserver request latency (p99, by verb)[​](#kube-apiserver-request-latency-p99-by-verb "Direct link to kube-apiserver request latency (p99, by verb)") ``` histogram_quantile(0.99, sum by (le, verb, cluster, vcluster_project, vcluster_name) ( rate(apiserver_request_duration_seconds_bucket{vcluster_name!=""}[5m]) ) ) ``` **Why:** Shows the tail latency of API server requests broken down by operation type (GET, LIST, PUT, POST, PATCH, DELETE, WATCH). The p99 captures outliers that averages hide. WATCH is expected to show 60s (long-poll). #### kube-apiserver request latency (p95, non-WATCH)[​](#kube-apiserver-request-latency-p95-non-watch "Direct link to kube-apiserver request latency (p95, non-WATCH)") ``` histogram_quantile(0.95, sum by (le, verb, cluster, vcluster_project, vcluster_name) ( rate(apiserver_request_duration_seconds_bucket{verb!~"WATCH|CONNECT", vcluster_name!=""}[5m]) ) ) ``` **Why:** Excludes long-running connections to focus on latency for synchronous API calls. #### etcd backend latency (p99, by operation)[​](#etcd-backend-latency-p99-by-operation "Direct link to etcd backend latency (p99, by operation)") ``` histogram_quantile(0.99, sum by (le, operation, cluster, vcluster_project, vcluster_name) ( rate(etcd_request_duration_seconds_bucket{vcluster_name!=""}[5m]) ) ) ``` **Why:** [etcd](https://etcd.io/) is the persistence backend. High latencies here (especially for `get` and `list`) propagate to every API call. ### Traffic[​](#traffic "Direct link to Traffic") #### kube-apiserver request rate (by verb)[​](#kube-apiserver-request-rate-by-verb "Direct link to kube-apiserver request rate (by verb)") ``` sum by (verb, cluster, vcluster_project, vcluster_name) ( rate(apiserver_request_total{vcluster_name!=""}[5m]) ) ``` **Why:** The most fundamental measure of cluster workload. Shows how many requests per second the API server handles, broken down by verb. #### kube-apiserver request rate (by resource)[​](#kube-apiserver-request-rate-by-resource "Direct link to kube-apiserver request rate (by resource)") ``` topk(10, sum by (resource, cluster, vcluster_project, vcluster_name) ( rate(apiserver_request_total{vcluster_name!=""}[5m]) ) ) ``` **Why:** Identifies which Kubernetes resources generate the most API traffic, revealing "hot" resource types. #### Network I/O rate (by virtual namespace)[​](#network-io-rate-by-virtual-namespace "Direct link to Network I/O rate (by virtual namespace)") ``` topk(10, sum by (vcluster_name, vcluster_virtual_namespace) ( rate(container_network_receive_bytes_total{vcluster_name!="", vcluster_virtual_namespace!=""}[5m]) ) ) ``` ``` topk(10, sum by (vcluster_name, vcluster_virtual_namespace) ( rate(container_network_transmit_bytes_total{vcluster_name!="", vcluster_virtual_namespace!=""}[5m]) ) ) ``` **Why:** Measures network throughput per namespace, revealing which workloads generate the most network traffic. #### REST client outbound request rate (by code)[​](#rest-client-outbound-request-rate-by-code "Direct link to REST client outbound request rate (by code)") ``` sum by (code, cluster, vcluster_project, vcluster_name) ( rate(rest_client_requests_total{vcluster_name!=""}[5m]) ) ``` **Why:** How many outbound API calls the control-plane components make. ### Errors[​](#errors "Direct link to Errors") #### kube-apiserver error rate (4xx/5xx, by code)[​](#kube-apiserver-error-rate-4xx5xx-by-code "Direct link to kube-apiserver error rate (4xx/5xx, by code)") ``` sum by (code, cluster, vcluster_project, vcluster_name) ( rate(apiserver_request_total{code=~"[45]..", vcluster_name!=""}[5m]) ) ``` **Why:** HTTP-level error rates broken down by status code. #### kube-apiserver error ratio (errors / total)[​](#kube-apiserver-error-ratio-errors--total "Direct link to kube-apiserver error ratio (errors / total)") ``` sum by (cluster, vcluster_project, vcluster_name) ( rate(apiserver_request_total{code=~"5..", vcluster_name!=""}[5m]) ) / sum by (cluster, vcluster_project, vcluster_name) ( rate(apiserver_request_total{vcluster_name!=""}[5m]) ) ``` **Why:** The fraction of server-side errors. A ratio above 1% is a red flag. #### etcd request errors[​](#etcd-request-errors "Direct link to etcd request errors") ``` sum by (operation, cluster, vcluster_project, vcluster_name) ( rate(etcd_request_errors_total{vcluster_name!=""}[5m]) ) ``` **Why:** Backend storage errors directly impact cluster health. #### Container OOM kills[​](#container-oom-kills "Direct link to Container OOM kills") ``` sum by (vcluster_name, vcluster_virtual_namespace, vcluster_virtual_pod) ( rate(container_oom_events_total{vcluster_name!=""}[5m]) ) ``` **Why:** Out-of-memory kills indicate resource misconfiguration. #### REST client error rate (outbound 5xx)[​](#rest-client-error-rate-outbound-5xx "Direct link to REST client error rate (outbound 5xx)") ``` sum by (host, cluster, vcluster_project, vcluster_name) ( rate(rest_client_requests_total{code=~"5..", vcluster_name!=""}[5m]) ) ``` **Why:** Errors when control-plane components call external APIs. ### Saturation[​](#saturation "Direct link to Saturation") #### Container CPU usage (top pods)[​](#container-cpu-usage-top-pods "Direct link to Container CPU usage (top pods)") ``` topk(10, sum by (vcluster_name, vcluster_virtual_namespace, vcluster_virtual_pod) ( rate(container_cpu_usage_seconds_total{vcluster_name!="", container!="", vcluster_virtual_pod!=""}[5m]) ) ) ``` **Why:** Shows the most CPU-hungry pods across the fleet. #### Container memory working set (top pods)[​](#container-memory-working-set-top-pods "Direct link to Container memory working set (top pods)") ``` topk(10, sum by (vcluster_name, vcluster_virtual_namespace, vcluster_virtual_pod) ( container_memory_working_set_bytes{vcluster_name!="", container!="", vcluster_virtual_pod!=""} ) ) ``` **Why:** Working set is the "real" memory usage that matters for OOM decisions. #### CPU throttling ratio (by pod)[​](#cpu-throttling-ratio-by-pod "Direct link to CPU throttling ratio (by pod)") ``` topk(10, sum by (vcluster_name, vcluster_virtual_namespace, vcluster_virtual_pod) ( rate(container_cpu_cfs_throttled_periods_total{vcluster_name!="", vcluster_virtual_pod!=""}[5m]) ) / sum by (vcluster_name, vcluster_virtual_namespace, vcluster_virtual_pod) ( rate(container_cpu_cfs_periods_total{vcluster_name!="", vcluster_virtual_pod!=""}[5m]) ) ) ``` **Why:** Shows which pods are being throttled by cgroup CPU limits. #### kube-apiserver inflight requests[​](#kube-apiserver-inflight-requests "Direct link to kube-apiserver inflight requests") ``` apiserver_current_inflight_requests{vcluster_name!=""} ``` **Why:** Shows current request concurrency for mutating vs read-only. When this approaches flow control limits, requests start queuing. #### kube-apiserver flow-control queue depth[​](#kube-apiserver-flow-control-queue-depth "Direct link to kube-apiserver flow-control queue depth") ``` sum by (priority_level, cluster, vcluster_project, vcluster_name) ( apiserver_flowcontrol_current_inqueue_requests{vcluster_name!=""} ) ``` **Why:** Requests waiting in priority-level queues. Non-zero means the API server is saturated for that priority level. #### Workqueue depth (by queue name)[​](#workqueue-depth-by-queue-name "Direct link to Workqueue depth (by queue name)") ``` topk(10, workqueue_depth{vcluster_name!=""} ) ``` **Why:** Controller work queues. Growing depth means controllers can't keep up with the event rate. ## Grafana dashboards[​](#grafana-dashboards "Direct link to Grafana dashboards") note This section assumes Grafana is already deployed. For setup instructions, see the [Deploy Grafana](https://vcluster.com/docs/platform/maintenance/monitoring/aggregating-metrics.md#deploy-grafana) section in Aggregating Metrics. Two Grafana dashboards are provided for visualizing metrics collected by the OTel Collectors. Both dashboards use the identity labels enriched at collection time, so all panels use straightforward PromQL queries without joins. ### Import a dashboard[​](#import-a-dashboard "Direct link to Import a dashboard") 1. Download the dashboard JSON file from the relevant section below. 2. Open Grafana and navigate to **Dashboards**. 3. Click **New** > **Import**. 4. Upload the `.json` file or paste its contents. 5. Select your Prometheus data source and click **Import**. ### vCluster projects dashboard[​](#vcluster-projects-dashboard "Direct link to vCluster projects dashboard") A platform admin overview of vCluster projects across Shared Nodes and Private Nodes tenant clusters. Use this dashboard to monitor resource consumption and API health at the project level. **Dashboard JSON file:** [Download dashboard JSON](https://vcluster.com/docs/assets/files/vcluster-projects-1232c32b1e4e2578ed45ef480cedc1db.json) **Template variables:** | Variable | Description | | ------------ | ----------------------------------------- | | `datasource` | Prometheus data source | | `cluster` | Filter by cluster (supports multi-select) | | `project` | Filter by project (supports multi-select) | **Panels:** | Section | Panels | | ------------------------- | ---------------------------------------------------------------------------------------------------------------- | | Project summary | Total Projects, Total tenant clusters, Total Pods, Total CPU (cores), Total Memory, Max Error Rate (stat panels) | | Resource usage by project | CPU by Project (cores), Memory by Project, Pods by Project, Tenant clusters by Project (bar charts) | | API health by project | API Request Rate, API Error Rate (5xx), API P95 Latency (time series) | | Resource trends | CPU Usage Trend (stacked), Memory Usage Trend (stacked) (time series) | ### vCluster detail dashboard[​](#vcluster-detail-dashboard "Direct link to vCluster detail dashboard") A detailed drill-down view for individual tenant clusters. Use this dashboard to investigate resource consumption and control plane health for a specific tenant cluster. **Dashboard JSON file:** [Download dashboard JSON](https://vcluster.com/docs/assets/files/vcluster-detail-a4c1ceb01e5640a2d4e87773c41f7ab3.json) **Template variables:** | Variable | Description | | ------------ | ----------------------- | | `datasource` | Prometheus data source | | `cluster` | Filter by cluster | | `project` | Filter by project | | `vcluster` | Filter by vCluster name | **Panels:** | Section | Panels | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | vCluster overview | Pods, Namespaces, Total CPU (cores), Total Memory, API Request Rate, API Error Rate (stat panels) | | API server | API Request Rate by Resource, API Latency P95 by Verb, API Errors by Status Code, In-Flight Requests, Long-Running Requests (CONNECT) (time series) | | CPU | CPU by Virtual Namespace (stacked), CPU by Pod (top 15), CPU Throttling by Pod (ratio), CPU by Node (time series) | | Memory | Memory by Virtual Namespace (stacked), Memory by Pod (top 15), RSS Memory by Pod (top 15), OOM Events by Pod (time series) | | Network | Network Receive by Pod (top 10), Network Transmit by Pod (top 10), Network Packet Drops, Network Errors (time series) | | Disk I/O | Disk Read by Pod (top 10), Disk Write by Pod (top 10) (time series) | | Filesystem | Filesystem Usage vs Limit, Filesystem I/O Throughput (time series) | --- # Metrics Platform process logging To configure log level or output format for the platform pod, see [Platform Process Logging](https://vcluster.com/docs/platform/maintenance/logging/platform-logging.md). ## Metrics[​](#metrics "Direct link to Metrics") vCluster Platform exposes several [Prometheus-conformant](https://prometheus.io/docs/practices/naming/) metrics that can be scraped. ### Create the Prometheus servicemonitor[​](#create-the-prometheus-servicemonitor "Direct link to Create the Prometheus servicemonitor") The vCluster Platform chart includes an optional `ServiceMonitor`. Install the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) before enabling it. The chart creates the `ServiceMonitor` in the Platform release namespace by default. Modify the following with your specific values to generate a copyable command: RELEASE\_NAMEvcluster-platform RELEASE\_NAMESPACEvcluster-platform ``` helm upgrade vcluster-platform vcluster-platform --repo https://charts.loft.sh/ \ --namespace vcluster-platform \ --reuse-values \ --set serviceMonitor.enabled=true ``` If Prometheus discovers `ServiceMonitor` resources in another namespace, also set `serviceMonitor.namespace`. Add any labels required by the Prometheus `serviceMonitorSelector` through `serviceMonitor.labels`. note When `serviceMonitor.enabled=true`, the platform exposes metrics on port `8080`. If your environment has firewall rules or network policies, make sure that port `8080` is open for your Prometheus server to scrape metrics. ### Access the metrics endpoint directly without a ServiceMonitor[​](#access-the-metrics-endpoint-directly-without-a-servicemonitor "Direct link to Access the metrics endpoint directly without a ServiceMonitor") When collecting metrics directly without a ServiceMonitor, it's important to understand that vCluster Platform typically runs with multiple replicas for high availability. To get complete metrics coverage, you need to collect metrics from all platform pods rather than through a single endpoint. #### Collect metrics from all platform pods[​](#collect-metrics-from-all-platform-pods "Direct link to Collect metrics from all platform pods") Each vCluster Platform pod exposes metrics on port `8080` at the `/metrics` endpoint. Your metrics collection tool should scrape metrics from all replicas to ensure complete coverage. The Platform chart applies these labels to its pods: * `app=loft` identifies vCluster Platform pods. * `release=` identifies the Helm release. The default release name is `vcluster-platform`. The chart doesn't apply a `component` label to Platform pods by default. By default, vCluster Platform requires an `Authorization` HTTP header field with a Kubernetes bearer token. The token must have access to the non-resource URL `/metrics` in the Kubernetes cluster where vCluster Platform is installed. If the header isn't provided, vCluster Platform denies the request. Turn off metrics authentication To scrape metrics without authentication, turn it off using the environment variable `INSECURE_METRICS=true` in the vCluster Platform Helm chart. Example of accessing metrics from a specific pod using port-forward: ``` # Port-forward to a specific platform pod kubectl port-forward -n vcluster-platform loft-xxxx 8080:8080 # Access metrics with authentication token curl http://localhost:8080/metrics -H "Authorization: Bearer eyJhbGci..." # HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend. # TYPE apiserver_audit_event_total counter apiserver_audit_event_total 0 # HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend. # TYPE apiserver_audit_requests_rejected_total counter apiserver_audit_requests_rejected_total 0 ... ``` ### Which metrics are exposed?[​](#which-metrics-are-exposed "Direct link to Which metrics are exposed?") vCluster Platform collects metrics from its internal components and merges them together into a single metrics endpoint at `/metrics`. The following component metrics are merged together: * **API gateway metrics**: vCluster Platform exposes metrics for its internal API gateway, such as the total amount of requests (`apigateway_ENDPOINT_request_total`), request latency (`apigateway_ENDPOINT_request_duration_seconds`), and request sizes (`apigateway_ENDPOINT_response_sizes`) for the different endpoints. The metrics have different labels depending on the endpoint. The following endpoints and metrics are available: * **ui**: all requests that target vCluster Platform UI assets (metrics are `apigateway_ui_request_total`, `apigateway_ui_request_duration_seconds`, `apigateway_ui_response_sizes`) * **auth**: all requests that target vCluster Platform authentication endpoints (such as login and token refresh) * **kubernetes**: all forwarded requests that target a Kubernetes cluster. The targeted backend is provided by a label with one of three values. `management` targets the integrated vCluster Platform Kubernetes API server. `cluster` targets a request forwarded to a connected cluster. `vcluster` targets a tenant cluster deployed by vCluster Platform. Further labels include Kubernetes resource, API version, API group, and subresource. * **oidc**: all requests that target the internal OIDC server * **Integrated Kubernetes API server metrics**: all internal Kubernetes API server metrics are exposed. An incomplete list can be found in the [Kubernetes repository](https://github.com/kubernetes/apiserver/blob/master/pkg/endpoints/metrics/metrics.go). * **Integrated controller manager metrics**: all vCluster Platform management cluster controller metrics are exposed. * **Go runtime metrics**: some general metrics about the Go runtime and resource usage are exposed. --- # Monitoring Use [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md) for fleet-wide workload metrics. It provides authenticated ingestion, trusted identity labels, scoped queries, and GPU observability through vCluster Platform. You can also scrape the Platform's own Prometheus metrics. The older OpenTelemetry pages in this section remain as reference architectures for deployments that can't use Fleet Observability. They aren't Platform-managed monitoring features. ## Workload metrics[​](#workload-metrics "Direct link to Workload metrics") | If you want to | Use | | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | Monitor one connected cluster with Prometheus and Grafana | [Single-cluster monitoring](#single-cluster-monitoring) | | Monitor workloads across a fleet | [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md) | | Query metrics with trusted project and tenant identity | [Fleet Observability labels](https://vcluster.com/docs/platform/maintenance/observability/labels-reference.md) | | Monitor Platform components | [Platform health metrics](#platform-health-metrics) | | Understand older collector architectures | [Fleet metrics architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md) | ### Single-cluster monitoring[​](#single-cluster-monitoring "Direct link to Single-cluster monitoring") For a single connected cluster, install a current `kube-prometheus-stack` Helm chart from the Platform UI. The chart deploys Prometheus, Grafana, the Prometheus Operator, and cluster exporters. Check the [upstream releases](https://github.com/prometheus-community/helm-charts/releases?q=kube-prometheus-stack) for the latest chart version and confirm it supports your Kubernetes version before installing. 1. Go to Infrastructure > Control Plane Clusters and select a cluster. 2. Navigate to the **Apps** tab. 3. Click **Install App** and configure the Helm chart with the following settings. Set **Chart Version** to the version you verified in the previous step. | Setting | Value | | -------------------- | ---------------------------------------------------- | | Chart Repository URL | `https://prometheus-community.github.io/helm-charts` | | Chart Name | `kube-prometheus-stack` | | Chart Version | `` | | Namespace | `kube-prometheus-stack` | | Release Name | `kube-prometheus-stack` | 4. Review the [upstream chart configuration](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#configuration) for storage, retention, resource requests, and ingress. 5. Click **Install**. After installation, you have a complete monitoring setup with Prometheus scraping cluster metrics and Grafana for visualization. If your tenant clusters run in Shared Nodes mode, see [Prometheus node metrics on shared nodes](https://vcluster.com/docs/platform/maintenance/monitoring/prometheus-shared-nodes.md) to configure kubelet scraping. warning Don't use `kube-prometheus-stack` if you want to aggregate metrics across multiple tenant clusters. Use [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md) instead. ## Reference architectures[​](#reference-architectures "Direct link to Reference architectures") The following guides predate Fleet Observability. Use them only when you need to own the collector and backend configuration directly: * [Fleet monitoring with OpenTelemetry](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-monitoring-otel.md) covers Shared Nodes and Private Nodes collector examples, PromQL queries, and Grafana dashboards. * [Aggregating metrics with OpenTelemetry](https://vcluster.com/docs/platform/maintenance/monitoring/aggregating-metrics.md) shows a smaller collector-to-Prometheus example. * [Prometheus node metrics on shared nodes](https://vcluster.com/docs/platform/maintenance/monitoring/prometheus-shared-nodes.md) explains kubelet discovery and scraping behavior in the Shared Nodes tenancy model. These architectures use self-reported labels and don't provide Platform-enforced access scoping. See [Fleet metrics architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md) before using them. ## Platform health metrics[​](#platform-health-metrics "Direct link to Platform health metrics") vCluster Platform exposes [Prometheus-conformant metrics](https://vcluster.com/docs/platform/maintenance/monitoring/metrics.md) from its internal components, including the API gateway, integrated Kubernetes [API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/), controller manager, and Go runtime. These metrics cover request counts, latency, and error rates for all platform operations. Use a [Prometheus `ServiceMonitor`](https://vcluster.com/docs/platform/maintenance/monitoring/metrics.md#create-the-prometheus-servicemonitor) to scrape these metrics automatically, or [access the `/metrics` endpoint directly](https://vcluster.com/docs/platform/maintenance/monitoring/metrics.md#access-the-metrics-endpoint-directly-without-a-servicemonitor). For platform pod log level and output format, see [Platform Process Logging](https://vcluster.com/docs/platform/maintenance/logging/platform-logging.md). For workload log collection, platform audit logs, and platform process logs, see [Logging](https://vcluster.com/docs/platform/maintenance/logging/overview.md). --- # Prometheus node metrics on shared nodes In the Shared Nodes tenancy model, vCluster virtualizes node IP addresses by default as part of its kubelet proxying feature. Prometheus requires specific configuration to scrape kubelet and cAdvisor metrics in this environment. This doesn't affect Private Nodes tenant clusters, which have dedicated real nodes with directly reachable IPs. ## How it works[​](#how-it-works "Direct link to How it works") vCluster enables kubelet proxying by default (`networking.advanced.proxyKubelets.byIP: true` and `byHostname: true` in the Helm values). This feature lets vCluster intercept kubelet traffic and rewrite pod names and namespaces from control plane cluster values to their virtual equivalents. As part of this, vCluster assigns each synced node a virtual identity: * With `byIP: true`, vCluster replaces the node's IP address with the ClusterIP of a per-node Kubernetes Service in the vCluster namespace on the control plane cluster. * With `byHostname: true`, vCluster replaces the node hostname with a synthetic name like `.nodes.vcluster.com`. Prometheus discovers scrape targets by reading node objects from the tenant cluster's Kubernetes API. It then attempts to connect to those virtual addresses. If the per-node service ClusterIP isn't routable from where Prometheus runs, the scrape times out. This most commonly happens when Prometheus is outside the tenant cluster or on a separate connected cluster. ## Approaches[​](#approaches "Direct link to Approaches") The right approach depends on where Prometheus runs: * **Prometheus outside the tenant cluster** (on the control plane cluster or a central monitoring instance): use [Option 1](#option-1-otel-fleet-monitoring-reference-architecture). * **Prometheus inside the tenant cluster**: use [Option 2](#option-2-api-server-proxy-scraping-prometheus-inside-the-tenant-cluster). * **Specific requirements that prevent Options 1 or 2**: use [Option 3](#option-3-disable-kubelet-proxying). info [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md)'s bundled collector doesn't currently scrape node and cAdvisor metrics from Shared Nodes tenant clusters. Its self-scrape pattern only authenticates against the host kubelet on Private Nodes tenant clusters. Use Option 1 for node and cAdvisor metrics on Shared Nodes until that gap closes. ### Option 1: OTel fleet monitoring (reference architecture)[​](#option-1-otel-fleet-monitoring-reference-architecture "Direct link to Option 1: OTel fleet monitoring (reference architecture)") Deploy an OpenTelemetry Collector on the control plane cluster. The collector scrapes cAdvisor and kubelet metrics using the control plane cluster's real node IPs, then pushes enriched metrics to a central Prometheus using `remote_write`. Because the collector operates at the control plane cluster level, it never reads tenant cluster node objects or depends on virtual node IPs. This approach also adds self-reported identity labels such as `vcluster_name` and `vcluster_project` to every metric. It doesn't provide the authenticated labels or scoped read access available through Fleet Observability. See [Fleet monitoring with OpenTelemetry](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-monitoring-otel.md) for a complete setup guide covering both Shared Nodes and Private Nodes tenancy models. ### Option 2: API server proxy scraping (Prometheus inside the tenant cluster)[​](#option-2-api-server-proxy-scraping-prometheus-inside-the-tenant-cluster "Direct link to Option 2: API server proxy scraping (Prometheus inside the tenant cluster)") Configure Prometheus to route kubelet and cAdvisor scrapes through the Kubernetes API server proxy instead of connecting to node IPs directly. vCluster intercepts requests that go through the API server and forwards them to the real kubelet on the control plane cluster. It returns them with pod names and namespaces already rewritten to their virtual equivalents, so no additional relabeling is needed. Configure the cAdvisor and kubelet jobs to target the API server and set the metrics path to the node proxy endpoint: Prometheus scrape config using the API server proxy ``` scrape_configs: - job_name: cadvisor scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor - job_name: kubelet scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/$1/proxy/metrics ``` ### Option 3: Disable kubelet proxying[​](#option-3-disable-kubelet-proxying "Direct link to Option 3: Disable kubelet proxying") Setting `byHostname` and `byIP` to `false` causes vCluster to pass through real node addresses from the control plane cluster unchanged. Prometheus can then discover and scrape them directly. vcluster.yaml ``` networking: advanced: proxyKubelets: byHostname: false byIP: false ``` info With kubelet proxying disabled: * **vCluster doesn't rewrite metrics.** Pod and namespace labels reflect the control plane cluster's internal naming, not the tenant cluster's virtual names. On a shared node, metrics from different tenant clusters have overlapping pod/namespace label values. * **Real node IPs are visible** in the tenant cluster's node objects, which reveals control plane network topology to tenant cluster users. --- # Configure edge collectors Edge collectors are OpenTelemetry Collectors that scrape or receive metrics near the workloads, then send OTLP to the Write Gateway. Each collector request must include a bearer token and scope headers that match a `metrics-writer` access key. Deploy collectors once the gateway is running and a `metrics-writer` key is in place, as described in [Install the observability gateway](https://vcluster.com/docs/platform/maintenance/observability/install-observability-gateway.md) and [Configure metrics access](https://vcluster.com/docs/platform/maintenance/observability/configure-metrics-access.md). ## Deploy collectors with Argo CD[​](#deploy-collectors-with-argo-cd "Direct link to Deploy collectors with Argo CD") Collectors run on every tenant cluster and control plane cluster you want metrics from. If you use the [Argo CD integration](https://vcluster.com/docs/platform/integrations/argocd.md), deploy one `cluster-collector` `ArgoCDApplication` to each cluster instead of configuring the OpenTelemetry Collector by hand. vCluster Standalone tenant clusters The out-of-the-box collector path relies on the Argo CD integration, which vCluster Standalone tenant clusters can't use. There is no bundled collector for them. Configure a collector manually instead, as described below. Each collector authenticates to the gateway with a `metrics-writer` bearer token. Create a Secret named `otel-otlp-auth` in the collector namespace before you deploy, or pass the token through the `bearerToken` parameter. otel-otlp-auth.yaml ``` apiVersion: v1 kind: Secret metadata: name: otel-otlp-auth namespace: observability type: Opaque stringData: authorization: "Bearer REPLACE_WITH_METRICS_WRITER_KEY" ``` Create the writer key first, as described in [Create a writer key for a tenant cluster](https://vcluster.com/docs/platform/maintenance/observability/configure-metrics-access.md#create-a-writer-key-for-a-tenant-cluster) or [Create a writer key for a control plane cluster](https://vcluster.com/docs/platform/maintenance/observability/configure-metrics-access.md#create-a-writer-key-for-a-control-plane-cluster), then deploy the collectors. The walkthrough below deploys to a tenant cluster. For a control plane cluster, swap `destination.cluster` in for `destination.virtualCluster`, as described at the end of this section. * Platform UI * YAML Platform creates a namespace named `p-` for each project, so the default project's namespace is `p-default`. Create the `ArgoCDApplication` object in your project namespace: fleet-edge-collectors.yaml ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplication metadata: name: fleet-edge-collectors namespace: p-my-project # replace with your project namespace spec: displayName: "Fleet edge collectors" destination: virtualCluster: name: my-vcluster # replace with your tenant cluster name target: vCluster # deploy into the tenant cluster templateRef: name: cluster-collector parameters: destinationNamespace: observability otlpEndpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318 project: default ``` 1. Select your project from the projects dropdown at the top of the left navigation bar. 2. Navigate to Tenant Clusters and open the tenant cluster you want to collect from. 3. Select the ArgoCD Apps tab and click Add Argo CD Application. 4. In the Basics section, enter a Display Name such as `Fleet edge collectors`. 5. Select Deploy to tenant cluster to deploy into the tenant cluster itself. 6. From the Argo CD Application Template dropdown, select the Fleet Observability: Cluster Collector template. 7. Fill the Template Parameters. Set destinationNamespace to `observability` and otlpEndpoint to the Write Gateway endpoint. Use the in-cluster Service endpoint for collectors in the Platform cluster, or the gateway LoadBalancer address for collectors in a tenant cluster elsewhere. 8. Click Save to add the Argo CD Application to the tenant cluster. | Parameter | Default | Description | | ------------------------ | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `destinationNamespace` | `observability` | Namespace for the collectors. | | `project` | `default` | Argo CD project that owns the application. | | `otlpEndpoint` | Fleet Write Gateway Service | OTLP HTTP endpoint the collectors push to. Use the in-cluster Service endpoint (`:4318`) for collectors in the Platform cluster, or the gateway LoadBalancer address (`https://:4318`) for collectors elsewhere. See [Expose the gateway with a LoadBalancer](https://vcluster.com/docs/platform/maintenance/observability/install-observability-gateway.md#expose-the-gateway-with-a-loadbalancer). | | `bearerToken` | *(empty)* | `metrics-writer` bearer token. When empty, the collectors read the `otel-otlp-auth` Secret. | | `otlpCaData` | *(empty)* | PEM CA data used to verify the gateway serving certificate. | | `otlpInsecureSkipVerify` | `false` | Skip gateway TLS verification. Use only for test gateways. | | `otelDaemonResources` | *(empty)* | Resource requests and limits for the `otel-daemon` collector container, as a multiline YAML block. See the example below. When empty, the chart default applies, which sets no requests or limits. | | `otelDeployResources` | *(empty)* | Resource requests and limits for the `otel-deploy` collector container, in the same format as `otelDaemonResources`. When empty, the chart default applies, which sets no requests or limits. | Both resource parameters are multiline parameters that take the contents of a Kubernetes `resources` field. fleet-edge-collectors.yaml ``` spec: parameters: otelDaemonResources: | requests: cpu: 100m memory: 256Mi limits: memory: 256Mi otelDeployResources: | requests: cpu: 200m memory: 512Mi limits: memory: 512Mi ``` The collectors stamp the request scope from the destination, so target a tenant cluster for tenant-scoped writes or a control plane cluster for cluster-scoped writes. Set `destination.cluster` instead of `destination.virtualCluster` to target a control plane cluster. Don't set `vcluster_platform_*` labels in collector configuration; the Write Gateway owns that namespace. See [Protect platform labels](#protect-platform-labels) for the label rules, and [Gateway endpoints](#gateway-endpoints) below for the scope headers the template sets on your behalf. On a cluster with the GPU stack deployed, this template also scrapes GPU metrics without extra configuration. See [The cluster-collector template scrapes GPU metrics automatically](https://vcluster.com/docs/platform/maintenance/observability/gpu-observability-templates.md#the-cluster-collector-template-scrapes-gpu-metrics-automatically). ## Configure collectors manually[​](#configure-collectors-manually "Direct link to Configure collectors manually") To configure a collector yourself instead of using the bundled template, or to customize behavior the template doesn't expose as a parameter, wire it up directly. ### Gateway endpoints[​](#gateway-endpoints "Direct link to Gateway endpoints") Inside the Platform cluster, the fleet gateway Service is: ``` gateway-fleet-observability.vcluster-platform.svc.cluster.local ``` This endpoint assumes the example connector ID `fleet-observability`. For another ID, replace the Service name with `gateway-`. Use this OTLP endpoint: | Protocol | Endpoint | | --------- | ------------------------------------------------------------------------------ | | OTLP HTTP | `https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318` | The examples on this page use OTLP HTTP on port `4318`. The gateway also accepts OTLP gRPC on port `4317`, but the bundled OpenTelemetry Collector sends over HTTP, so use `4318`. The gateway always uses TLS. The OpenTelemetry Collector `otlp_http` exporter accepts the base URL shown above. Direct OTLP HTTP clients send metrics to the `/v1/metrics` path on that host. Mount the CA that issued the gateway serving certificate into the collector and set the exporter `tls.ca_file`. The default Platform-managed certificate stores its trust anchor in `ca.crt` in the `gateway--tls` Secret. Collectors outside the Platform cluster can't resolve the in-cluster Service DNS name. Expose the gateway with a LoadBalancer, then use the LoadBalancer address in the exporter endpoint instead of the Service name, for example `https://:4318`. See [Expose the gateway with a LoadBalancer](https://vcluster.com/docs/platform/maintenance/observability/install-observability-gateway.md#expose-the-gateway-with-a-loadbalancer) to enable it and retrieve the address. The exporter `endpoint` and TLS settings in the examples below are otherwise unchanged; only the host differs. ### Store the writer key[​](#store-the-writer-key "Direct link to Store the writer key") Create a Secret for the collector, using a `metrics-writer` access key from [Create a writer key for a tenant cluster](https://vcluster.com/docs/platform/maintenance/observability/configure-metrics-access.md#create-a-writer-key-for-a-tenant-cluster) or [Create a writer key for a control plane cluster](https://vcluster.com/docs/platform/maintenance/observability/configure-metrics-access.md#create-a-writer-key-for-a-control-plane-cluster): metrics-writer-token.yaml ``` apiVersion: v1 kind: Secret metadata: name: metrics-writer-token namespace: observability type: Opaque stringData: token: REPLACE_WITH_METRICS_WRITER_KEY ``` Mount or project this Secret into your collector pod and expose it as an environment variable or file. ### Tenant cluster collector[​](#tenant-cluster-collector "Direct link to Tenant cluster collector") Use tenant scope headers when a collector writes metrics for a specific tenant cluster. otel-collector-tenant-exporter.yaml ``` exporters: otlp_http/fleet: endpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318 tls: ca_file: /etc/otel-gateway-tls/ca.crt headers: Authorization: Bearer ${env:METRICS_WRITER_TOKEN} X-Vcluster-Platform-Project: my-project X-Vcluster-Platform-Instance: my-vcluster service: pipelines: metrics: receivers: [prometheus] processors: [batch] exporters: [otlp_http/fleet] ``` The `metrics-writer` key must include a matching `spec.scope.virtualClusters` entry. ### Control plane cluster collector[​](#control-plane-cluster-collector "Direct link to Control plane cluster collector") Use cluster scope headers when a collector writes metrics for a control plane cluster. otel-collector-cluster-exporter.yaml ``` exporters: otlp_http/fleet: endpoint: https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:4318 tls: ca_file: /etc/otel-gateway-tls/ca.crt headers: Authorization: Bearer ${env:METRICS_WRITER_TOKEN} X-Vcluster-Platform-Cluster: local-cluster service: pipelines: metrics: receivers: [prometheus] processors: [batch] exporters: [otlp_http/fleet] ``` The `metrics-writer` key must include a matching `spec.scope.clusters` entry. ### Scrape custom targets[​](#scrape-custom-targets "Direct link to Scrape custom targets") Use the collector's `prometheus` receiver to scrape exporters such as kubelet, cAdvisor, DCGM exporter, or application endpoints. The gateway doesn't require a specific scrape receiver. It only requires OTLP metrics with valid authentication and scope. Example DCGM scrape job: dcgm-prometheus-receiver.yaml ``` receivers: prometheus: config: scrape_configs: - job_name: dcgm-exporter kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name] action: keep regex: dcgm-exporter - source_labels: [__meta_kubernetes_pod_ip] target_label: __address__ replacement: ${1}:9400 ``` ## Protect platform labels[​](#protect-platform-labels "Direct link to Protect platform labels") Don't set labels with the `vcluster_platform_*` prefix in edge collectors. The Write Gateway strips this namespace from incoming resource, scope, and datapoint attributes before stamping authenticated labels. Use your own label prefix for application or infrastructure dimensions. ## Verify writes[​](#verify-writes "Direct link to Verify writes") Check the Write Gateway metrics endpoint: ``` CONNECTOR_ID=fleet-observability GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID" -o jsonpath='{.items[0].metadata.name}') kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8889:8889 curl http://127.0.0.1:8889/metrics ``` Then query through the Query Proxy as described in [Query fleet metrics](https://vcluster.com/docs/platform/maintenance/observability/query-fleet-metrics.md). --- # Configure metrics access Fleet Observability uses scoped access keys to separate write access from read access. The gateway never trusts the metrics payload to define scope. Each request is authenticated with Platform and authorized against the requested cluster or tenant cluster. ## Access roles[​](#access-roles "Direct link to Access roles") | Role | Used by | Allows | | ---------------- | ----------------------------------- | -------------------------------------------------------------------- | | `metrics-writer` | Edge collectors | Write metrics for the access key's scoped cluster or tenant cluster. | | `metrics-reader` | Grafana, PromQL clients, automation | Read metrics for the access key's scoped cluster or tenant cluster. | The roles are intentionally separate. A `metrics-writer` key can't read metrics, and a `metrics-reader` key can't write metrics. A key can carry both roles when an integration needs both paths. ## Scope choices[​](#scope-choices "Direct link to Scope choices") Scope metrics keys as tightly as possible: * Use `spec.scope.clusters` for control plane cluster metrics. * Use `spec.scope.virtualClusters` for tenant cluster metrics. * Use `*` only for trusted fleet-wide automation. The Platform authorization check still applies the permissions of the user or subject that owns the key. Scope narrows permission. It doesn't expand it. Examples are blueprints, not prescriptions The snippets below are one example of how to create access keys. In particular, the line that generates the `KEY` value shows just one way to produce a random secret. Use whatever key generation, secret storage, and provisioning workflow fits your environment. Treat these commands as general blueprints rather than the required procedure. ## Create a writer key for a tenant cluster[​](#create-a-writer-key-for-a-tenant-cluster "Direct link to Create a writer key for a tenant cluster") Create one `metrics-writer` key for each tenant cluster collector, or for each automation unit that writes on behalf of one tenant cluster. Set `PROJECT` and `TENANT_CLUSTER` to the target tenant cluster's project and instance name. The `KEY` value is a freshly generated secret, and the command prints it as `token: ` at the end. Modify the following with your specific values to generate a copyable command: PROJECTdefault TENANT\_CLUSTERmy-vcluster ``` PROJECT=default TENANT_CLUSTER=my-vcluster KEY=$(head -c 20 /dev/urandom | od -An -tx1 | tr -d ' \n') kubectl apply -f - <` for each project, so the default project's namespace is `p-default`. Create the `ArgoCDApplication` object in your project namespace: fleet-metrics-backend.yaml ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplication metadata: name: fleet-metrics-backend namespace: p-my-project # replace with your project namespace spec: displayName: "Fleet metrics backend" destination: cluster: name: loft-cluster # the reserved name for Platform's local cluster; don't change this templateRef: name: fleet-observability-prometheus parameters: destinationNamespace: vcluster-platform prometheusStorageSize: 20Gi project: default ``` 1. Go to Infrastructure > Control Plane Clusters, then click the local cluster (named `loft-cluster` by default) to open its detail page. 2. Select the ArgoCD Apps tab and click Add Argo CD Application. 3. In the Basics section, enter a Display Name such as `Fleet metrics backend`. 4. From the Argo CD Application Template dropdown, select the Fleet Observability: Prometheus template. 5. Fill the Template Parameters. Set destinationNamespace to the namespace Platform is installed in, which defaults to `vcluster-platform`, and adjust the storage size or chart version if needed. 6. Click Create Argo CD Application. | Parameter | Default | Description | | ------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `destinationNamespace` | `vcluster-platform` | Namespace for the Prometheus backend. | | `project` | `default` | Argo CD project that owns the application. | | `repoURL` | Prometheus community Helm repo | Helm repository for the Prometheus chart. | | `version` | `29.14.0` | Prometheus Helm chart version. | | `prometheusStorageSize` | `20Gi` | Persistent volume size for Prometheus storage. | | `metricsBackendTLSSecret` | *(empty)* | Existing mTLS Secret for the backend. When empty, Platform self-signs a Secret named `metrics-backend-mtls`. | | `resources` | *(empty)* | Resource requests and limits for the Prometheus server container, as a multiline YAML block. See the example below. When empty, the chart default applies, which sets no requests or limits. | `resources` is a multiline parameter that takes the contents of a Kubernetes `resources` field. fleet-metrics-backend.yaml ``` spec: parameters: resources: | requests: cpu: 500m memory: 2Gi limits: memory: 2Gi ``` The connector you create below must use these endpoints when the template is deployed in its default namespace: ``` stringData: metricsBackendOtlpEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local/api/v1/otlp metricsBackendPromQLEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local metricsBackendCertSecretName: metrics-backend-mtls ``` The `/api/v1/otlp` path is required for OTLP ingestion. If you change `destinationNamespace`, replace `vcluster-platform` in both endpoints. If you supply `metricsBackendTLSSecret`, use that Secret name instead of `metrics-backend-mtls`. If `destinationNamespace` differs from the namespace where Platform is installed, Prometheus and the gateway no longer share a namespace, so the auto-generated mTLS Secret can't serve both sides on its own. See [Configure backend mTLS across namespaces or clusters](#configure-backend-mtls-across-namespaces-or-clusters). ## Create the connector Secret[​](#create-the-connector-secret "Direct link to Create the connector Secret") Create the Fleet Observability connector in the Platform namespace. Use no more than 51 lowercase letters, numbers, or hyphens for the connector ID, like the `fleet-observability` example. That keeps every name Platform derives from it predictable: the gateway resources as `gateway-`, the managed certificate Secret as `gateway--tls` (see [Customize the gateway serving certificate](#customize-the-gateway-serving-certificate)), and the `app.kubernetes.io/instance` label as the connector ID itself. A longer or dotted connector ID is a technically valid Secret name, but Platform normalizes the derived names inconsistently, so they stop matching each other. Stick to a short, dot-free ID instead of working around this. The `loft.sh/connector-type: observability` label is required. Platform only recognizes and reconciles a Secret as an observability connector when this label is present. Mark the connector as the fleet default with the `platform.vcluster.com/fleet-observability-connector` label set to `"true"`. Only one connector can be the fleet default. * Platform UI * YAML fleet-observability-connector.yaml ``` apiVersion: v1 kind: Secret metadata: name: fleet-observability namespace: vcluster-platform labels: loft.sh/connector-type: observability platform.vcluster.com/fleet-observability-connector: "true" type: Opaque stringData: metricsBackendOtlpEndpoint: https://metrics.example.com/otlp metricsBackendPromQLEndpoint: https://metrics.example.com ``` Apply the Secret: ``` kubectl apply -f fleet-observability-connector.yaml ``` 1. Click Connectors and select the Observability tab. 2. Click Connect Observability. 3. In the Display name field, enter a human-readable name for the connector. The Observability Connector ID is auto-generated from the display name. The connector ID determines the gateway resource names. The first observability connector is automatically marked as the fleet default. 4. In the Metrics Backend OTLP Endpoint field, enter the OTLP endpoint that receives metrics from the Write Gateway. 5. In the Metrics Backend PromQL Endpoint field, enter the Prometheus HTTP API endpoint queried by the Query Proxy. 6. Optionally configure TLS for the metrics backend and the gateway. See [Connector keys](#connector-keys) for the available settings. 7. Click Connect Observability. Platform reconciles the following resources: * `Deployment/gateway-fleet-observability` * `Service/gateway-fleet-observability` * Grafana datasource ConfigMaps for fleet, control plane cluster, and tenant cluster views when Grafana datasource reconciliation is enabled. ## Connector keys[​](#connector-keys "Direct link to Connector keys") | Key | Required | Description | | ------------------------------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `metricsBackendOtlpEndpoint` | Yes | OTLP endpoint that receives metrics from the Write Gateway. Provide the base URL only; the gateway appends `/v1/metrics` (for example, `https://prom-prometheus-server.vcluster-platform/api/v1/otlp` receives writes at `/api/v1/otlp/v1/metrics`). | | `metricsBackendPromQLEndpoint` | Yes | Prometheus HTTP API endpoint queried by the Query Proxy. | | `metricsBackendInsecureSkipVerify` | No | Set to `"true"` only for test backends with unverifiable TLS. | | `metricsBackendServerName` | No | TLS server name override for the Query Proxy's PromQL reads. It doesn't affect OTLP writes; the OTLP endpoint's certificate must still match the endpoint hostname. | | `observabilityGatewayCertSecretName` | No | Custom Secret with `tls.crt` and `tls.key` for gateway serving TLS. Include `ca.crt` when clients must trust a private CA. Platform creates a self-signed serving certificate when this key is omitted. | | `metricsBackendCertSecretName` | No | Secret with `ca.crt`, `tls.crt`, and `tls.key` for mTLS to the backend. The gateway trusts the backend with `ca.crt` and presents the certificate and key as its client identity. | | `gatewayResources` | No | YAML `resources` block for the Write Gateway container. | | `queryProxyResources` | No | YAML `resources` block for the Query Proxy container. | | `serviceType` | No | Gateway Service type. Valid values are `ClusterIP`, the default, and `LoadBalancer`. | | `serviceAnnotations` | No | YAML map of annotations applied to a `LoadBalancer` gateway Service. | | `grafanaUrl` | No | Grafana upstream URL used by the Platform `/grafana/` reverse proxy. | | `grafanaInsecureSkipVerify` | No | Set to `"true"` only to test a Grafana endpoint with an unverifiable certificate. | | `grafanaCertSecretName` | No | Grafana serving certificate Secret. Platform uses its `ca.crt` to trust the Grafana upstream. | | `fleetObservabilityDashboardUid` | No | UID of the fleet dashboard embedded by the Fleet Observability page. | Bring-your-own Grafana requirements When you point `grafanaUrl` at a Grafana you manage instead of the bundled Grafana, that Grafana must: * **Serve from the `/grafana` subpath.** Set `server.root_url` and `server.serve_from_sub_path` so Grafana works behind the Platform `/grafana/` reverse proxy. See [`serve_from_sub_path`](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#serve_from_sub_path). * **Allow embedding.** Set `security.allow_embedding = true` so the Fleet Observability page and each tenant cluster's Observability tab can embed the dashboards in an iframe. See [`allow_embedding`](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#allow_embedding). ## Customize the gateway serving certificate[​](#customize-the-gateway-serving-certificate "Direct link to Customize the gateway serving certificate") The gateway always serves OTLP and query traffic over TLS. When you don't specify a certificate Secret, Platform creates and manages a self-signed Secret named `gateway--tls`. This only works for connector IDs of 51 characters or fewer; see [Create the connector Secret](#create-the-connector-secret) for why longer IDs need a custom certificate Secret instead. To use your own certificate, create a Secret in the Platform namespace. Platform never validates or modifies a custom certificate, so it must already cover every hostname a client connects to. At minimum, include the gateway's in-cluster Service hostname, `gateway-..svc.cluster.local`, since Platform-generated Grafana datasources and any in-cluster collector always connect through it. If you expose the gateway with a `LoadBalancer` (see [Expose the gateway with a LoadBalancer](#expose-the-gateway-with-a-loadbalancer) below), also include its external hostname or IP as a SAN. gateway-tls.yaml ``` apiVersion: v1 kind: Secret metadata: name: fleet-gateway-tls namespace: vcluster-platform type: kubernetes.io/tls stringData: tls.crt: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- tls.key: | -----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY----- ca.crt: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` Reference it from the connector: ``` stringData: observabilityGatewayCertSecretName: fleet-gateway-tls ``` Edge collectors use HTTPS for OTLP HTTP or TLS for OTLP gRPC in both the managed and custom certificate cases. Distribute `ca.crt` to collectors that don't already trust the issuing CA. Grafana datasources generated by Platform trust the `ca.crt` from the gateway serving certificate Secret. A certificate signed by a public CA still needs to cover the hostnames above; once it does, `ca.crt` can be omitted, since clients already trust the public CA through their system trust stores. ## Expose the gateway with a LoadBalancer[​](#expose-the-gateway-with-a-loadbalancer "Direct link to Expose the gateway with a LoadBalancer") The gateway Service defaults to `ClusterIP`. To make the gateway reachable from collectors and query clients outside the Platform cluster, set `serviceType` to `LoadBalancer`. You can optionally apply provider-specific Service annotations: ``` stringData: serviceType: LoadBalancer serviceAnnotations: | service.beta.kubernetes.io/aws-load-balancer-scheme: internal ``` For a Platform-managed certificate, the controller waits for the LoadBalancer address and adds it to the certificate as a subject alternative name before deploying the gateway. A custom certificate needs that same SAN added by hand, since Platform never modifies it. See [Customize the gateway serving certificate](#customize-the-gateway-serving-certificate) for how clients trust the resulting certificate either way. ### Retrieve the load balancer address[​](#retrieve-the-load-balancer-address "Direct link to Retrieve the load balancer address") The cloud provider assigns the address asynchronously, so it stays empty until provisioning finishes. Read it from the gateway Service: ``` CONNECTOR_ID=fleet-observability kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID" \ -o jsonpath='{range .status.loadBalancer.ingress[*]}{.hostname}{.ip}{"\n"}{end}' ``` The provider reports either a hostname (for example, AWS) or an IP address (for example, GCP and Azure). Build the reachable endpoints from that address: | Endpoint | Port | Purpose | | ------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------- | | `https://:4318` | `4318` | OTLP HTTP writes. Use this for collectors. | | `https://:4317` | `4317` | OTLP gRPC writes. The gateway accepts gRPC, but the bundled collector sends over HTTP, so use `4318` for collectors. | | `https://:8081` | `8081` | Prometheus HTTP API through the Query Proxy. Use this for external query clients. | Use this address wherever a collector or query client connects from outside the Platform cluster. ## Use backend mTLS[​](#use-backend-mtls "Direct link to Use backend mTLS") To authenticate the gateway to your backend with mTLS, create a Secret: backend-client-tls.yaml ``` apiVersion: v1 kind: Secret metadata: name: metrics-backend-client-tls namespace: vcluster-platform type: kubernetes.io/tls stringData: ca.crt: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- tls.crt: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- tls.key: | -----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY----- ``` Reference it from the connector: ``` stringData: metricsBackendCertSecretName: metrics-backend-client-tls ``` ### Configure backend mTLS across namespaces or clusters[​](#configure-backend-mtls-across-namespaces-or-clusters "Direct link to Configure backend mTLS across namespaces or clusters") The bundled Prometheus template's default mTLS setup works because Prometheus and the gateway both mount the same Secret object. By default they share the namespace Platform itself is installed in (`vcluster-platform`, unless you renamed it), so `metricsBackendTLSSecret` (the Prometheus side) and `metricsBackendCertSecretName` (the connector side) both point at the same auto-generated `metrics-backend-mtls` Secret. That single-Secret shortcut only works when Prometheus and the gateway share a namespace and a cluster. What to do instead depends on where Prometheus runs. #### Different namespace, same cluster[​](#different-namespace-same-cluster "Direct link to Different namespace, same cluster") If you set `destinationNamespace` to a namespace other than the one Platform is installed in, Prometheus stays on the local cluster, so its bootstrap certificate still covers the right in-cluster hostname (`prom-prometheus-server..svc.cluster.local`). A single Secret object just can't be mounted in two namespaces at once, so copy it instead of sharing one object: * Leave `metricsBackendTLSSecret` empty so the Argo CD template bootstraps the shared certificate for you, named `metrics-backend-mtls` in Prometheus's `destinationNamespace`. See [Deploy the metrics backend with Argo CD](#deploy-the-metrics-backend-with-argo-cd). * Once Prometheus reports ready, read that generated Secret and create an identical copy in the Platform namespace, then reference it from the connector's `metricsBackendCertSecretName`. Setting `metricsBackendTLSSecret` to a name of your own disables this bootstrap entirely, so only use it if you're supplying your own certificate material for both Secrets instead of copying the generated one. #### A different cluster, or a backend other than the bundled Prometheus[​](#a-different-cluster-or-a-backend-other-than-the-bundled-prometheus "Direct link to A different cluster, or a backend other than the bundled Prometheus") The bundled workflow above deploys Prometheus to the local cluster, and its bootstrap certificate only covers in-cluster Service hostnames. Copying that Secret doesn't work once the gateway reaches the backend through an external hostname, because the certificate has no SAN for it. Create the gateway-side Secret as described in [Use backend mTLS](#use-backend-mtls) above, but with two differences: * The backend's server certificate must cover the hostname used in `metricsBackendOtlpEndpoint` and `metricsBackendPromQLEndpoint`, not an in-cluster Service name. * Configure the backend to trust the CA that issues the gateway's client certificate, the same way the bundled Prometheus config trusts it through `client_ca_file`. Without this, the backend rejects the gateway's mTLS handshake even if the gateway trusts the backend. ## Verify the gateway[​](#verify-the-gateway "Direct link to Verify the gateway") Check that the gateway pod is running: ``` CONNECTOR_ID=fleet-observability kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID" ``` Check the service ports: ``` CONNECTOR_ID=fleet-observability kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID" ``` Expected service ports: | Port | Name | Purpose | | ------ | ----------- | -------------------------------------------- | | `4317` | `otlp` | OTLP gRPC writes. | | `4318` | `otlp-http` | OTLP HTTP writes. | | `8081` | `query` | Prometheus HTTP API through the Query Proxy. | Next, create metrics access keys in [Configure metrics access](https://vcluster.com/docs/platform/maintenance/observability/configure-metrics-access.md). --- # Fleet Observability labels reference The Write Gateway owns the full `vcluster_platform_*` label namespace. These labels are stamped from authenticated Platform context and are safe to use for dashboards, recording rules, and backend queries. warning Don't set labels with the `vcluster_platform_*` prefix in collectors or applications. The Write Gateway strips caller-supplied platform labels before adding authenticated values. ## Scope-controlling labels[​](#scope-controlling-labels "Direct link to Scope-controlling labels") The Query Proxy uses these labels for read authorization. | Label | Meaning | Header equivalent | | ---------------------------- | ------------------------------- | ------------------------------ | | `vcluster_platform_cluster` | Control plane cluster identity. | `X-Vcluster-Platform-Cluster` | | `vcluster_platform_project` | Tenant cluster project. | `X-Vcluster-Platform-Project` | | `vcluster_platform_instance` | VirtualClusterInstance name. | `X-Vcluster-Platform-Instance` | Use either cluster scope or tenant scope, not both: * Cluster scope: `vcluster_platform_cluster` * Tenant scope: `vcluster_platform_project` and `vcluster_platform_instance` ## Descriptive labels[​](#descriptive-labels "Direct link to Descriptive labels") These labels are stamped at write time for filtering and grouping, but they don't authorize reads. | Label | Meaning | | ----------------------------------- | -------------------------------------------------------------------------------------------- | | `vcluster_platform_hosted_on` | Registered control plane cluster that hosts the tenant cluster. | | `vcluster_platform_placement` | Placement category, such as `hosted`, `standalone`, or `external`. | | `vcluster_platform_shape` | Node arrangement, such as `control_plane`, `shared_nodes`, or `private_nodes`. | | `vcluster_platform_node_management` | Private-node management mode, such as `manual` or `auto`. Empty for non-private-node shapes. | ## Matcher rules[​](#matcher-rules "Direct link to Matcher rules") The Query Proxy restricts matchers on scope-controlling labels: | Matcher | Scope labels | Example | | -------------------------------------- | ------------ | ------------------------------------------- | | Equality | Allowed | `vcluster_platform_instance="api"` | | Literal alternation regular expression | Allowed | `vcluster_platform_instance=~"api\|worker"` | | Negative equality | Rejected | `vcluster_platform_instance!="api"` | | Negative regular expression | Rejected | `vcluster_platform_instance!~"api"` | | Wildcard regular expression | Rejected | `vcluster_platform_instance=~".*"` | Descriptive labels can use normal PromQL matcher forms because they don't determine authorization scope. ## Label examples[​](#label-examples "Direct link to Label examples") Tenant cluster metric: ``` vcluster_platform_project="team-a" vcluster_platform_instance="api" vcluster_platform_hosted_on="local-cluster" vcluster_platform_placement="hosted" vcluster_platform_shape="shared_nodes" ``` Control plane cluster metric: ``` vcluster_platform_cluster="local-cluster" ``` Private-nodes tenant cluster metric: ``` vcluster_platform_project="gpu-team" vcluster_platform_instance="inference" vcluster_platform_hosted_on="local-cluster" vcluster_platform_placement="hosted" vcluster_platform_shape="private_nodes" vcluster_platform_node_management="auto" ``` ## Migration from older labels[​](#migration-from-older-labels "Direct link to Migration from older labels") [Aggregating metrics](https://vcluster.com/docs/platform/maintenance/monitoring/aggregating-metrics.md) and [Fleet monitoring with OpenTelemetry](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-monitoring-otel.md) predate the Fleet Observability gateway and use their own self-reported label schemes. These aren't migrations in the sense of a renamed field. The authenticated `vcluster_platform_*` labels come from a different collection and authorization model, so some older labels have no direct equivalent. Prefer `vcluster_platform_*` for anything going through the Fleet Observability gateway. | Older label | Source guide | Meaning | `vcluster_platform_*` equivalent | | ---------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cluster` | Fleet monitoring with OpenTelemetry | Control plane cluster hosting the tenant cluster (this guide's pipeline always pairs it with tenant identity) | `vcluster_platform_hosted_on`. `vcluster_platform_cluster` is reserved for cluster-scoped metrics with no tenant identity, which this guide's pipeline doesn't produce. | | `vcluster_name` | Fleet monitoring with OpenTelemetry | Tenant cluster name | `vcluster_platform_instance` | | `vcluster_project` | Fleet monitoring with OpenTelemetry | Project | `vcluster_platform_project` | | `vcluster_project_namespace` | Fleet monitoring with OpenTelemetry | Namespace backing the tenant cluster | No direct equivalent — the gateway doesn't expose the backing namespace as a scope label | | `vcluster_user` | Fleet monitoring with OpenTelemetry | Tenant cluster owner | No direct equivalent — not part of the authenticated scope model | | `vcluster_virtual_namespace` | Fleet monitoring with OpenTelemetry | Namespace as seen inside the tenant cluster | No direct equivalent | | `vcluster_virtual_pod` | Fleet monitoring with OpenTelemetry | Pod as seen inside the tenant cluster | No direct equivalent | | `loft_project_name` | Aggregating Metrics | Project | `vcluster_platform_project` | | `loft_virtualcluster_name` | Aggregating Metrics | Tenant cluster name | `vcluster_platform_instance` | | `loft_space_name` | Aggregating Metrics | Space name (Space-type tenants only) | No direct equivalent | For the status and operational ownership of each architecture, see [Fleet metrics architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md). --- # Observe GPU nodes with NVSentinel [NVSentinel](https://github.com/NVIDIA/NVSentinel) is NVIDIA's open source, Kubernetes-native service for GPU fault handling. It detects hardware and software faults on GPU nodes through DCGM, system logs, and cloud provider maintenance events. It can respond by cordoning nodes, draining workloads, and triggering node reboots or terminations. When the [Argo CD integration](https://vcluster.com/docs/platform/integrations/argocd.md) is enabled, vCluster Platform ships an `ArgoCDApplicationTemplate` for NVSentinel alongside the other [GPU observability templates](https://vcluster.com/docs/platform/maintenance/observability/gpu-observability-templates.md). You reference the template from an `ArgoCDApplication` to deploy NVSentinel into a cluster that runs GPU nodes, typically a tenant cluster with private nodes. NVSentinel metrics then flow through the Fleet Observability pipeline like any other workload metrics and appear on the bundled GPU dashboards. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * The [Argo CD integration](https://vcluster.com/docs/platform/integrations/argocd.md) is enabled. * An Argo CD connector is registered on the target cluster. See [Connect to Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md). * The [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html) is deployed in the same cluster, per the [deployment order](https://vcluster.com/docs/platform/maintenance/observability/gpu-observability-templates.md#deployment-order). * cert-manager is deployed, if any component you enable needs internal TLS. See [When you need cert-manager](#when-you-need-cert-manager). * A default StorageClass exists, or one is named in the template parameters, if any component you enable needs the MongoDB store. * Prometheus Operator CRDs exist, if you enable the PodMonitor. ## What the template deploys[​](#what-the-template-deploys "Direct link to What the template deploys") The template deploys the upstream `nvsentinel` Helm chart from `oci://ghcr.io/nvidia/nvsentinel`. In an `ArgoCDApplication`, reference it through `spec.templateRef.name: nvsentinel`. The Platform UI lists it as **NVSentinel**. NVSentinel is composed of independent components that you enable per deployment: | Component | Default | What it does | | ---------------------- | ------- | --------------------------------------------------------------------------------- | | GPU Health Monitor | `true` | Watches GPU hardware through DCGM for thermal issues, ECC errors, and XID events. | | Syslog Health Monitor | `true` | Scans system logs on each node for fault patterns. | | NIC Health Monitor | `false` | Watches network interface health. | | CSP Health Monitor | `false` | Pulls maintenance events from cloud provider APIs. | | Fault Quarantine | `false` | Cordons and taints nodes with detected faults. | | Node Drainer | `false` | Evicts workloads from quarantined nodes. | | Fault Remediation | `false` | Creates maintenance resources to recover failing nodes after drain completes. | | Janitor | `false` | Executes the node reboots and terminations that Fault Remediation requests. | | Health Events Analyzer | `false` | Analyzes patterns across health events and produces recommended actions. | | MongoDB Store | `false` | Persistent event storage; the coordination layer for the remediation components. | With the defaults, NVSentinel only observes health events and metrics. Nothing acts on nodes. The remediation pipeline (Fault Quarantine, Node Drainer, Fault Remediation, Janitor) is opt-in. ### Component dependencies the template resolves for you[​](#component-dependencies-the-template-resolves-for-you "Direct link to Component dependencies the template resolves for you") Some components can't start without others. The template enables the dependencies automatically, so the deployed application can differ from the toggles you set: * **MongoDB store** turns on whenever Fault Quarantine, Node Drainer, Fault Remediation, or Health Events Analyzer is enabled, even if you leave **Enable MongoDB Store** off. Those components coordinate through MongoDB change streams and crash without it. * **Janitor** turns on whenever Fault Remediation is enabled, because Fault Remediation can't reboot or terminate nodes without it. * The Janitor's default provider is `kind`, which only **simulates** reboots and terminations. Set the **Janitor Provider** parameter to your cloud provider (or `generic` for bare metal) before relying on real remediation. See [Janitor provider parameters](#janitor-provider-parameters). note Node Drainer is the exception. It needs Fault Quarantine to have anything to drain, but the template doesn't turn on Fault Quarantine for you the way it does for the dependencies above. Set both `enableFaultQuarantine` and `enableNodeDrainer` to `"true"` yourself, as the [deploy example](#deploy-nvsentinel) does. Enabling Node Drainer alone deploys a drainer with no quarantined nodes to evict from. ### When you need cert-manager[​](#when-you-need-cert-manager "Direct link to When you need cert-manager") With the observe-only defaults, NVSentinel runs without cert-manager. It becomes required when you enable any of the following: * **The internal MongoDB store.** TLS/mTLS is enabled by default, and cert-manager creates the server, client, and CA certificates. Fault Quarantine, Node Drainer, and Health Events Analyzer turn the MongoDB store on automatically, so they indirectly require cert-manager too. * **The Janitor or Fault Remediation.** The Janitor uses TLS certificates for its validating webhook and its provider gRPC endpoint. * **Chart features that aren't exposed as template parameters**, such as preflight checks (the mutating admission webhook needs TLS, and cert-manager is the default provider) and internal PostgreSQL with the chart's TLS configuration. ## Deploy NVSentinel[​](#deploy-nvsentinel "Direct link to Deploy NVSentinel") Deploy NVSentinel into the cluster whose GPU nodes it should watch. For a tenant cluster with private GPU nodes, target the tenant cluster itself. * Platform UI * YAML Platform creates a namespace named `p-` for each project, so the default project's namespace is `p-default`. Create the `ArgoCDApplication` object in your project namespace: nvsentinel.yaml ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplication metadata: name: nvsentinel namespace: p-my-project # replace with your project namespace spec: displayName: "NVSentinel" destination: virtualCluster: name: my-vcluster # replace with your tenant cluster name target: vCluster # deploy into the tenant cluster templateRef: name: nvsentinel parameters: destinationNamespace: nvsentinel project: default dryRun: "true" # keep dry run on until you have verified detection enableFaultQuarantine: "true" enableNodeDrainer: "true" ``` 1. Select your project from the projects dropdown at the top of the left navigation bar. 2. Navigate to Tenant Clusters and open the tenant cluster with the GPU nodes to monitor. 3. Select the ArgoCD Apps tab and click Add Argo CD Application. 4. In the Basics section, enter a Display Name such as `NVSentinel`. 5. Select Deploy to tenant cluster to deploy into the tenant cluster itself. 6. From the Argo CD Application Template dropdown, select the NVSentinel template. 7. Fill the Template Parameters. Keep Dry Run enabled for the first rollout, and enable the components you need. The DCGM defaults match a standard GPU Operator install. 8. Click Create Argo CD Application. ## Dry run[​](#dry-run "Direct link to Dry run") The template defaults to `dryRun: "true"`. As with any dry run, NVSentinel runs the full detection and decision pipeline but skips changes to node state or workloads. **Dry run still writes real node annotations and labels, and still increments the metrics that feed the bundled GPU dashboard.** When a fault is detected in dry-run mode, NVSentinel: * Logs the intended action, for example `Cordoning node`. * Writes its own annotations and the `dgxc.nvidia.com/nvsentinel-state` label on the node to track the quarantine decision. * Increments quarantine metrics such as `fault_quarantine_current_quarantined_nodes` and `fault_quarantine_cordons_applied_total`, which feed the **Quarantined Nodes** panel on the bundled GPU dashboard. During a dry run, NVSentinel doesn't: * Cordon the node. `kubectl get node -o jsonpath='{.spec.unschedulable}'` stays empty and the node keeps accepting workloads. * Evict workloads. * Reboot or terminate nodes. Because the metrics increment but nodes are not cordoned or evicted during a dry run, both the node object and the dashboard can look like a genuine quarantine happened. Use dry run to validate detection end to end. Inject or wait for a health event, confirm the logs, node annotations, and metrics, and then set `dryRun: "false"` to let NVSentinel cordon, drain, and remediate for real. Before turning dry run off, make sure the Janitor provider is set to a real provider, since the default `kind` provider only simulates reboots and terminations. ## Uninstall and cleanup[​](#uninstall-and-cleanup "Direct link to Uninstall and cleanup") Deleting the NVSentinel `ArgoCDApplication` prunes the workloads, but two artifacts are retained by design and stay in the cluster until you remove them: * **MongoDB PVCs.** The three `datadir-mongodb-*` PVCs (10Gi each by default, 30Gi total) survive uninstall so health-event history isn't destroyed by accident. * **Namespace.** The destination namespace (`nvsentinel`) stays `Active` with zero pods. Deleting it also removes the namespaced cert-manager resources (Certificates and Issuers) that NVSentinel created. Clean up retained NVSentinel artifacts ``` # MongoDB data (destroys health-event history) kubectl delete pvc -n nvsentinel datadir-mongodb-0 datadir-mongodb-1 datadir-mongodb-2 kubectl delete namespace nvsentinel ``` Neither carries blocking finalizers, so deletion completes cleanly. warning Deleting the MongoDB PVCs permanently deletes NVSentinel's health-event history. To remove the rest of the GPU stack, including the retained cert-manager CRDs, see [Tear down the GPU stack](https://vcluster.com/docs/platform/maintenance/observability/gpu-observability-templates.md#tear-down-the-gpu-stack). ## Parameters[​](#parameters "Direct link to Parameters") The template groups parameters by component. All boolean parameters are passed as strings (`"true"` / `"false"`). ### Core parameters[​](#core-parameters "Direct link to Core parameters") | Parameter | Default | Description | | ---------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------- | | `destinationNamespace` | `nvsentinel` | Namespace NVSentinel deploys into. | | `project` | `default` | Argo CD project that owns the application. | | `version` | `v1.13.0` | NVSentinel Helm chart version. | | `repoURL` | `oci://ghcr.io/nvidia/nvsentinel` | Chart repository. Change to use a mirror or private registry. | | `releaseName` | `nvsentinel` | Helm release name. | | `chart` | `nvsentinel` | Chart name pulled from the repository. | | `imageTag` | `v1.13.0` | NVSentinel image tag. | | `dryRun` | `true` | Log and record intended actions without cordoning, draining, or remediating. See [Dry run](#dry-run). | ### DCGM parameters[​](#dcgm-parameters "Direct link to DCGM parameters") | Parameter | Default | Description | | -------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------- | | `dcgmEndpoint` | `nvidia-dcgm.gpu-operator.svc` | DCGM service the GPU Health Monitor reads from. The default matches a standard GPU Operator install. | | `dcgmPort` | `5555` | DCGM service port. | ### Component toggles[​](#component-toggles "Direct link to Component toggles") | Parameter | Default | Description | | ---------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enableGpuHealthMonitor` | `true` | GPU hardware health through DCGM. | | `enableSyslogHealthMonitor` | `true` | System log fault patterns. | | `enableNicHealthMonitor` | `false` | Network interface health. | | `enableCspHealthMonitor` | `false` | Cloud provider maintenance events. | | `enableFaultQuarantine` | `false` | Cordon and taint faulty nodes. Turns on the MongoDB store automatically. | | `enableNodeDrainer` | `false` | Evict workloads from quarantined nodes. Turns on the MongoDB store automatically, but not Fault Quarantine. [Enable that yourself too](#component-dependencies-the-template-resolves-for-you). | | `enableFaultRemediation` | `false` | Create maintenance resources to recover failing nodes. Turns on the MongoDB store and the Janitor automatically. | | `enableJanitor` | `false` | Execute reboots and terminations. Turns on automatically with Fault Remediation. | | `enableHealthEventsAnalyzer` | `false` | Analyze patterns across health events. Turns on the MongoDB store automatically. | ### MongoDB store parameters[​](#mongodb-store-parameters "Direct link to MongoDB store parameters") | Parameter | Default | Description | | --------------------- | --------- | ----------------------------------------------------------------------------------------------------- | | `enableMongodbStore` | `false` | Enable the MongoDB datastore explicitly. Turns on automatically with the components listed above. | | `mongodbStorageClass` | *(empty)* | StorageClass for the MongoDB PVCs. Empty uses the cluster default. | | `mongodbStorageSize` | `10Gi` | Volume size per MongoDB replica. The chart runs three replicas, so the default provisions 30Gi total. | ### Node labeler parameters[​](#node-labeler-parameters "Direct link to Node labeler parameters") | Parameter | Default | Description | | ----------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `assumeDriverInstalled` | `false` | Enable when NVIDIA drivers are preinstalled on the host rather than deployed by GPU Operator driver pods (`driver.enabled=false`). The labeler then marks all GPU nodes as having the driver installed instead of watching for driver pods. Without it there are no driver pods to detect, so the syslog and metadata DaemonSets never start on those nodes. | ### Metrics and operations parameters[​](#metrics-and-operations-parameters "Direct link to Metrics and operations parameters") | Parameter | Default | Description | | --------------------- | --------- | ------------------------------------------------------------------------------------------------------ | | `enablePodMonitor` | `false` | Create a PodMonitor for NVSentinel metrics. Requires Prometheus Operator CRDs. | | `podMonitorInterval` | `30s` | PodMonitor scrape interval. | | `enableNetworkPolicy` | `true` | Restrict ingress to metrics ports. Disable if it blocks a same-namespace webhook such as cert-manager. | | `enableTracing` | `false` | Enable OpenTelemetry tracing. | | `tracingEndpoint` | *(empty)* | OTLP gRPC endpoint (`host:port`) for trace ingestion. Required when tracing is enabled. | | `enableAuditLogging` | `false` | Enable audit logging. | ### Janitor provider parameters[​](#janitor-provider-parameters "Direct link to Janitor provider parameters") The Janitor talks to your infrastructure provider to reboot and terminate nodes. Pick the provider with `janitorProvider` and fill the matching provider section. The `kind` and `kwok` providers only simulate actions and are meant for test clusters. | Parameter | Default | Description | | -------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `janitorProvider` | `kind` | One of `kind` or `kwok` (both simulated), `aws`, `gcp`, `azure`, `oci`, `nebius`, or `generic` (bare metal or on-prem). | | `janitorServiceAccountAnnotationKey` | *(empty)* | Annotation key on the Janitor ServiceAccount for cloud identity, for example `eks.amazonaws.com/role-arn` or `iam.gke.io/gcp-service-account`. Set together with the value. | | `janitorServiceAccountAnnotationValue` | *(empty)* | Value for the annotation key above, for example the IAM role ARN or GCP service account email. | Select the tab for your Janitor provider: * AWS * GCP * Azure * OCI * Nebius * Generic | Parameter | Default | Description | | ----------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `janitorAwsRegion` | *(empty)* | AWS region of the EKS cluster (for example `us-east-1`). The Janitor uses this one region for all EC2 calls, so every node must live in it. Multiple availability zones within the region are fine. | | `janitorAwsAccountId` | *(empty)* | AWS account ID (12 digits), required for IRSA. | | `janitorAwsIamRoleName` | *(empty)* | IAM role name for the Janitor ServiceAccount. The role needs `ec2:RebootInstances` and `ec2:TerminateInstances`. | | Parameter | Default | Description | | -------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `janitorGcpProject` | *(empty)* | GCP project ID of the GKE cluster. Optional; the Janitor auto-detects each node's project from its `providerID`. | | `janitorGcpZone` | *(empty)* | GCP zone where cluster nodes run (for example `us-central1-a`). Optional; auto-detected per node from its `providerID`. | | `janitorGcpServiceAccount` | *(empty)* | GCP service account name for Workload Identity, without the `@project.iam.gserviceaccount.com` suffix. Needs `compute.instances.reset`. | | Parameter | Default | Description | | ---------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------- | | `janitorAzureSubscriptionId` | *(empty)* | Azure subscription ID (UUID). | | `janitorAzureResourceGroup` | *(empty)* | Resource group of the AKS cluster. Derived per node from its `providerID`, so set it only if detection fails. | | `janitorAzureLocation` | *(empty)* | Azure region of the AKS cluster (for example `eastus`). Derived per node from its `providerID`, so not normally required. | | `janitorAzureClientId` | *(empty)* | Managed identity client ID (UUID) for Workload Identity. The identity needs the Virtual Machine Contributor role. | | Parameter | Default | Description | | --------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------- | | `janitorOciRegion` | *(empty)* | OCI region of the OKE cluster (for example `us-ashburn-1`). One Janitor works in a single region, so all nodes must be in it. | | `janitorOciCompartment` | *(empty)* | Compartment OCID containing cluster resources. | | `janitorOciCredentialsFile` | *(empty)* | Path to an OCI credentials file. Leave empty to use workload identity. | | `janitorOciProfile` | `DEFAULT` | Profile name in the credentials file. | | `janitorOciPrincipalId` | *(empty)* | Principal OCID for Workload Identity (the dynamic group or service principal OCID). | | Parameter | Default | Description | | -------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `janitorNebiusServiceAccountKeySecret` | *(empty)* | Name of an existing Kubernetes Secret holding the Nebius service account key as `sa-credentials.json`. The service account needs `compute.instances.stop` and `compute.instances.start`. Zone and region are auto-detected per node. | The `generic` provider reboots nodes with a privileged Job scheduled onto the node. It can't terminate instances. | Parameter | Default | Description | | ----------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `janitorGenericRebootImage` | *(empty)* | Container image for the reboot Job. Empty uses the chart default (busybox). | | `janitorGenericUseSysrqReboot` | `false` | Use Linux Magic SysRq for an immediate kernel reboot instead of a `chroot /host reboot`. Enable only when nodes allow SysRq through `/proc/sysrq-trigger`. | | `janitorGenericRebootJobNamespace` | *(empty)* | Namespace for reboot Jobs. Empty uses the release namespace. | | `janitorGenericRebootJobTTLSeconds` | *(empty)* | TTL in seconds for completed reboot Jobs. Empty uses the chart default (3600). | | `janitorGenericImagePullSecrets` | *(empty)* | Comma-separated image pull Secret names for the reboot Job. | ## Related pages[​](#related-pages "Direct link to Related pages") * [GPU observability templates](https://vcluster.com/docs/platform/maintenance/observability/gpu-observability-templates.md) for the full GPU template stack, deployment order, DCGM metric families, and per-pod GPU attribution. * [Install the observability gateway](https://vcluster.com/docs/platform/maintenance/observability/install-observability-gateway.md) for the metrics backend and gateway setup. * [Configure edge collectors](https://vcluster.com/docs/platform/maintenance/observability/configure-edge-collectors.md) for deploying collectors with Argo CD or by hand. * [Query fleet metrics](https://vcluster.com/docs/platform/maintenance/observability/query-fleet-metrics.md) for scoped PromQL access to NVSentinel and DCGM metrics. * [Troubleshooting](https://vcluster.com/docs/platform/maintenance/observability/troubleshooting.md) for gateway and collector failures. --- # Fleet Observability | | | Enterprise | | | | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Fleet Observability gives platform operators a central telemetry path for tenant clusters and control plane clusters. Telemetry flows over OTLP through a vCluster Platform observability gateway before it reaches your backend. The gateway authenticates writes, stamps platform-owned labels, and exposes a scoped Prometheus query API for Grafana and other read clients. Use this guide when you need fleet-wide visibility across projects, tenant clusters, shared nodes, private nodes, or connected clusters. ## Get started[​](#get-started "Direct link to Get started") Follow the [Fleet Observability quickstart](https://vcluster.com/docs/platform/maintenance/observability/quickstart.md) to deploy the default pipeline end to end. It covers the metrics backend, gateway, access keys, and edge collectors, using the bundled Argo CD templates. For other backends, manual collector configuration, or control plane cluster targets, see the individual setup pages in the page guide below. Scope Fleet Observability is currently intended for platform administrators. It provides the gateway, query proxy, metrics access-key roles, label contract, and both an out-of-the-box and a manual configuration path for collectors and backends. The bundled Grafana template provides pre-provisioned dashboards today. Platform automatically manages fleet, tenant cluster, and control plane cluster datasource ConfigMaps. Per-tenant connector selection, access-key generation, and automatic deployment and reconciliation of the observability applications are planned as follow-up work. The current release focuses on metrics. Edge collectors and the gateway communicate over OTLP. Not available on embedded Kubernetes Fleet Observability runs the metrics gateway and Grafana datasources on the platform cluster, so it requires a platform cluster that can run those applications. It is currently not available when the platform runs on embedded Kubernetes, which backs the external-database and multi-region platform features. In those deployments Fleet Observability is turned off automatically. ## Architecture[​](#architecture "Direct link to Architecture") Fleet Observability write and query paths through the vCluster Platform observability gateway The architecture has three main parts: * **Edge collectors** are OpenTelemetry Collectors that collect metrics from control plane clusters, tenant clusters, and optional exporters. They send OTLP to the Write Gateway. * **Write Gateway** is the vCluster-built ingestion and enforcement layer. It receives OTLP, authenticates each write with a `metrics-writer` access key, authorizes the requested scope with the Platform API, strips caller-supplied platform labels, and stamps the authenticated `vcluster_platform_*` labels. * **Query Proxy** exposes a Prometheus-compatible read API. It authenticates each read, authorizes the requested scope with a `metrics-reader` key or user session, rewrites PromQL so reads stay scope-bound, and forwards the query to your metrics backend. ## What platform manages[​](#what-platform-manages "Direct link to What platform manages") An observability connector Secret tells Platform how to reach your metrics backend. When the connector is present, Platform deploys and reconciles: * The observability gateway: the Write Gateway and Query Proxy that front your backend. * The Grafana datasources that power the fleet, tenant cluster, and control plane cluster dashboards and metrics explorer. You must provide a backend that can import OTLP metrics and serve the Prometheus HTTP API. You must also configure the edge collectors that send metrics to the gateway. If you enable the [Argo CD integration](https://vcluster.com/docs/platform/integrations/argocd.md), Platform ships default `ArgoCDApplicationTemplate` resources for the metrics backend, Grafana, and edge collectors. You can leverage these templates to create the `ArgoCDApplication` resources that deploy them, instead of wiring them up by hand. Each setup page below covers its own Argo CD template alongside the manual path. ## Security model[​](#security-model "Direct link to Security model") Fleet Observability doesn't rely on backend tenant isolation. The gateway and query proxy are the enforcement layer. * Metrics writers must authenticate with an access key that has the `metrics-writer` role and a matching cluster or tenant cluster scope. * Metrics readers must authenticate with a `metrics-reader` key, a user access key, or a Grafana session passed through to Platform. * Platform authorizes both read and write requests with SelfSubjectAccessReview. * The Write Gateway owns the full `vcluster_platform_*` label namespace. Caller-supplied labels with that prefix, or dotted variants that would sanitize to that prefix, are removed before authenticated labels are added. * Positive authorization results are cached for a short TTL. On Platform outage, cached decisions expire and the system fails closed. ## Pluggable observability[​](#pluggable-observability "Direct link to Pluggable observability") The bundled components are the default. Replace any of them as your environment requires: * **Metrics backend.** Swap the bundled Prometheus for any backend that ingests OTLP and serves the Prometheus HTTP API. Integration is limited to an OTLP-enabled endpoint with mTLS for authentication, so backends that require other auth methods aren't supported yet. * **Edge collectors.** Run your own OpenTelemetry Collectors instead of the bundled template. See [Configure edge collectors](https://vcluster.com/docs/platform/maintenance/observability/configure-edge-collectors.md). * **Grafana.** The bundled Grafana is tightly coupled to the provided dashboards and datasources, and to Platform SSO with access-key cookie forwarding for scoped queries. Replacing it is possible but involved. ## Page guide[​](#page-guide "Direct link to Page guide") | Task | Page | | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | Deploy the default pipeline end to end with Argo CD | [Fleet Observability quickstart](https://vcluster.com/docs/platform/maintenance/observability/quickstart.md) | | Deploy the metrics backend, then create the connector and gateway | [Install the observability gateway](https://vcluster.com/docs/platform/maintenance/observability/install-observability-gateway.md) | | Create scoped metrics access keys | [Configure metrics access](https://vcluster.com/docs/platform/maintenance/observability/configure-metrics-access.md) | | Send metrics from collectors to the gateway | [Configure edge collectors](https://vcluster.com/docs/platform/maintenance/observability/configure-edge-collectors.md) | | Deploy Grafana and query metrics through it or PromQL | [Query fleet metrics](https://vcluster.com/docs/platform/maintenance/observability/query-fleet-metrics.md) | | Deploy the GPU observability stack | [GPU observability templates](https://vcluster.com/docs/platform/maintenance/observability/gpu-observability-templates.md) | | Detect and remediate GPU node faults | [Observe GPU nodes with NVSentinel](https://vcluster.com/docs/platform/maintenance/observability/nvsentinel-gpu-observability.md) | | Understand the platform label contract | [Labels reference](https://vcluster.com/docs/platform/maintenance/observability/labels-reference.md) | | Diagnose common failures | [Troubleshooting](https://vcluster.com/docs/platform/maintenance/observability/troubleshooting.md) | ## Related monitoring guides[​](#related-monitoring-guides "Direct link to Related monitoring guides") Fleet Observability is the recommended fleet metrics feature. The [fleet metrics architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md) page explains older operator-maintained examples for environments that can't use the gateway. For centralized workload log collection across your fleet, see [Central HostPath Mapper](https://vcluster.com/docs/platform/maintenance/logging/central-hostpath-mapper.md). To understand the log types vCluster produces, see [vCluster Logging](https://vcluster.com/docs/vcluster/manage/logging.md). --- # Query fleet metrics Query fleet metrics through the bundled Grafana, the fastest path once it's deployed, or directly against the Query Proxy's Prometheus-compatible HTTP API for automation and your own query clients. In both cases the Query Proxy authorizes each request with Platform, injects the requested scope into PromQL, and forwards the rewritten query to your metrics backend. ## Deploy the bundled Grafana with Argo CD[​](#deploy-the-bundled-grafana-with-argo-cd "Direct link to Deploy the bundled Grafana with Argo CD") Grafana is optional. Deploy it when you want the pre-provisioned fleet dashboards and Platform-authenticated access. If you use the [Argo CD integration](https://vcluster.com/docs/platform/integrations/argocd.md), reference the `fleet-observability-grafana` `ArgoCDApplicationTemplate`. * Platform UI * YAML Platform creates a namespace named `p-` for each project, so the default project's namespace is `p-default`. Create the `ArgoCDApplication` object in your project namespace: fleet-grafana.yaml ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplication metadata: name: fleet-grafana namespace: p-my-project # replace with your project namespace spec: displayName: "Fleet Grafana" destination: cluster: name: loft-cluster # the reserved name for Platform's local cluster; don't change this templateRef: name: fleet-observability-grafana parameters: destinationNamespace: vcluster-platform platformHost: platform.example.com project: default ``` 1. Go to Infrastructure > Control Plane Clusters, then click the local cluster (named `loft-cluster` by default) to open its detail page. 2. Select the ArgoCD Apps tab and click Add Argo CD Application. 3. In the Basics section, enter a Display Name such as `Fleet Grafana`. 4. From the Argo CD Application Template dropdown, select the Fleet Observability: Grafana template. 5. Fill the Template Parameters. Set platformHost to your Platform host. 6. Click Create Argo CD Application. | Parameter | Default | Description | | ---------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `destinationNamespace` | `vcluster-platform` | Namespace for Grafana. | | `project` | `default` | Argo CD project that owns the application. | | `platformHost` | Platform host | Platform host used to authenticate Grafana users. | | `grafanaAdminGroup` | *(empty)* | Group mapped to the Grafana admin role. | | `grafanaCertSecret` | *(empty)* | TLS Secret for Grafana serving. When empty, Platform self-signs a Secret named `grafana-server-tls`. | | `resources` | *(empty)* | Resource requests and limits for the Grafana container, as a multiline YAML block. See the example below. When empty, the chart default applies, which sets no requests or limits. | `resources` is a multiline parameter that takes the contents of a Kubernetes `resources` field. fleet-grafana.yaml ``` spec: parameters: resources: | requests: cpu: 100m memory: 256Mi limits: memory: 256Mi ``` Configure the fleet observability connector so Platform can proxy and embed the bundled Grafana dashboards. These values match the template defaults: ``` stringData: grafanaUrl: https://grafana.vcluster-platform.svc.cluster.local grafanaCertSecretName: grafana-server-tls fleetObservabilityDashboardUid: vcluster-fleet-observability ``` If you change `destinationNamespace`, update `grafanaUrl`. If you supply `grafanaCertSecret`, use that Secret name instead of `grafana-server-tls`. ## Query with the bundled Grafana[​](#query-with-the-bundled-grafana "Direct link to Query with the bundled Grafana") Once deployed, Platform runs a bundled Grafana pre-equipped with fleet dashboards and scoped datasources. It is the fastest way to explore fleet metrics without writing PromQL by hand. ### Sign in[​](#sign-in "Direct link to Sign in") The bundled Grafana is served at `https:///grafana/` and reverse-proxied by Platform. Sign in with Platform SSO; the local Grafana login form is disabled. Your Grafana role follows your Platform group membership: members of the template's `grafanaAdminGroup` get the Grafana Admin role, everyone else gets Viewer. The Grafana role governs Grafana features, not which metrics you can read. ### Dashboards[​](#dashboards "Direct link to Dashboards") Three dashboards ship pre-provisioned: | Dashboard | View | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | **Fleet Observability** | Fleet-wide summary: tenant cluster counts, GPU inventory and utilization, hardware-error hotspots, and per-tenant scrape health. | | **Cluster Metrics** | A single tenant cluster or control plane cluster. Scope comes from the selected datasource, so there are no dashboard filters. | | **GPU Overview** | GPU fleet view: DCGM utilization, thermals, hardware-error signals, and NVSentinel fault detection. | Each dashboard has a **Datasource** dropdown. Pick the datasource for the scope you want to view. The Platform UI also surfaces these dashboards directly: the **Fleet Observability** page shows the fleet and GPU views, and each tenant cluster's **Observability** tab shows its scoped Cluster Metrics dashboard. ![Fleet Observability dashboard showing tenant cluster counts, ready nodes, GPU inventory, per-project tenant cluster breakdowns, resource utilization, and top clusters by memory and CPU across the fleet](/docs/img/fleet-observability/fleet-observability-overview-dashboard.png) The Fleet Observability dashboard: a fleet-wide summary across projects and tenant clusters. ![GPU Overview dashboard showing GPU count, GPUs in use, utilization, temperature, total power, ECC error counts, and per-GPU utilization, filterable by GPU model and node](/docs/img/fleet-observability/fleet-observability-gpu-detail-dashboard.png) The GPU Overview dashboard: GPU utilization, thermals, and hardware-error signals, filterable by GPU model and node. ![Cluster Metrics dashboard for a single tenant cluster showing ready nodes, GPUs, resource utilization, pod count, and attached GPUs by type](/docs/img/fleet-observability/tenant-cluster-observability-dashboard.png) The Cluster Metrics dashboard for a single tenant cluster, opened from its Observability tab. ### Datasources and scope[​](#datasources-and-scope "Direct link to Datasources and scope") Platform auto-provisions Prometheus datasources that point at the Query Proxy. You don't create them by hand: * A **fleet-wide** datasource with no scope headers, for fleet and platform-admin views. Its UID is derived from the observability connector Secret's `metadata.uid`. * A **per-tenant-cluster** datasource for each tenant cluster, scoped to its project and instance. Its UID is derived from the tenant cluster's `status.serviceUID`. * A **per-control-plane-cluster** datasource for each control plane cluster. Tenant cluster datasources inject scope through request headers: ``` jsonData: httpHeaderName1: X-Vcluster-Platform-Project httpHeaderName2: X-Vcluster-Platform-Instance keepCookies: - loft_access_key secureJsonData: httpHeaderValue1: my-project httpHeaderValue2: my-vcluster ``` Control plane cluster datasources set `X-Vcluster-Platform-Cluster` instead, and the fleet-wide datasource sets no scope headers. ### Authorization with your access key[​](#authorization-with-your-access-key "Direct link to Authorization with your access key") Every datasource forwards your signed-in Platform access key as the `loft_access_key` cookie to the Query Proxy through Grafana's `keepCookies` setting. This is the session-cookie path noted in [Create a reader key](https://vcluster.com/docs/platform/maintenance/observability/configure-metrics-access.md#create-a-reader-key), so Grafana users don't need a static `metrics-reader` key. The Query Proxy authorizes every query as your user, so the datasource you pick can't widen your access. If you select the fleet-wide datasource, you still only get the metrics your access key is allowed to read. Only Platform admins receive fleet-wide results from it. ## Endpoint[​](#endpoint "Direct link to Endpoint") Query the Query Proxy's HTTP API directly when you automate reads or bring your own query client. Inside the Platform cluster: ``` https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:8081/api/v1/query ``` This endpoint assumes the example connector ID `fleet-observability`. For another ID, replace the Service name with `gateway-`. To connect your own datasource or query client from outside the Platform cluster, expose the gateway with a LoadBalancer and use its address on port `8081` instead of the in-cluster Service name: ``` https://:8081/api/v1/query ``` See [Expose the gateway with a LoadBalancer](https://vcluster.com/docs/platform/maintenance/observability/install-observability-gateway.md#expose-the-gateway-with-a-loadbalancer) to enable it and retrieve the address. Scope headers and `metrics-reader` authorization are unchanged; only the host differs. The gateway always uses TLS. For the default Platform-managed certificate, read `ca.crt` from the `gateway--tls` Secret and make it available to your query client. The examples use its local path through the `GATEWAY_CA_FILE` environment variable: ``` CONNECTOR_ID=fleet-observability kubectl get secret -n vcluster-platform "gateway-$CONNECTOR_ID-tls" -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/gateway-ca.crt export GATEWAY_CA_FILE=/tmp/gateway-ca.crt ``` ## Query a tenant cluster[​](#query-a-tenant-cluster "Direct link to Query a tenant cluster") Send a `metrics-reader` token from [Create a reader key](https://vcluster.com/docs/platform/maintenance/observability/configure-metrics-access.md#create-a-reader-key) and tenant scope headers: ``` curl -G \ --cacert "$GATEWAY_CA_FILE" \ -H "Authorization: Bearer $METRICS_READER_TOKEN" \ -H "X-Vcluster-Platform-Project: my-project" \ -H "X-Vcluster-Platform-Instance: my-vcluster" \ --data-urlencode 'query=sum(rate(container_cpu_usage_seconds_total[5m]))' \ https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:8081/api/v1/query ``` The Query Proxy injects tenant scope into vector selectors before forwarding the query. The backend receives an equivalent query constrained by: ``` vcluster_platform_project="my-project", vcluster_platform_instance="my-vcluster" ``` ## Query a control plane cluster[​](#query-a-control-plane-cluster "Direct link to Query a control plane cluster") ``` curl -G \ --cacert "$GATEWAY_CA_FILE" \ -H "Authorization: Bearer $METRICS_READER_TOKEN" \ -H "X-Vcluster-Platform-Cluster: local-cluster" \ --data-urlencode 'query=sum(rate(apiserver_request_total[5m]))' \ https://gateway-fleet-observability.vcluster-platform.svc.cluster.local:8081/api/v1/query ``` The backend query is constrained by: ``` vcluster_platform_cluster="local-cluster" ``` ## Use labels in PromQL[​](#use-labels-in-promql "Direct link to Use labels in PromQL") You can also include scope labels directly in PromQL. Equality matchers and literal alternation regular expression matchers are allowed on scope-controlling labels. ``` sum by (vcluster_platform_project, vcluster_platform_instance) ( rate(container_cpu_usage_seconds_total{ vcluster_platform_project="my-project", vcluster_platform_instance=~"api|worker" }[5m]) ) ``` The Query Proxy rejects unsafe scope matchers such as `!=`, `!~`, `.*`, or regexes that aren't top-level literal alternations. Descriptive labels can use normal PromQL matchers because they don't drive authorization. ## Example queries[​](#example-queries "Direct link to Example queries") CPU by tenant cluster: ``` sum by (vcluster_platform_project, vcluster_platform_instance) ( rate(container_cpu_usage_seconds_total[5m]) ) ``` Memory by tenant cluster: ``` sum by (vcluster_platform_project, vcluster_platform_instance) ( container_memory_working_set_bytes ) ``` API request rate by cluster: ``` sum by (vcluster_platform_cluster, verb) ( rate(apiserver_request_total[5m]) ) ``` GPU utilization by tenant cluster, when DCGM metrics are collected: ``` avg by (vcluster_platform_project, vcluster_platform_instance, gpu) ( DCGM_FI_DEV_GPU_UTIL ) ``` ## Query failures[​](#query-failures "Direct link to Query failures") If a query is rejected, check: * The bearer token has `metrics-reader` scope or the signed-in user can read metrics for the requested resource. * The scope headers match the key scope. * Scope-controlling label matchers use `=` or a literal `=~` alternation. * The Platform API is reachable from the Query Proxy. For more details, see [Troubleshooting](https://vcluster.com/docs/platform/maintenance/observability/troubleshooting.md). --- # Fleet Observability quickstart This walks through the default, out-of-the-box path. It deploys the bundled Prometheus backend and edge collectors with Argo CD to one tenant cluster, using YAML and `kubectl` throughout. Each step links to a reference page with the full parameter list and the Platform UI walkthrough. See those pages for alternatives, such as bringing your own backend or targeting a control plane cluster. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * The [Argo CD integration](https://vcluster.com/docs/platform/integrations/argocd.md) is enabled on both the cluster where Platform is installed and the tenant cluster you collect from, each referencing an Argo CD connector. See [Connect to Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md). * `kubectl` access to a project namespace. Platform creates a namespace named `p-` for each project, so the default project's namespace is `p-default`. Installations with a customized project namespace prefix, including installations upgraded from older versions, can use a different namespace. * A tenant cluster to collect metrics from. ## 1. Deploy the metrics backend[​](#1-deploy-the-metrics-backend "Direct link to 1. Deploy the metrics backend") Deploy the bundled Prometheus backend to the local cluster, the cluster where Platform itself is installed: fleet-metrics-backend.yaml ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplication metadata: name: fleet-metrics-backend namespace: p-my-project # replace with your project namespace spec: displayName: "Fleet metrics backend" destination: cluster: name: loft-cluster # the reserved name for Platform's local cluster; don't change this templateRef: name: fleet-observability-prometheus parameters: destinationNamespace: vcluster-platform prometheusStorageSize: 20Gi project: default ``` ``` kubectl apply -f fleet-metrics-backend.yaml ``` Confirm the application landed in your project's namespace: ``` kubectl get argocdapplications -n p-default ``` Substitute your project's namespace if you're not using the default project. For other backends, the Platform UI, and the full parameter list, see [Deploy the metrics backend with Argo CD](https://vcluster.com/docs/platform/maintenance/observability/install-observability-gateway.md#deploy-the-metrics-backend-with-argo-cd). ## 2. Create the connector and deploy the gateway[​](#2-create-the-connector-and-deploy-the-gateway "Direct link to 2. Create the connector and deploy the gateway") Create the observability connector Secret. These endpoints match the backend's default namespace from step 1. The Secret below also configures the connector for the bundled Grafana deployed in step 6, using that template's default certificate Secret name: fleet-observability-connector.yaml ``` apiVersion: v1 kind: Secret metadata: name: fleet-observability namespace: vcluster-platform labels: loft.sh/connector-type: observability platform.vcluster.com/fleet-observability-connector: "true" type: Opaque stringData: metricsBackendOtlpEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local/api/v1/otlp metricsBackendPromQLEndpoint: https://prom-prometheus-server.vcluster-platform.svc.cluster.local metricsBackendCertSecretName: metrics-backend-mtls grafanaUrl: https://grafana.vcluster-platform.svc.cluster.local grafanaCertSecretName: grafana-server-tls fleetObservabilityDashboardUid: vcluster-fleet-observability ``` ``` kubectl apply -f fleet-observability-connector.yaml ``` Platform deploys the gateway from this Secret. Check that the gateway pod comes up: ``` kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance=fleet-observability ``` For connector keys, TLS options, and the Platform UI, see [Install the observability gateway](https://vcluster.com/docs/platform/maintenance/observability/install-observability-gateway.md). ## 3. Create a metrics-writer key[​](#3-create-a-metrics-writer-key "Direct link to 3. Create a metrics-writer key") Create a `metrics-writer` access key scoped to your tenant cluster. Set `PROJECT` and `TENANT_CLUSTER` to your tenant cluster's project and instance name. The command prints the generated token as `token: ` at the end. Modify the following with your specific values to generate a copyable command: PROJECTdefault TENANT\_CLUSTERmy-vcluster ``` PROJECT=default TENANT_CLUSTER=my-vcluster KEY=$(head -c 20 /dev/urandom | od -An -tx1 | tr -d ' \n') kubectl apply -f - </grafana/` with Platform SSO to see the pre-provisioned Fleet Observability, Cluster Metrics, and GPU Overview dashboards. For the connector settings Grafana needs and the full dashboard list, see [Deploy the bundled Grafana with Argo CD](https://vcluster.com/docs/platform/maintenance/observability/query-fleet-metrics.md#deploy-the-bundled-grafana-with-argo-cd). ## Next steps[​](#next-steps "Direct link to Next steps") * [Query fleet metrics](https://vcluster.com/docs/platform/maintenance/observability/query-fleet-metrics.md) for dashboards and PromQL access. * [GPU observability templates](https://vcluster.com/docs/platform/maintenance/observability/gpu-observability-templates.md) to add the GPU stack and NVSentinel. * [Troubleshooting](https://vcluster.com/docs/platform/maintenance/observability/troubleshooting.md) if something isn't working. --- # Troubleshooting Fleet Observability Use this page to diagnose the Fleet Observability path from edge collectors through the observability gateway and into your metrics backend. ## Gateway resources aren't created[​](#gateway-resources-arent-created "Direct link to Gateway resources aren't created") Check the connector Secret: ``` kubectl get secret -n vcluster-platform fleet-observability -o yaml ``` Verify: * The Secret is in the Platform namespace. * `metadata.labels["loft.sh/connector-type"]` is `observability`. * The fleet connector has the `platform.vcluster.com/fleet-observability-connector` label set to `"true"`. * `metricsBackendOtlpEndpoint` and `metricsBackendPromQLEndpoint` are set. The gateway isn't reconciled until both backend endpoints are present. ## Multiple fleet connectors[​](#multiple-fleet-connectors "Direct link to Multiple fleet connectors") Only one connector can carry the fleet label: ``` platform.vcluster.com/fleet-observability-connector: "true" ``` If multiple connector Secrets have this label, Platform still creates per-connector gateway resources, but it doesn't publish the fleet-wide platform admin Grafana datasource. Remove the fleet label from all but one connector. ## Collector writes are rejected[​](#collector-writes-are-rejected "Direct link to Collector writes are rejected") Check the collector exporter configuration: * The `Authorization` header uses a valid bearer token. * The token is a `metrics-writer` access key. * The request includes either `X-Vcluster-Platform-Cluster` or the `X-Vcluster-Platform-Project` and `X-Vcluster-Platform-Instance` pair. * The header values match the key scope. Check gateway logs: ``` kubectl logs -n vcluster-platform deploy/gateway-fleet-observability -c otel-gateway ``` Common causes: | Symptom | Likely cause | | ------------------------- | ---------------------------------------------------------------------------------------- | | Missing bearer token | Collector didn't set the `Authorization` header. | | Scope denied | Key scope doesn't include the requested cluster or tenant cluster. | | Ambiguous scope | Request included both cluster and tenant scope, or multiple values for one scope header. | | Headerless write rejected | The key has zero or multiple concrete tenant scopes, so scope can't be derived. | ## Queries are rejected[​](#queries-are-rejected "Direct link to Queries are rejected") Check query-proxy logs: ``` kubectl logs -n vcluster-platform deploy/gateway-fleet-observability -c query-proxy ``` Common causes: | Symptom | Likely cause | | ------------------------------ | --------------------------------------------------------------------------------------------------------------- | | HTTP `400` | Missing scope for a non-admin user, invalid scope matcher, or malformed PromQL. | | HTTP `403` | The user or key is authenticated but not authorized for the requested metrics scope. | | Query succeeds for admins only | The datasource is using the unscoped fleet-admin path. Add tenant or cluster scope headers for non-admin users. | | Regex query rejected | Scope-controlling labels only allow equality or literal alternation regular expression matchers. | ## Platform labels are missing[​](#platform-labels-are-missing "Direct link to Platform labels are missing") If metrics arrive without `vcluster_platform_*` labels: * Confirm the metrics went through the Write Gateway. Direct writes to the backend don't get platform labels. * Confirm the collector used a valid metrics writer key and scope headers. * Confirm the backend stores labels from OTLP datapoint attributes. * Check whether the metric is expected to have tenant labels or only cluster labels. The Write Gateway strips caller-supplied platform labels and then stamps authenticated values. Empty authenticated values are dropped. ## Backend TLS failures[​](#backend-tls-failures "Direct link to Backend TLS failures") If gateway pods start but can't write or query the backend: * Verify `metricsBackendOtlpEndpoint` and `metricsBackendPromQLEndpoint`. * Set `metricsBackendServerName` when the backend certificate's DNS name differs from the endpoint host. * Use `metricsBackendCertSecretName` when the backend requires mTLS. This Secret is also the source of trust for a private backend CA. It must contain `ca.crt`, `tls.crt`, and `tls.key`. * Use `metricsBackendInsecureSkipVerify: "true"` only as a temporary test setting. `remote error: tls: certificate required` This error means the backend rejected the connection because the gateway didn't present a client certificate, not because the gateway distrusts the backend's certificate. `metricsBackendInsecureSkipVerify` only skips the gateway's verification of the backend's server certificate, so it can't fix this. Set `metricsBackendCertSecretName` instead. The bundled Prometheus backend enforces mTLS and expects this client certificate. When deployed with its default parameters, it's the auto-generated `metrics-backend-mtls` Secret. See [Deploy the metrics backend with Argo CD](https://vcluster.com/docs/platform/maintenance/observability/install-observability-gateway.md#deploy-the-metrics-backend-with-argo-cd). ## Gateway TLS failures[​](#gateway-tls-failures "Direct link to Gateway TLS failures") The gateway always serves OTLP and query traffic over TLS. If collectors or query clients can't connect: * For the Platform-managed certificate, read `ca.crt` from the `gateway--tls` Secret and configure the client to trust it. * For a custom private certificate, include `ca.crt` in the Secret referenced by `observabilityGatewayCertSecretName`. * Verify that the client uses `https` for OTLP HTTP and query traffic or enables TLS for OTLP gRPC. * If the gateway uses a `LoadBalancer` Service and a Platform-managed certificate, wait for the LoadBalancer address before expecting the gateway Deployment to become available. ## Observability pods are OOMKilled or evicted[​](#observability-pods-are-oomkilled-or-evicted "Direct link to Observability pods are OOMKilled or evicted") The bundled templates deploy Prometheus, Grafana, and the collectors without an explicit `resources` block, so those pods likely end up in the `BestEffort` QoS class, which the kubelet evicts first under node pressure. Check the QoS class, last container state, and status message: ``` kubectl describe pod -n vcluster-platform ``` Look for `QoS Class`, `Last State`, and `Message` in the output. The fix depends on which failure you see: * `OOMKilled`: raise the memory limit on the affected template's `resources` parameter. * `Evicted`: set memory requests so the kubelet ranks the pod above pods that exceed their own requests. Only memory, disk, and PID pressure trigger eviction, not CPU. Set requests and limits on the affected template: * Prometheus: the `resources` parameter. See [Deploy the metrics backend with Argo CD](https://vcluster.com/docs/platform/maintenance/observability/install-observability-gateway.md#deploy-the-metrics-backend-with-argo-cd). * Grafana: the `resources` parameter. See [Deploy the bundled Grafana with Argo CD](https://vcluster.com/docs/platform/maintenance/observability/query-fleet-metrics.md#deploy-the-bundled-grafana-with-argo-cd). * Collectors: the `otelDaemonResources` and `otelDeployResources` parameters. See [Deploy collectors with Argo CD](https://vcluster.com/docs/platform/maintenance/observability/configure-edge-collectors.md#deploy-collectors-with-argo-cd). Prometheus is the most likely to be OOMKilled, because its memory grows with active series count. ## Platform API outage[​](#platform-api-outage "Direct link to Platform API outage") The gateway and query proxy fail closed when Platform authorization is unavailable. Previously allowed decisions can continue only until their short positive cache TTL expires. Expected behavior: * Collectors may queue and retry writes locally, depending on their exporter settings. * Queries fail visibly after cached authorization decisions expire. * Writes and reads recover when the Platform API is reachable again. ## Useful checks[​](#useful-checks "Direct link to Useful checks") Gateway pods: ``` CONNECTOR_ID=fleet-observability kubectl get pods -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID" ``` Gateway service: ``` CONNECTOR_ID=fleet-observability kubectl get svc -n vcluster-platform "gateway-$CONNECTOR_ID" ``` Write Gateway metrics: ``` CONNECTOR_ID=fleet-observability GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID" -o jsonpath='{.items[0].metadata.name}') kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8889:8889 curl http://127.0.0.1:8889/metrics ``` Query Proxy metrics: ``` CONNECTOR_ID=fleet-observability GATEWAY_POD=$(kubectl get pod -n vcluster-platform -l app.kubernetes.io/name=gateway,app.kubernetes.io/instance="$CONNECTOR_ID" -o jsonpath='{.items[0].metadata.name}') kubectl port-forward -n vcluster-platform pod/$GATEWAY_POD 8080:8080 curl http://127.0.0.1:8080/metrics ``` --- # High Availability Installation | | | Enterprise | | | | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [High-Availability Mode](https://vcluster.com/docs/platform/maintenance/reliability-scaling/high-availability.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). By default, the platform runs in a cluster as a single replica without high availability. Licensed Feature Running the platform in high availability mode is an enterprise feature. Platform HA keeps the vCluster Platform management plane available during replica or node failures. Tenant cluster availability also depends on the tenant cluster control plane and the Kubernetes cluster that hosts it. For more information, see [What happens during control plane outages?](https://vcluster.com/docs/platform/understand/control-plane-outages.md). ## Enable high availability and run the platform with multiple replicas[​](#enable-high-availability-and-run-the-platform-with-multiple-replicas "Direct link to Enable high availability and run the platform with multiple replicas") To configure the platform to be highly available, scale up the replicas and the platform starts with leader election mode. You can do this using Helm on an existing platform installation: Run the platform with 3 replicas ``` helm upgrade loft vcluster-platform --repo https://charts.loft.sh/ \ --namespace vcluster-platform \ --reuse-values \ --set replicaCount=3 ``` Now check that 3 replicas are running: Verify running replicas ``` kubectl get po -n vcluster-platform ``` Expected output: ``` NAME READY STATUS RESTARTS AGE loft-84bfdb746c-7t922 1/1 Running 0 40s loft-84bfdb746c-jpwlz 1/1 Running 0 40s loft-84bfdb746c-9c4jx 1/1 Running 0 40s ``` Recommended number of replicas For optimal high availability, it is recommended to run at least 3 replicas of the platform. ## How does it work?[​](#how-does-it-work "Direct link to How does it work?") To understand how the platform can be made highly available and scaled to multiple replicas, it's important to examine its core components: 1. **API Server**: The platform starts an internal Kubernetes API server that handles most of the functional requests it receives. Since the platform saves everything in Kubernetes resources rather than storing data in volumes, it can easily scale horizontally. 2. **Controller**: A significant part of the platform reacts to Kubernetes resource changes and modifies other Kubernetes resources in return. This component is similar to the [`kube-controller-manager`](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/) which executes the core control loops for Kubernetes. Running multiple instances of this component would result in conflicting changes and race conditions among the controllers, which is why leader election is used. Leader election Leader election is a process that allows running multiple instances where a single leader is elected to execute the actual capability, while all other instances are waiting. When the leader fails for any reason, another replica takes over. For simplicity and performance, both components are compiled into the same binary in the platform. Scaling of the platform works differently than with Kubernetes components by using a combination of leader election and simple horizontal scaling to ensure high availability. The modes for a platform replica are: 1. **Leader Mode**: If a platform replica runs as a leader, all capability is started normally, so both API server and controllers are running. 2. **Non-Leader Mode**: If a platform replica runs as a non-leader, all capability except the controllers are started, so only the API server is running. If the leader fails for any reason and this replica is made the new leader, the platform simply starts the controller component within this replica. If configured, the platform automatically determines which replica to run in leader mode and which to run in non-leader mode. With this approach, it is possible to run multiple instances of the API server and allow horizontal scaling for that component, while ensuring the controllers are only run once. --- # Uninstall This guide provides instructions on how to uninstall the platform from your Kubernetes cluster. warning This action is performed against the cluster the kube-context is pointing to, not the platform instance that is logged in. It does not require logging in to the platform. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Your current kube-context must have administrative privileges, which you can verify with `kubectl auth can-i create clusterrole -A` info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm` installed: Helm v3.10.0+ is required for deploying the platform. Refer to the [Helm Installation Guide](https://helm.sh/docs/intro/install/) if you need to install it. * `kubectl` installed: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. ## Uninstall[​](#uninstall "Direct link to Uninstall") There are two ways to uninstall the platform. * Using vCluster CLI * Using Helm info This feature was introduced in **vCluster** version v0.24 ### Delete the platform[​](#delete-the-platform "Direct link to Delete the platform") Note that the [VirtualClusterInstances](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance/.md) CRDs managed with driver Helm are deleted, but the underlying tenant clusters are not uninstalled. On the Kubernetes cluster where the platform is installed, and all connected clusters, run: Delete the platform using vCluster CLI ``` vcluster platform destroy ``` info The command does the following: * Uninstalls Helm releases related to vCluster Platform. * Deletes CustomResourceDefinitions (CRDs) created by the platform. * Optionally deletes the namespace where the platform is installed when the `--delete-namespace` flag is provided. * Removes cluster roles and cluster role bindings created by the platform. In order to force delete all the platform resources, you can use the flag `--force-remove-finalizers`. This will remove the finalizers from the resources, forcing the platform resources to be deleted immediately. Force delete the platform using vCluster CLI ``` vcluster platform destroy --force-remove-finalizers ``` warning Removing finalizers may cause unintended behaviours like leaving some resources behind, but will ensure the platform is uninstalled. ### Delete Helm release[​](#delete-helm-release "Direct link to Delete Helm release") Delete the platform Helm release ``` helm delete -n vcluster-platform loft ``` ### Delete namespace (optional)[​](#delete-namespace-optional "Direct link to Delete namespace (optional)") Deleting the namespace ensures that all resources associated with the platform are removed. If you have deleted the namespace **without** deleting the vcluster-platform and loft-agent Helm releases beforehand, ensure the api-services and webhooks are deleted correctly: Delete API service and webhook ``` kubectl delete apiservice v1.cluster.loft.sh kubectl delete validatingwebhookconfiguration loft-agent ``` ### Delete CRDs (optional)[​](#delete-crds-optional "Direct link to Delete CRDs (optional)") Delete platform CRDs ``` kubectl api-resources --api-group='storage.loft.sh' -o name | xargs kubectl delete crd ``` warning Deleting the platform CRDs will also remove all platform objects inside your cluster, such as Cluster, Team, and User. This action cannot be undone. ## Troubleshoot[​](#troubleshoot "Direct link to Troubleshoot") If a resource is stuck or encounters an error during deletion, refer to the [troubleshooting](https://vcluster.com/docs/platform/use-platform/troubleshooting/.md) section for assistance. --- # Lifecycle policy The following defines the platform's lifecycle policy for releases and maintenance. The platform aims to release on a regular cadence and follow [semantic versioning](https://semver.org/). We encourage users to run the latest stable release to stay up to date on features and security patches. Due to the rapid development of the platform, each minor version receives active support for 6 months after its release. Active support ends on the End of Support (EOS) date. For 3 months after the EOS date, the version receives only critical security updates. After the End of Life (EOL) date, the version receives no updates. During active support, updates from the active development branch can be backported upon request. Long Term Support (LTS) releases have longer support windows. See [LTS releases](#lts-releases). Versions not on the following list are no longer in any support window. ## Platform supported versions[​](#platform-supported-versions "Direct link to Platform supported versions") | Release | Release Date | End of Support (EOS) End Date | End of Life (EOL) End Date | Default vCluster Version | | :--------------------------------------: | ------------ | ----------------------------- | -------------------------- | ------------------------ | | v4.12 (LTS) | 2026 Sep 08 | 2027 Oct 15 | 2028 Apr 15 | v0.37 | | v4.11 | 2026 Jul 22 | 2027 Jan 22 | 2027 Apr 22 | v0.36 | | v4.10 | 2026 Jun 15 | 2026 Dec 15 | 2027 Mar 15 | v0.35 | | v4.9 | 2026 Apr 29 | 2026 Oct 29 | 2027 Jan 29 | v0.34 | | v4.8 | 2026 Mar 16 | 2026 Sep 16 | 2026 Dec 16 | v0.33 | | v4.7 | 2026 Feb 18 | 2026 Aug 18 | 2026 Nov 18 | v0.32 | | *Versions below are no longer supported* | | | | | | v4.6 | 2026 Jan 29 | 2026 Jul 29 | 2026 Oct 29 | v0.31 | | v4.5 | 2025 Oct 29 | 2026 Apr 29 | 2026 Jul 29 | v0.30 | | v4.4 | 2025 Sep 09 | 2026 Mar 09 | 2026 Jun 09 | v0.28 | | v4.3 | 2025 Jun 03 | 2025 Dec 03 | 2026 Mar 06 | v0.25 | | v4.2 | 2024 Dec 17 | 2025 Sep 17 | 2025 Dec 17 | v0.21 | | v4.1 | 2024 Nov 12 | 2025 Feb 12 | 2025 May 12 | v0.21 | | v4.0 | 2024 Oct 15 | 2025 Jan 15 | 2025 Apr 15 | v0.20 | | v3.4 | 2024 Feb 29 | 2025 Apr 01\* | 2025 Jul 01 | v0.19 | \* Due to the number of breaking changes from vCluster v0.19 to v0.20 and vCluster Platform v3.4 to v4.0, v3.4 has an extended EOS end date. Review the [migration guide](https://platform-v4-5--vcluster-docs-site.netlify.app/docs/platform/reference/migrations/4-0-migration) to upgrade your platform from v3.4 to v4.0. ## LTS releases[​](#lts-releases "Direct link to LTS releases") vCluster Platform v4.12 and vCluster v0.37 are designated LTS releases. LTS support covers the two releases as a pair, because the platform and the clusters it manages are built and tested together. To stay on LTS, run vCluster Platform v4.12 with vCluster v0.37 for your clusters. vMetal is part of vCluster Platform, so vMetal in v4.12 has the same LTS dates. | Phase | Standard release | LTS release | | --------------------------- | ---------------- | ----------- | | Active support | 6 months | 12 months | | End of Support (EOS) period | 3 months | 6 months | During active support, you can request backports of fixes from the active development branch, including support for newer Kubernetes versions. During the EOS period, LTS releases receive only critical security updates. If you need support for a newer Kubernetes version during this period, contact vCluster Labs support. Clusters that run a vCluster version other than v0.37 follow the standard lifecycle for that version. Upgrading either product past its LTS release takes you off LTS. See the [vCluster lifecycle policy](https://vcluster.com/docs/vcluster/manage/upgrade/supported_versions.md) for vCluster dates. Support terminology * **End of Support (EOS)**: The period when active support for a version has ended and only critical security updates are provided. This period begins on the EOS date and lasts 3 months, or 6 months for LTS releases. * **End of Life (EOL)**: The period when all support, including security updates, has ended for a version. This period begins on the EOL date and continues indefinitely. ## Feature Versioning[​](#feature-versioning "Direct link to Feature Versioning") While most features we release are fully supported, some are introduced at an earlier stage of development. These stages generally align with [Kubernetes conventions](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-stages), with the exception of [experimental features](https://www.vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/) which may be removed at any time. * **Alpha**: Newly introduced features which may include breaking changes in future releases. * **Beta**: Features that are more stable and considered safe, but not recommended for business critical applications. --- # Migrate the platform to a different cluster This guide explains how to migrate the platform to a different Kubernetes cluster, which might be necessary during cluster decommissioning or cloud platform migration. ## When to migrate[​](#when-to-migrate "Direct link to When to migrate") Migrating the platform to a new Kubernetes cluster is necessary in certain cases, such as infrastructure upgrades, cluster decommissioning, or scaling needs. Migrate the platform in the following situations: * The existing cluster is being decommissioned. The platform must move to a new cluster. * The platform is migrating to a new cluster. The original cluster is going to be cleaned up. * The platform is migrating to a new cluster. The original cluster is going to remain connected. ### Migration considerations[​](#migration-considerations "Direct link to Migration considerations") warning The platform must run on a dedicated Kubernetes cluster. It must have its own external access (using ingress or LoadBalancer) and DNS record before migration. Ensure these conditions are met before proceeding. Review the following factors before migrating. These scenarios require a different migration approach. * **Loft Router configurations**
If the platform uses the Loft Router (``), special DNS handling is required. The DNS records must be updated to reflect the new cluster. Failure to update DNS settings can cause routing issues and service downtime. * **Automated deployment tools**
If the platform is managed by an automated deployment tool such as `ArgoCD`, migration must be coordinated with GitOps workflows. Changes to cluster configuration must be reflected in the repository before deployment. Skipping this step can lead to drift between the declared and actual state of the platform. * **Tenant cluster dependencies**
If the platform has tenant cluster instances running on the same Kubernetes control plane cluster, these instances do not migrate automatically. They require a separate migration process after the platform has been moved. Migrating only the platform without considering tenant clusters can cause service disruptions and dependency issues. * **Tenant cluster agent on the destination cluster**
If the destination Kubernetes cluster has a tenant cluster agent, it must be decommissioned before migration. The agent can interfere with the new platform installation, potentially leading to scheduling conflicts and unexpected behavior. Not addressing these considerations before migration can lead to DNS misconfigurations, deployment failures, and conflicts with existing workloads. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") ### Base prerequisites[​](#base-prerequisites "Direct link to Base prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Your current kube-context must have administrative privileges, which you can verify with `kubectl auth can-i create clusterrole -A` info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm` installed: Helm v3.10.0+ is required for deploying the platform. Refer to the [Helm Installation Guide](https://helm.sh/docs/intro/install/) if you need to install it. * `kubectl` installed: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. ### Platform specific prerequisites[​](#platform-specific-prerequisites "Direct link to Platform specific prerequisites") * Ensure at least 1 GB of available storage space for the backup. * Confirm that you have access to DNS management for the platform's domain. * Only one platform instance can run in a single Kubernetes cluster. Do not install the platform more than once in the same control plane cluster. * Plan for 15–30 minutes of downtime during the migration process. Communicate this downtime to end users in advance. * Use the same platform version for both the source and destination installations. Migration does not support version changes. ## Migration steps[​](#migration-steps "Direct link to Migration steps") 1. ### Install the platform in the new cluster[​](#install-platform-new-cluster "Direct link to Install the platform in the new cluster") Use the following configuration settings to complete the installation in the new cluster: * Use the Helm values from the original installation, making adjustments to values like `ingressClass` or `storageClass` as necessary. * If your project namespaces use the `loft-p-` prefix, set `projectNamespacePrefix: loft-p-` in your configuration. 2. Install the platform in the new cluster by following the [quick start guide](https://vcluster.com/docs/platform/install/quick-start-guide.md). Using a temporary hostname during installation When using cloud-managed ingress, use a temporary hostname to prevent conflicts with the running platform instance. 3. ### Prepare for migration[​](#prepare-for-migration "Direct link to Prepare for migration") Send downtime communication to end users. 4. Optional: Stop `ArgoCD` sync of any tenant clusters. 5. [Perform a backup of the source platform](https://vcluster.com/docs/platform/maintenance/backup-restore/backup-restore-platform.md). 6. ### Execute migration[​](#execute-migration "Direct link to Execute migration") Scale down the source platform: Scale down the platform deployment ``` kubectl scale deployment loft --replicas=0 -n vcluster-platform ``` 7. [Restore the backup to the new installation](https://vcluster.com/docs/platform/maintenance/backup-restore/backup-restore-platform.md#restore-vcluster-platform-from-backup). 8. Apply the license certificate to the new cluster: Capture certificate data from the source cluster ``` TARGET_NAMESPACE='vcluster-platform' CA_CERT=$(kubectl get secret loft-cert -n vcluster-platform -o jsonpath="{.data.ca\.crt}") TLS_CERT=$(kubectl get secret loft-cert -n vcluster-platform -o jsonpath="{.data.tls\.crt}") PRIVATE_KEY=$(kubectl get secret loft-cert -n vcluster-platform -o jsonpath="{.data.tls\.key}") cat < loft-cert-clean.yaml apiVersion: v1 kind: Secret metadata: name: loft-cert namespace: ${TARGET_NAMESPACE} type: kubernetes.io/tls data: ca.crt: ${CA_CERT} tls.crt: ${TLS_CERT} tls.key: ${PRIVATE_KEY} EOF ``` Apply certificate to the target cluster ``` kubectl scale deployment loft --replicas=0 -n vcluster-platform kubectl delete secret loft-cert -n ${TARGET_NAMESPACE} --ignore-not-found kubectl create -f loft-cert-clean.yaml kubectl scale deployment loft --replicas=1 -n vcluster-platform ``` 9. Update the hostname in the platform configuration as described in the [domain configuration guide](https://vcluster.com/docs/platform/configure/installation-options/domain.md). 10. Update the DNS record to point to the new cluster, if needed. 11. When using the original cluster as a connected cluster, connect it to the new platform installation. ## Post-migration validation[​](#post-migration-validation "Direct link to Post-migration validation") 1. Log in to the platform UI and verify object presence: * Check for all expected projects. * Verify user access and permissions. * Confirm tenant cluster listings. 2. Verify license status in the platform UI: * Navigate to **Admin > Platform Config > License and Billing**. * Confirm license is active and correct. 3. Restart platform agents on connected clusters: Restart platform agent ``` kubectl rollout restart deployment loft -n vcluster-platform ``` 4. Validate the core capabilities of the platform: * Single sign-on: Test logging in with all configured providers. * Log access: Verify log retrieval from multiple tenant clusters. * Tenant cluster creation: Create a test cluster. * Auto sleep: Test the sleep/wake feature. 5. Optional: Restart `ArgoCD` sync if previously stopped. 6. Send end-of-downtime communication to end users. --- # Upgrade vCluster Platform Upgrade vCluster Platform using the vCluster CLI, Helm, or GitOps, matching however you originally installed it. If you installed in an air-gapped environment, see [Upgrade in an air-gapped environment](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade-air-gapped.md) instead of the steps below. warning If you are using a vCluster Platform version earlier than v4.0, upgrade to v4.0 first by following the [v3 to v4 upgrade instructions](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade-v3-to-v4.md). Platform 4.12 also removes the resources that deployed apps before this release. If you have any apps deployed, see [Upgrade to Platform 4.12](#upgrade-to-4-12) after completing the upgrade steps below. Upgrade the platform Upgrade considerations * Review the release notes for the target version to understand any breaking changes or new features. * Test the upgrade in a non-production environment before applying it to your production setup. Upgrade the platform using: * CLI * Helm To upgrade the platform using the `vcluster` CLI, update `$PLATFORM_VERSION` with a valid platform version and run: Upgrade the platform using vCluster CLI ``` RELEASE_NAMESPACE=vcluster-platform # Replace with the namespace of the platform deployment if different PLATFORM_VERSION='' # Set this to a specific version or leave empty for latest vcluster platform start --upgrade --version=$PLATFORM_VERSION --namespace=$RELEASE_NAMESPACE --values=vcluster-platform.yaml ``` To upgrade the platform using `helm`, update `$PLATFORM_VERSION` with a valid platform version and run: Upgrade the platform using helm ``` RELEASE_NAME=vcluster-platform # Replace with the release name if different RELEASE_NAMESPACE=vcluster-platform # Replace with the namespace of the platform deployment if different PLATFORM_VERSION='' # Set this to a specific version or leave empty for latest helm upgrade $RELEASE_NAME vcluster-platform -n $RELEASE_NAMESPACE --repository-config '' --repo https://charts.loft.sh \ --version $PLATFORM_VERSION \ --reuse-values \ -f vcluster-platform.yaml ``` Server-side apply with Helm 4 Helm 4 keeps the apply method a release was created with, so upgrades need no extra flags. If you switch a release to server-side apply with `--server-side=true`, the upgrade fails with a conflict on any field the platform has written itself. The most common one is `data.config` in the `loft-manager-config` secret after you edit the config in the UI. Add `--force-conflicts` to let Helm take ownership of those fields. Either way, keep the `config` section of your values file in sync with changes made in the UI. The upgrade replaces the secret with the values from the file. Versions and Values The `$PLATFORM_VERSION` environment variable specifies the vCluster Platform version to deploy.
The `vcluster-platform.yaml` file is optional and defines Helm values to use when upgrading the vCluster Platform deployment. ## Upgrade a GitOps-managed deployment[​](#upgrade-a-gitops-managed-deployment "Direct link to Upgrade a GitOps-managed deployment") If you installed the platform using GitOps (see [Install with ArgoCD](https://vcluster.com/docs/platform/install/gitops.md)), upgrade by updating the chart version in your GitOps manifests and letting your GitOps tool apply it, rather than running `vcluster platform start` or `helm upgrade` directly against the cluster. * ArgoCD * Flux Update `spec.source.targetRevision` on the platform `Application` to the target version, then sync. Update the platform Application ``` apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: vcluster-platform namespace: argocd spec: source: targetRevision: "4.12.0" # Replace with the target version chart: vcluster-platform repoURL: "https://charts.loft.sh" ``` Commit the change to the repository ArgoCD watches. With automated sync enabled, ArgoCD applies the new chart version on its next sync. To apply it right away instead of waiting, trigger a sync: Trigger a sync ``` argocd app sync vcluster-platform ``` The platform creates and updates its own CRDs when it starts, so the `Application` needs no CRD-specific sync options. Update `spec.chart.spec.version` on the platform `HelmRelease` to the target version, then reconcile. Update the platform HelmRelease ``` apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: vcluster-platform namespace: vcluster-platform spec: chart: spec: version: "4.12.0" # Replace with the target version chart: vcluster-platform sourceRef: kind: HelmRepository name: vcluster-platform ``` Commit the change to the repository Flux watches. Flux applies the new chart version on its next reconciliation. To apply it right away instead of waiting, trigger a reconciliation: Trigger a reconciliation ``` flux reconcile helmrelease vcluster-platform -n vcluster-platform ``` The platform creates and updates its own CRDs when it starts, so the `HelmRelease` needs no CRD policy settings. ## Verify the upgrade[​](#verify-the-upgrade "Direct link to Verify the upgrade") These checks apply to every upgrade method. Replace `vcluster-platform` with your platform namespace if you installed it elsewhere. Wait for the rollout to finish. Wait for the rollout ``` kubectl rollout status deployment/loft -n vcluster-platform ``` Confirm the deployment runs the version you upgraded to. Check the running image ``` kubectl get deployment -n vcluster-platform loft \ -o jsonpath='{.spec.template.spec.containers[0].image}' ``` The platform upgrades connected agents automatically once its own upgrade completes, so you don't upgrade them separately. To check one, run the same command against the connected cluster's context, using the namespace the agent is installed in. If the rollout doesn't finish, inspect the pod and its logs. Inspect the platform pod ``` kubectl get pods -n vcluster-platform -l app=loft kubectl logs -n vcluster-platform -l app=loft --tail=50 ``` ## Upgrade to Platform 4.12[​](#upgrade-to-4-12 "Direct link to Upgrade to Platform 4.12") If you have apps deployed on a cluster, tenant cluster, or space, upgrading to Platform 4.12 doesn't disrupt them. Each app's underlying Helm release keeps running exactly as before. What changes is how the platform manages that release. Platform 4.12 removes the old resources that used to represent a deployed app: * The old **Task** API. * The **HelmRelease** resource, in both its cluster-scoped variant and the legacy `virtualcluster.loft.sh/v1` API group variant. * The chart-discovery endpoints that backed the old Apps UI. Once these are gone, the platform can no longer update, upgrade, or delete an app through the old flow. Until you migrate it, an app shows up as a plain Helm release with no platform-side management. Migrate each app to keep managing it through the platform. Not the vCluster chart's own HelmRelease This removal doesn't affect the `spec.template.helmRelease` field on a tenant cluster or tenant cluster template. That field describes the vCluster chart's own Helm release, and is unrelated to the removed app-deployment resources. ### Migrate an existing app deployment[​](#migrate-an-existing-app-deployment "Direct link to Migrate an existing app deployment") These resources are replaced by [**AppInstance**](https://vcluster.com/docs/platform/api/resources/appinstance.md), which deploys an [App](https://vcluster.com/docs/platform/understand/what-are-apps.md) into a tenant cluster, space, or connected cluster. Migrating an existing app to AppInstance is a UI action, not a CLI or API migration. To find the migration action, open the App's cluster or tenant cluster and click the **Apps** button. This is the same view used to [update an installed App](https://vcluster.com/docs/platform/use-platform/apps/updating.md#update-an-installed-app-version). The Apps table's Type column tells these apart. A **Legacy App** row shows the **Migrate** action when its original App still exists, and the App must include the version referenced by the Helm release. A **Template App** row is declared in the tenant cluster or space configuration itself, continues to use that configuration, and doesn't need this migration. The **Migrate** action: 1. Reads the app name and version from the annotations the old flow set on the existing Helm release. 2. Opens a prefilled AppInstance form so you can review the release name, namespace, app version, and parameters before saving. 3. Creates an AppInstance that runs `helm upgrade --install` against the existing release name and namespace. This adopts the release in place instead of installing a second release. Because saving the AppInstance performs a Helm upgrade, review the prefilled version and parameters first. Whether the application's workloads restart depends on the chart and resulting manifest changes. Migrate each Legacy App that you want the platform to continue managing. There's no deadline, but an App remains unmanaged by the platform until you migrate it. If the action isn't available, confirm that the original App and referenced version still exist. Restore the App definition before migrating, or manage the existing Helm release outside the platform. ### Move standalone snapshot credentials[​](#move-standalone-snapshot-credentials "Direct link to Move standalone snapshot credentials") Platform 4.12 ignores `snapshots.auto.storage..credential.secretNamespace` for standalone tenant clusters, and instead always reads the credential Secret from the Platform project namespace that contains the VirtualClusterInstance. Before upgrading each affected standalone tenant cluster: 1. Find the credential `secretName` and any `secretNamespace` configured for S3, OCI, or Azure snapshot storage. 2. Copy or recreate that Secret in the Platform project namespace that contains the VirtualClusterInstance. Skip this step when the Secret is already there. 3. Remove `secretNamespace` from the tenant cluster configuration and any template that supplies it. 4. Confirm that the Secret contains the required keys for the configured storage provider. Configuration before Platform 4.12 ``` snapshots: auto: storage: type: s3 s3: credential: # Platform 4.12 ignores this namespace for standalone tenant clusters. secretNamespace: backup-secrets secretName: aws-credentials ``` Configuration for Platform 4.12 ``` snapshots: auto: storage: type: s3 s3: credential: # Create this Secret in the VirtualClusterInstance project namespace. secretName: aws-credentials ``` Pod-based tenant clusters continue to resolve `secretNamespace` in their connected control plane cluster. For standalone requirements and storage limitations, see [Auto snapshots on standalone tenant clusters](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/snapshots.md#auto-snapshots-on-standalone-tenant-clusters). --- # Upgrade in an air-gapped environment This page covers upgrading vCluster Platform when it was installed in an air-gapped environment (see [With Offline License Key](https://vcluster.com/docs/platform/install/air-gapped.md) or [With Offline License Server](https://vcluster.com/docs/platform/install/air-gapped/with-offline-license-server.md)). Unlike the [regular upgrade](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade.md), an air-gapped upgrade starts by re-populating your private registry with the target version's images and Helm chart. Your control plane cluster has no route to the public registries vCluster Labs publishes to, so it can only pull what you've mirrored. ## Step 1 - Populate the registry with the target version[​](#step-1---populate-the-registry-with-the-target-version "Direct link to Step 1 - Populate the registry with the target version") On a machine with internet access, repeat the registry-population steps from the install guide for the version you're upgrading to. This is the same `download-images.sh`, `push-images.sh`, `helm pull`, and `helm push` workflow used during install. See [Populate images to private registry](https://vcluster.com/docs/platform/install/air-gapped.md#populate-images-to-private-registry) and [Populate the Helm charts to a private registry](https://vcluster.com/docs/platform/install/air-gapped.md#populate-the-helm-charts-to-a-private-registry) for the full walkthrough, including multi-platform images and insecure registries. ### Platform images and chart[​](#platform-images-and-chart "Direct link to Platform images and chart") Modify the following with your specific values to generate a copyable command: PLATFORM\_VERSION4.12.0 REGISTRYecr.io/myteam Mirror the platform images and chart ``` export PLATFORM_VERSION=4.12.0 export REGISTRY=ecr.io/myteam wget -O images.txt https://github.com/loft-sh/loft/releases/download/v${PLATFORM_VERSION}/images.txt wget -O download-images.sh https://github.com/loft-sh/loft/releases/download/v${PLATFORM_VERSION}/download-images.sh wget -O push-images.sh https://github.com/loft-sh/loft/releases/download/v${PLATFORM_VERSION}/push-images.sh chmod +x ./download-images.sh ./push-images.sh ./download-images.sh --image-list images.txt --images loft-images.tar.gz ./push-images.sh --registry ${REGISTRY} --images loft-images.tar.gz --image-list images.txt helm pull vcluster-platform --repo https://charts.loft.sh --version ${PLATFORM_VERSION} helm push vcluster-platform-${PLATFORM_VERSION}.tgz oci://${REGISTRY}/charts ``` ### vCluster images and chart[​](#vcluster-images-and-chart "Direct link to vCluster images and chart") Do this only if you're also moving tenant clusters to a new vCluster version. Both projects publish a release file named `images.txt`, so these commands save the vCluster list as `vcluster-images.txt` and pass it to both scripts explicitly. The scripts default to `images.txt`, which at this point still holds the platform list from the previous step. Modify the following with your specific values to generate a copyable command: VCLUSTER\_VERSION0.37.5 REGISTRYecr.io/myteam Mirror the vCluster images and chart ``` export VCLUSTER_VERSION=0.37.5 export REGISTRY=ecr.io/myteam wget -O vcluster-images.txt https://github.com/loft-sh/vcluster/releases/download/v${VCLUSTER_VERSION}/images.txt ./download-images.sh --image-list vcluster-images.txt --images vcluster-images.tar.gz ./push-images.sh --registry ${REGISTRY} --images vcluster-images.tar.gz --image-list vcluster-images.txt helm pull vcluster --repo https://charts.loft.sh --version ${VCLUSTER_VERSION} helm push vcluster-${VCLUSTER_VERSION}.tgz oci://${REGISTRY}/charts ``` After mirroring, update the chart version on any tenant cluster templates that pin a vCluster version. For the optional image sets covering other Kubernetes distributions and versions, see [Pull and push images for vCluster](https://vcluster.com/docs/platform/install/air-gapped.md#pull-and-push-images-for-vcluster). ## Step 2 - Upgrade the platform deployment[​](#step-2---upgrade-the-platform-deployment "Direct link to Step 2 - Upgrade the platform deployment") Run `helm upgrade` against the chart version you just pushed, reusing your existing `vcluster-platform.yaml` values file. That file already points the platform and its agents at your private registry, so those settings carry over automatically. The relevant keys are `imageRef`, `env.DEFAULT_IMAGE_REGISTRY`, `env.DEFAULT_VCLUSTER_CHART_REPO`, and `agentValues.env`, as set up under [Configure and install vCluster Platform and agent](https://vcluster.com/docs/platform/install/air-gapped.md#configure-install-platform-agent). One value doesn't carry over. `imageRef.tag` pins the image to the version you installed, and it overrides the chart version. The command below sets it for this upgrade. Update it in `vcluster-platform.yaml` as well, so later upgrades don't revert to the old image. Modify the following with your specific values to generate a copyable command: PLATFORM\_VERSION4.12.0 REGISTRYecr.io/myteam PLATFORM\_NAMESPACEvcluster-platform Upgrade the platform using helm ``` export PLATFORM_VERSION=4.12.0 export REGISTRY=ecr.io/myteam export PLATFORM_NAMESPACE=vcluster-platform helm upgrade vcluster-platform oci://${REGISTRY}/charts/vcluster-platform:${PLATFORM_VERSION} \ --version ${PLATFORM_VERSION} \ --namespace ${PLATFORM_NAMESPACE} \ --values vcluster-platform.yaml \ --set imageRef.tag=${PLATFORM_VERSION} ``` If you use Helm 4 with `--server-side=true`, see the note on server-side apply conflicts in [Upgrade vCluster Platform](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade.md). warning vCluster Platform exclusively supports the default `secret` backend for storing Helm releases. Alternative `HELM_DRIVER` configurations (such as `configmap` or `sql`) are not supported. ## Step 3 - Verify the deployment[​](#step-3---verify-the-deployment "Direct link to Step 3 - Verify the deployment") Confirm the platform and agent deployments are running the target image from your private registry: Verify the deployed image ``` kubectl get deployment -n ${PLATFORM_NAMESPACE} loft \ -o jsonpath='{.spec.template.spec.containers[0].image}' ``` The platform automatically upgrades connected agents to the same version once the platform upgrade completes, pulling from the same private registry. --- # Upgrade from v3 to v4 Who needs this page This page only applies if you're upgrading vCluster Platform from a v3.x release to v4.0. If you're already running v4.0 or later, use the [regular upgrade instructions](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade.md) instead. Before attempting this major upgrade, upgrade to the latest v3.x version first. Review the [older upgrade docs](https://loft.sh/docs/admin/upgrade-loft) on how to upgrade. Upgrade using the CLI and not the UI, since there are known issues upgrading versions through the UI. ## Pre-upgrade recommendations[​](#pre-upgrade-recommendations "Direct link to Pre-upgrade recommendations") 1. Read through the v4.0 release notes. Review the [release notes for v4.0](https://github.com/loft-sh/loft-enterprise/releases/tag/v4.0.0) to read the updates in the release and understand any behavior changes. 2. vCluster versions are required on tenant clusters and tenant cluster templates. One of the breaking changes in v4.0 is the requirement of a vCluster version defined on your tenant cluster. Previously, if you deployed a tenant cluster without editing the version field, the version wasn't set in the `spec` of the resource. With each upgrade of the platform, these tenant clusters were automatically upgraded to the latest default vCluster version. Due to the [breaking changes introduced in v0.20](https://roadmap.vcluster.com/changelog/vcluster-v020-ga), automatic upgrades of the vCluster version are no longer supported. You must set a vCluster version in the spec of each tenant cluster and tenant cluster template. Confirm that the resource YAML includes the versions of your tenant clusters and tenant cluster templates before upgrading. Otherwise, your tenant clusters might encounter errors that require you to define these versions after the upgrade. The platform can't detect which version you upgraded from, so it can't set the correct version automatically. You must determine and set it manually. 3. Define the vCluster version on existing tenant clusters. In the UI, edit each tenant cluster or tenant cluster template and type the version in the version field. The resource YAML automatically updates the spec to include it. If the resource YAML already has `spec.template.helmRelease.chart.version` set, the version is defined. Example of spec ``` spec: template: helmRelease: chart: version: 0.19.7 ``` ## Download the latest vCluster CLI[​](#download-the-latest-vcluster-cli "Direct link to Download the latest vCluster CLI") With the deprecation of the Loft CLI, review the [Loft CLI to vCluster CLI Migration docs](https://vcluster.com/docs/vcluster/reference/migrations/loft-cli-vcluster-cli-migration). ## Required changes in your platform configuration before upgrading[​](#required-changes-in-your-platform-configuration-before-upgrading "Direct link to Required changes in your platform configuration before upgrading") v4.0 adds the ability to change the prefix of the namespaces created for projects. In previous versions the prefix was `loft-p-`, and in v4.0.0 the new default is `p-`. Because the default changed, you must set the previous prefix in the platform configuration as part of the upgrade command. Future upgrades reuse the configuration values, so you only need to set it once. ### Append the project prefix to upgrade commands[​](#append-the-project-prefix-to-upgrade-commands "Direct link to Append the project prefix to upgrade commands") * CLI * Helm To upgrade from v3 to v4 using the vCluster CLI, run the following. The command pins v4.0.0, the target of this upgrade: Platform Namespace The following steps assume the existing installation is in the default namespace `loft`. The default namespace for v4 is vcluster-platform, so we specify the namespace in the steps. ``` PLATFORM_NAMESPACE="loft" # Update if the platform is installed to a different namespace PLATFORM_VERSION="4.0.0" vcluster platform start --upgrade --namespace $PLATFORM_NAMESPACE --version=$PLATFORM_VERSION --values=<(cat < Config` UI or through editing the `loft-manager-config` secret found in the namespace vCluster Platform is installed in. Editing this config causes loft to restart, which should be unnecessary for adding OIDC platform clients. Therefore, managing vCluster Platform OIDC clients has been moved to its own UI, and the clients their own objects. You can manage these OIDC clients through the OIDC Provider tab of the admin page or through Kubernetes Secrets. See [Adding OIDC Clients to vCluster Platform OIDC Using Secrets](https://vcluster.com/docs/platform/administer/authentication/oidc-provider.md#adding-oidc-clients-to-vcluster-platform-oidc-using-secrets). The `oidc.clients` field is deprecated in vCluster Platform version 4.0 and is removed in version 5.0. ### Migrate OIDC clients from admin config to new OIDC clients[​](#migrate-oidc-clients-from-admin-config-to-new-oidc-clients "Direct link to Migrate OIDC clients from admin config to new OIDC clients") 1. Navigate to `Admin > OIDC Provider`. 2. Select `Add Client`. 3. In a separate tab navigate to `Admin > Config`. 4. Copy over the `name`, `clientID`, and `clientSecrets` fields from the first element of `oidc.clients` to the open editor on the `OIDC Provider` page. 5. Copy each value from the `oidc.clients.redirectURIs` list to the `Allowed redirect URIs` editor field on the OIDC Provider page. Separate each URI by a newline. 6. Select "Save". 7. Repeat steps 1-6 for all clients. 8. Copy the `oidc.clients` to a safe location. These can be discarded once all steps have been successful. 9. Delete the `oidc.clients` field. This causes vCluster Platform to restart. 10. Validate OIDC clients function normally. ## Troubleshoot[​](#troubleshoot "Direct link to Troubleshoot") ### Forgot the project prefix[​](#forgot-the-project-prefix "Direct link to Forgot the project prefix") If you accidentally performed the upgrade without setting the `projectNamespacePrefix`, then the pod of the platform is in a `CrashLoopBackoff` with a log similar to: ``` cmd/main.go:107 error executing root command {"component": "loft", "error": "init (4): set default project namespace prefix: seems like you have upgraded the platform from an earlier version that uses 'loft-p-' as a project namespace prefix. This has been changed to be 'p-' in the current version. Please set 'projectNamespacePrefix: loft-p-' in the platform config to get rid of this error"} ``` Follow the instructions on how to [append the project prefix](#append-the-project-prefix-to-upgrade-commands) and upgrade again. --- # Platform glossary This page contains definitions for common terms used throughout the platform documentation. ### Access Key A secure credential in the platform that allows programmatic access to platform resources and API endpoints. **Related terms:** The Platform, User ### Admission Control A Kubernetes feature that intercepts requests to the API server prior to persistence of the object, allowing for validation, mutation, or rejection of requests based on configured policies. **Related terms:** API Server, Control Plane ### API Server The core component of Kubernetes that exposes the Kubernetes API. It is the front-end for the Kubernetes control plane and handles all REST operations, validating and configuring data for API objects. **Related terms:** Control Plane, rate-limiting ### Auto Sleep A platform feature that allows clusters or namespaces to be paused when inactive, conserving resources and reducing costs. **Related terms:** The Platform ### Auto Sleep A platform feature that allows clusters or namespaces to be paused when inactive, conserving resources and reducing costs. **Related terms:** The Platform ### Certified Stack A StackTemplate distributed with vCluster Platform as part of a pinned, tested integration bundle. Certified Stack resources are view-only; copy one before customizing it. **Related terms:** StackTemplate, StackInstance ### Cluster template A predefined configuration in the platform for creating clusters with consistent settings, policies, and resources. **Related terms:** Template ### Compaction The process of removing superseded data from etcd to free up storage space. Regular compaction prevents etcd from running out of disk space. **Related terms:** etcd, Defragmentation ### Control Plane The container orchestration layer that exposes the API and interfaces to define, deploy, and manage the lifecycle of containers. In vCluster, each cluster has its own control plane components. **Related terms:** API Server, vCluster ### Defragmentation The process of releasing unused space in etcd's backend database after compaction. While compaction marks space as free, defragmentation reclaims it. **Related terms:** etcd, Compaction ### etcd A distributed key-value store that provides reliable storage for Kubernetes cluster data. In vCluster, etcd can be deployed externally or embedded within the vCluster pod. **Related terms:** Control Plane ### K0s A lightweight, certified Kubernetes distribution that can be used in clusters as an alternative to K3s or standard Kubernetes. **Related terms:** K3s, K8s ### K3s A lightweight, certified Kubernetes distribution often used as the default distribution for clusters due to its minimal resource requirements and fast startup time. **Related terms:** K8s, K0s ### K8s The standard Kubernetes distribution that can be used in clusters, offering full compatibility with upstream Kubernetes features. **Related terms:** K3s, K0s ### Namespace A Kubernetes namespace that can be managed by the platform, providing isolation for resources within a cluster. **Related terms:** Project ### Project A logical grouping of resources in the platform that provides team isolation and resource quotas. Projects help organize clusters and namespace resources. **Related terms:** The Platform, Team ### Stack task One node in a Stack dependency graph. A task creates either an AppInstance or an ArgoCDApplication and can wait for dependencies, capture outputs, and pass values to later tasks. **Related terms:** StackTemplate, StackInstance ### StackInstance A project-scoped platform resource that deploys an inline or referenced stack definition to one cluster or control plane cluster and reports aggregate and per-task status. **Related terms:** StackTemplate, Stack task, Cluster, Control plane cluster ### StackTemplate A cluster-scoped, reusable platform resource that defines parameters and a dependency graph of App or Argo CD application tasks. A StackInstance resolves and deploys the definition. **Related terms:** StackInstance, Stack task, Certified Stack, Template ### Team A group of users in the platform with shared access to specific projects and resources. **Related terms:** Project, User ### Template A reusable configuration pattern in the platform that defines the settings and resources for creating clusters, namespaces, applications, or stacks. **Related terms:** The Platform, Cluster template, StackTemplate ### The Platform The vCluster Platform that provides management, access control, and operational features for clusters across multiple control plane clusters. **Related terms:** Project ### User An individual account in the platform with specific access permissions to projects and resources. **Related terms:** Team --- # Platform annotations and labels reference This page documents the well-known annotations and labels in the `loft.sh` namespace used by vCluster Platform for managing clusters, projects, spaces, users, teams, and integrations. ## Cluster management[​](#cluster-management "Direct link to Cluster management") These annotations configure connected clusters in vCluster Platform. ### loft.sh/cluster-uid[​](#loft-sh-cluster-uid "Direct link to loft.sh/cluster-uid") **Type:** Annotation **Example:** `loft.sh/cluster-uid: "a1b2c3d4-e5f6-7890-abcd-ef1234567890"` **Used on:** Cluster **Set by:** Platform The unique identifier assigned to this cluster by vCluster Platform. Used internally for cluster identification and correlation. ### loft.sh/cluster-name[​](#loft-sh-cluster-name "Direct link to loft.sh/cluster-name") **Type:** Annotation **Example:** `loft.sh/cluster-name: "production-east"` **Used on:** NetworkPeer, Agent resources **Set by:** Platform Identifies the cluster name for network peer and agent resources. ### loft.sh/display-name[​](#loft-sh-display-name "Direct link to loft.sh/display-name") **Type:** Annotation **Example:** `loft.sh/display-name: "Production East US"` **Used on:** Cluster, Project, Team, User **Set by:** User-configurable A human-readable display name shown in the platform UI. Can be different from the resource's actual name. ### loft.sh/ingress-suffix[​](#loft-sh-ingress-suffix "Direct link to loft.sh/ingress-suffix") **Type:** Annotation **Example:** `loft.sh/ingress-suffix: "vclusters.example.com"` **Used on:** Cluster **Set by:** User-configurable Sets the domain suffix for vCluster ingress access points on this cluster. Required for external vCluster access. ### loft.sh/cluster-domain[​](#loft-sh-cluster-domain "Direct link to loft.sh/cluster-domain") **Type:** Annotation **Example:** `loft.sh/cluster-domain: "cluster.local"` **Used on:** Cluster **Set by:** User-configurable Specifies the cluster's internal DNS domain. Defaults to `cluster.local`. ### loft.sh/cluster-domain-target[​](#loft-sh-cluster-domain-target "Direct link to loft.sh/cluster-domain-target") **Type:** Annotation **Example:** `loft.sh/cluster-domain-target: "192.168.1.100"` **Used on:** Cluster **Set by:** User-configurable Specifies the target address for cluster domain resolution. ### loft.sh/direct-cluster-endpoint[​](#loft-sh-direct-cluster-endpoint "Direct link to loft.sh/direct-cluster-endpoint") **Type:** Annotation **Example:** `loft.sh/direct-cluster-endpoint: "https://cluster.example.com:6443"` **Used on:** Cluster **Set by:** User-configurable Specifies a direct endpoint for the cluster, enabling clients to connect directly instead of routing through the platform. ### loft.sh/direct-cluster-endpoint-insecure[​](#loft-sh-direct-cluster-endpoint-insecure "Direct link to loft.sh/direct-cluster-endpoint-insecure") **Type:** Annotation **Example:** `loft.sh/direct-cluster-endpoint-insecure: "true"` **Used on:** Cluster **Set by:** User-configurable When `true`, allows insecure TLS connections to the regional cluster endpoint. ### loft.sh/derp-endpoint[​](#loft-sh-derp-endpoint "Direct link to loft.sh/derp-endpoint") **Type:** Annotation **Example:** `loft.sh/derp-endpoint: "derp.example.com"` **Used on:** Cluster **Set by:** User-configurable Specifies a publicly accessible DERP relay endpoint for this cluster. ### loft.sh/derp-endpoint-insecure[​](#loft-sh-derp-endpoint-insecure "Direct link to loft.sh/derp-endpoint-insecure") **Type:** Annotation **Example:** `loft.sh/derp-endpoint-insecure: "true"` **Used on:** Cluster **Set by:** User-configurable When `true`, allows insecure connections to the DERP relay endpoint. ### loft.sh/streaming-connection-idle-timeout[​](#loft-sh-streaming-connection-idle-timeout "Direct link to loft.sh/streaming-connection-idle-timeout") **Type:** Annotation **Example:** `loft.sh/streaming-connection-idle-timeout: "14400"` **Used on:** Cluster **Set by:** User-configurable Sets the idle timeout, in seconds, for streaming connections such as exec, port-forward, and logs to this cluster. The setting applies to new connections. ### loft.sh/cluster-access[​](#loft-sh-cluster-access "Direct link to loft.sh/cluster-access") **Type:** Annotation **Example:** `loft.sh/cluster-access: "direct"` **Used on:** Cluster **Set by:** Platform Indicates the access method configured for this cluster. ### loft.sh/skip-direct-connection[​](#loft-sh-skip-direct-connection "Direct link to loft.sh/skip-direct-connection") **Type:** Annotation **Example:** `loft.sh/skip-direct-connection: "true"` **Used on:** Cluster **Set by:** User-configurable When `true`, forces connections through the platform proxy even when direct connection is available. ### loft.sh/cluster-role-cluster[​](#loft-sh-cluster-role-cluster "Direct link to loft.sh/cluster-role-cluster") **Type:** Label **Example:** `loft.sh/cluster-role-cluster: "true"` **Used on:** ClusterRole **Set by:** Platform Marks a ClusterRole as applicable at the cluster level. ### loft.sh/cluster-role-management[​](#loft-sh-cluster-role-management "Direct link to loft.sh/cluster-role-management") **Type:** Label **Example:** `loft.sh/cluster-role-management: "true"` **Used on:** ClusterRole **Set by:** Platform Marks a ClusterRole as a management role for the platform. ### loft.sh/account-cluster-role[​](#loft-sh-account-cluster-role "Direct link to loft.sh/account-cluster-role") **Type:** Label **Example:** `loft.sh/account-cluster-role: "true"` **Used on:** ClusterRole **Set by:** Platform Marks a ClusterRole as available for account-level assignment. ### loft.sh/space-cluster-role[​](#loft-sh-space-cluster-role "Direct link to loft.sh/space-cluster-role") **Type:** Label **Example:** `loft.sh/space-cluster-role: "true"` **Used on:** ClusterRole **Set by:** Platform Marks a ClusterRole as available for space-level assignment. ### loft.sh/cluster-account-template[​](#loft-sh-cluster-account-template "Direct link to loft.sh/cluster-account-template") **Type:** Label **Example:** `loft.sh/cluster-account-template: "default-template"` **Used on:** ClusterAccountTemplate **Set by:** Platform Identifies the cluster account template. ### loft.sh/account-templates-ignore-clusters[​](#loft-sh-account-templates-ignore-clusters "Direct link to loft.sh/account-templates-ignore-clusters") **Type:** Annotation **Example:** `loft.sh/account-templates-ignore-clusters: "cluster1,cluster2"` **Used on:** User, Team **Set by:** User-configurable Comma-separated list of clusters where account templates should not be applied for this user or team. ### loft.sh/agent-values[​](#loft-sh-agent-values "Direct link to loft.sh/agent-values") **Type:** Annotation **Example:** `loft.sh/agent-values: '{"resources":{"limits":{"memory":"512Mi"}}}'` **Used on:** Cluster **Set by:** User-configurable Extra Helm values that should be applied when deploying the platform agent to this cluster. ### loft.sh/cluster-ignore-agent[​](#loft-sh-cluster-ignore-agent "Direct link to loft.sh/cluster-ignore-agent") **Type:** Annotation **Example:** `loft.sh/cluster-ignore-agent: "true"` **Used on:** Cluster **Set by:** User-configurable When `true`, the platform will not deploy or manage an agent on this cluster. ### loft.sh/cluster-ignore-kiosk[​](#loft-sh-cluster-ignore-kiosk "Direct link to loft.sh/cluster-ignore-kiosk") **Type:** Annotation **Example:** `loft.sh/cluster-ignore-kiosk: "true"` **Used on:** Cluster **Set by:** User-configurable When `true`, the platform will not deploy or manage kiosk on this cluster. ### loft.sh/direct-cluster-endpoint-ca-data[​](#loft-sh-direct-cluster-endpoint-ca-data "Direct link to loft.sh/direct-cluster-endpoint-ca-data") **Type:** Annotation **Example:** `loft.sh/direct-cluster-endpoint-ca-data: "LS0tLS1CRUdJTi..."` **Used on:** Cluster **Set by:** User-configurable Base64-encoded certificate authority data for verifying the regional cluster endpoint certificate. ## Project management[​](#project-management "Direct link to Project management") These labels and annotations are used on project resources and project-owned namespaces. ### loft.sh/project[​](#loft-sh-project "Direct link to loft.sh/project") **Type:** Label **Example:** `loft.sh/project: "team-alpha"` **Used on:** Namespace, VirtualClusterInstance, SpaceInstance **Set by:** Platform Identifies the vCluster Platform project that owns this resource. ### loft.sh/project-namespace[​](#loft-sh-project-namespace "Direct link to loft.sh/project-namespace") **Type:** Annotation **Example:** `loft.sh/project-namespace: "p-team-alpha"` **Used on:** Various resources **Set by:** Platform The namespace where project resources are stored. ### loft.sh/project-role[​](#loft-sh-project-role "Direct link to loft.sh/project-role") **Type:** Label **Example:** `loft.sh/project-role: "true"` **Used on:** ClusterRole **Set by:** User-configurable Marks a ClusterRole as available for use as a project role. Required for ClusterRoles to appear in project member role selection. ### loft.sh/project-cluster-quota[​](#loft-sh-project-cluster-quota "Direct link to loft.sh/project-cluster-quota") **Type:** Label **Example:** `loft.sh/project-cluster-quota: "team-alpha-quota"` **Used on:** ResourceQuota **Set by:** Platform Links a ResourceQuota to a project's cluster quota. ### loft.sh/project-user-cluster-quota[​](#loft-sh-project-user-cluster-quota "Direct link to loft.sh/project-user-cluster-quota") **Type:** Label **Example:** `loft.sh/project-user-cluster-quota: "user-quota"` **Used on:** ResourceQuota **Set by:** Platform Links a ResourceQuota to a per-user quota within a project. ## Space management[​](#space-management "Direct link to Space management") These annotations and labels are used on spaces (namespaces) managed by the platform. ### loft.sh/space-instance-name[​](#loft-sh-space-instance-name "Direct link to loft.sh/space-instance-name") **Type:** Label **Example:** `loft.sh/space-instance-name: "dev-space"` **Used on:** Namespace **Set by:** Platform The name of the SpaceInstance that created this namespace. ### loft.sh/space-instance-namespace[​](#loft-sh-space-instance-namespace "Direct link to loft.sh/space-instance-namespace") **Type:** Label **Example:** `loft.sh/space-instance-namespace: "p-default"` **Used on:** Namespace **Set by:** Platform The namespace containing the SpaceInstance resource. ### loft.sh/space-instance-project[​](#loft-sh-space-instance-project "Direct link to loft.sh/space-instance-project") **Type:** Label **Example:** `loft.sh/space-instance-project: "default"` **Used on:** Namespace **Set by:** Platform The project that owns the SpaceInstance. ### loft.sh/owned[​](#loft-sh-owned "Direct link to loft.sh/owned") **Type:** Label **Example:** `loft.sh/owned: "true"` **Used on:** Namespace **Set by:** Platform Indicates that this namespace is owned by a specific user or team. ### loft.sh/space-constraints[​](#loft-sh-space-constraints "Direct link to loft.sh/space-constraints") **Type:** Label **Example:** `loft.sh/space-constraints: "restricted"` **Used on:** Namespace **Set by:** Platform Identifies the space constraints applied to this namespace. ### loft.sh/space-constraints-status[​](#loft-sh-space-constraints-status "Direct link to loft.sh/space-constraints-status") **Type:** Annotation **Example:** `loft.sh/space-constraints-status: "applied"` **Used on:** Namespace **Set by:** Platform Status of space constraints application. ### loft.sh/space-objects[​](#loft-sh-space-objects "Direct link to loft.sh/space-objects") **Type:** Annotation **Example:** `loft.sh/space-objects: '{"configmaps":["config1"]}'` **Used on:** Namespace **Set by:** Platform JSON object tracking space template objects created in this namespace. ### loft.sh/space-objects-status[​](#loft-sh-space-objects-status "Direct link to loft.sh/space-objects-status") **Type:** Annotation **Example:** `loft.sh/space-objects-status: "synced"` **Used on:** Namespace **Set by:** Platform Status of space objects synchronization. ### loft.sh/disable-space-creation[​](#loft-sh-disable-space-creation "Direct link to loft.sh/disable-space-creation") **Type:** Annotation **Example:** `loft.sh/disable-space-creation: "true"` **Used on:** Cluster **Set by:** User-configurable When `true`, disables direct space creation on this cluster. Spaces must be created through projects. ## vCluster instance management[​](#vcluster-instance-management "Direct link to vCluster instance management") These labels and annotations are used on vCluster instances managed by the platform. ### loft.sh/vcluster-instance-name[​](#loft-sh-vcluster-instance-name "Direct link to loft.sh/vcluster-instance-name") **Type:** Label **Example:** `loft.sh/vcluster-instance-name: "dev-vcluster"` **Used on:** Namespace, Pod **Set by:** Platform The name of the VirtualClusterInstance that created this vCluster. ### loft.sh/vcluster-instance-namespace[​](#loft-sh-vcluster-instance-namespace "Direct link to loft.sh/vcluster-instance-namespace") **Type:** Label **Example:** `loft.sh/vcluster-instance-namespace: "p-default"` **Used on:** Namespace, Pod **Set by:** Platform The namespace containing the VirtualClusterInstance resource. ### loft.sh/vcluster-instance-project[​](#loft-sh-vcluster-instance-project "Direct link to loft.sh/vcluster-instance-project") **Type:** Label **Example:** `loft.sh/vcluster-instance-project: "default"` **Used on:** Namespace, Pod **Set by:** Platform The project that owns the VirtualClusterInstance. ### platform.vcluster.com/vcluster-instance-name[​](#platform-vcluster-com-vcluster-instance-name "Direct link to platform.vcluster.com/vcluster-instance-name") **Type:** Label **Example:** `platform.vcluster.com/vcluster-instance-name: "dev-vcluster"` **Used on:** Resources associated with vCluster instances **Set by:** Platform The name of the tenant cluster an object is associated with. ### platform.vcluster.com/vcluster-instance-namespace[​](#platform-vcluster-com-vcluster-instance-namespace "Direct link to platform.vcluster.com/vcluster-instance-namespace") **Type:** Label **Example:** `platform.vcluster.com/vcluster-instance-namespace: "p-default"` **Used on:** Resources associated with vCluster instances **Set by:** Platform The namespace of the tenant cluster an object is associated with. ### vcluster.loft.sh/managed-by[​](#vcluster-loft-sh-managed-by "Direct link to vcluster.loft.sh/managed-by") **Type:** Label **Example:** `vcluster.loft.sh/managed-by: "loft"` **Used on:** vCluster resources **Set by:** Platform Indicates that this vCluster is managed by vCluster Platform. ### vcluster.loft.sh/vcluster-name[​](#vcluster-loft-sh-vcluster-name "Direct link to vcluster.loft.sh/vcluster-name") **Type:** Label **Example:** `vcluster.loft.sh/vcluster-name: "my-vcluster"` **Used on:** vCluster pods and resources **Set by:** Platform The name of the vCluster. ### vcluster.loft.sh/vcluster-namespace[​](#vcluster-loft-sh-vcluster-namespace "Direct link to vcluster.loft.sh/vcluster-namespace") **Type:** Label **Example:** `vcluster.loft.sh/vcluster-namespace: "vcluster-my-vcluster"` **Used on:** vCluster pods and resources **Set by:** Platform The namespace where the vCluster is deployed. ### vcluster.loft.sh/fake-node[​](#vcluster-loft-sh-fake-node "Direct link to vcluster.loft.sh/fake-node") **Type:** Label **Example:** `vcluster.loft.sh/fake-node: "true"` **Used on:** Node **Set by:** Platform Identifies nodes that are virtual/fake nodes created by vCluster. ### vcluster.loft.sh/dynamic-node-pool[​](#vcluster-loft-sh-dynamic-node-pool "Direct link to vcluster.loft.sh/dynamic-node-pool") **Type:** Label **Example:** `vcluster.loft.sh/dynamic-node-pool: "default-pool"` **Used on:** Node **Set by:** Platform Identifies the dynamic node pool this node belongs to. ### vcluster.loft.sh/control-plane-endpoint[​](#vcluster-loft-sh-control-plane-endpoint "Direct link to vcluster.loft.sh/control-plane-endpoint") **Type:** Annotation **Example:** `vcluster.loft.sh/control-plane-endpoint: "https://vcluster.example.com:443"` **Used on:** VirtualClusterInstance **Set by:** Platform The control plane endpoint for accessing this vCluster. ### vcluster.loft.sh/object-imported[​](#vcluster-loft-sh-object-imported "Direct link to vcluster.loft.sh/object-imported") **Type:** Annotation **Example:** `vcluster.loft.sh/object-imported: "true"` **Used on:** Various resources **Set by:** Platform Indicates that this resource was imported into a vCluster. ### loft.sh/hpm-enabled[​](#loft-sh-hpm-enabled "Direct link to loft.sh/hpm-enabled") **Type:** Annotation **Example:** `loft.sh/hpm-enabled: "true"` **Used on:** VirtualClusterInstance **Set by:** User-configurable Enables the Host Path Mapper for this vCluster instance. ### loft.sh/skip-helm-deploy[​](#loft-sh-skip-helm-deploy "Direct link to loft.sh/skip-helm-deploy") **Type:** Annotation **Example:** `loft.sh/skip-helm-deploy: "true"` **Used on:** VirtualClusterInstance **Set by:** User-configurable Skips Helm deployment for this vCluster. Use when managing vCluster deployment externally. ### loft.sh/database-vcluster[​](#loft-sh-database-vcluster "Direct link to loft.sh/database-vcluster") **Type:** Label **Example:** `loft.sh/database-vcluster: "my-vcluster"` **Used on:** Secret **Set by:** Platform Links a database secret to a specific vCluster. ### virtualcluster.loft.sh/latest-version[​](#virtualcluster-loft-sh-latest-version "Direct link to virtualcluster.loft.sh/latest-version") **Type:** Annotation **Example:** `virtualcluster.loft.sh/latest-version: "0.20.0"` **Used on:** VirtualClusterInstance **Set by:** Platform Stores the latest available vCluster version for upgrade notifications. ### vcluster.loft.sh/kubernetes-name[​](#vcluster-loft-sh-kubernetes-name "Direct link to vcluster.loft.sh/kubernetes-name") **Type:** Annotation **Example:** `vcluster.loft.sh/kubernetes-name: "my-vcluster"` **Used on:** Node **Set by:** Platform Identifies the Kubernetes name associated with the vCluster node. ### vcluster.loft.sh/object-namespace[​](#vcluster-loft-sh-object-namespace "Direct link to vcluster.loft.sh/object-namespace") **Type:** Annotation **Example:** `vcluster.loft.sh/object-namespace: "default"` **Used on:** Various synced resources **Set by:** Platform Indicates the original namespace of an object synced from the vCluster to the control plane cluster. ### vcluster.loft.sh/preserve-host-rule[​](#vcluster-loft-sh-preserve-host-rule "Direct link to vcluster.loft.sh/preserve-host-rule") **Type:** Annotation **Example:** `vcluster.loft.sh/preserve-host-rule: "loft-sleep-wakeup-rule"` **Used on:** HTTPRoute **Set by:** Platform Set on a control plane cluster HTTPRoute to tell vCluster's HTTPRoute [syncer](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync) to preserve a single named host-side HTTPRouteRule verbatim during tenant-to-host reconciliation. The platform sleep agent uses this to keep a managed wake-override rule alive while a tenant namespace sleeps. ### vcluster.loft.sh/preserve-request-mirror-filters[​](#vcluster-loft-sh-preserve-request-mirror-filters "Direct link to vcluster.loft.sh/preserve-request-mirror-filters") **Type:** Annotation **Example:** `vcluster.loft.sh/preserve-request-mirror-filters: "true"` **Used on:** HTTPRoute **Set by:** Platform Tells vCluster to preserve request mirror filters on this HTTPRoute during synchronization from tenant cluster to control plane cluster. ### vcluster.loft.sh/default-node-profile[​](#vcluster-loft-sh-default-node-profile "Direct link to vcluster.loft.sh/default-node-profile") **Type:** Annotation **Example:** `vcluster.loft.sh/default-node-profile: "gpu-enabled"` **Used on:** VirtualClusterInstance **Set by:** Platform Mirrors the owning project's default node profile onto the VirtualClusterInstance. The value is the node profile name. This annotation is absent when the project has no default profile configured. ### vcluster.loft.sh/node-claim-name[​](#vcluster-loft-sh-node-claim-name "Direct link to vcluster.loft.sh/node-claim-name") **Type:** Label **Example:** `vcluster.loft.sh/node-claim-name: "my-node-claim"` **Used on:** Secret (bootstrap-token) **Set by:** Platform Identifies the NodeClaim name associated with a bootstrap-token Secret used for node registration. ### vcluster.loft.sh/node-claim-namespace[​](#vcluster-loft-sh-node-claim-namespace "Direct link to vcluster.loft.sh/node-claim-namespace") **Type:** Label **Example:** `vcluster.loft.sh/node-claim-namespace: "default"` **Used on:** Secret (bootstrap-token) **Set by:** Platform Identifies the NodeClaim namespace associated with a bootstrap-token Secret used for node registration. ## Snapshot and restore[​](#snapshot-restore "Direct link to Snapshot and restore") These labels mark resources used for vCluster backup and restore operations. ### vcluster.loft.sh/snapshot-request[​](#vcluster-loft-sh-snapshot-request "Direct link to vcluster.loft.sh/snapshot-request") **Type:** Label **Example:** `vcluster.loft.sh/snapshot-request: "true"` **Used on:** ConfigMap, Secret **Set by:** Platform Marks ConfigMaps and Secrets as snapshot request resources for vCluster backup operations. ### vcluster.loft.sh/restore-request[​](#vcluster-loft-sh-restore-request "Direct link to vcluster.loft.sh/restore-request") **Type:** Label **Example:** `vcluster.loft.sh/restore-request: "true"` **Used on:** ConfigMap, Secret **Set by:** Platform Marks ConfigMaps and Secrets as restore request resources for vCluster restore operations. ## Auto sleep configuration[​](#sleep-mode-configuration "Direct link to Auto sleep configuration") These annotations configure auto sleep behavior. ### loft.sh/sleep-mode[​](#loft-sh-sleep-mode "Direct link to loft.sh/sleep-mode") **Type:** Label **Example:** `loft.sh/sleep-mode: "true"` **Used on:** Deployment, StatefulSet, ReplicaSet **Set by:** Platform Marks a workload that the platform scaled to zero for sleep. The platform uses this label to find the workload when it wakes the namespace. ### loft.sh/sleep-mode-replicas[​](#loft-sh-sleep-mode-replicas "Direct link to loft.sh/sleep-mode-replicas") **Type:** Annotation **Example:** `loft.sh/sleep-mode-replicas: "3"` **Used on:** Deployment, StatefulSet, ReplicaSet **Set by:** Platform Stores the original replica count before auto sleep scaled down the workload. ## Auto sleep annotations (sleepmode.loft.sh)[​](#sleepmode-annotations "Direct link to Auto sleep annotations (sleepmode.loft.sh)") These annotations in the `sleepmode.loft.sh` namespace control auto sleep behavior for namespaces and tenant clusters. See each annotation's **Used on** field for exactly where to set it. ### sleepmode.loft.sh/sleep-after[​](#sleepmode-loft-sh-sleep-after "Direct link to sleepmode.loft.sh/sleep-after") **Type:** Annotation **Example:** `sleepmode.loft.sh/sleep-after: "3600"` **Used on:** Namespace, VirtualClusterTemplate or inline VirtualClusterInstance under `spec.template.metadata.annotations` **Set by:** User-configurable Specifies the number of seconds of inactivity after which the namespace or tenant cluster should automatically sleep. ### sleepmode.loft.sh/delete-after[​](#sleepmode-loft-sh-delete-after "Direct link to sleepmode.loft.sh/delete-after") **Type:** Annotation **Example:** `sleepmode.loft.sh/delete-after: "86400"` **Used on:** Namespace, VirtualClusterTemplate or inline VirtualClusterInstance under `spec.template.metadata.annotations` **Set by:** User-configurable Specifies the number of seconds of inactivity after which the namespace or tenant cluster should be automatically deleted. ### sleepmode.loft.sh/sleep-schedule[​](#sleepmode-loft-sh-sleep-schedule "Direct link to sleepmode.loft.sh/sleep-schedule") **Type:** Annotation **Example:** `sleepmode.loft.sh/sleep-schedule: "0 20 * * *"` **Used on:** Namespace, VirtualClusterTemplate or inline VirtualClusterInstance under `spec.template.metadata.annotations` **Set by:** User-configurable Specifies a cron schedule for when the namespace or tenant cluster should automatically sleep. Uses standard cron format. ### sleepmode.loft.sh/wakeup-schedule[​](#sleepmode-loft-sh-wakeup-schedule "Direct link to sleepmode.loft.sh/wakeup-schedule") **Type:** Annotation **Example:** `sleepmode.loft.sh/wakeup-schedule: "0 8 * * 1-5"` **Used on:** Namespace, VirtualClusterTemplate or inline VirtualClusterInstance under `spec.template.metadata.annotations` **Set by:** User-configurable Specifies a cron schedule for when the namespace or tenant cluster should automatically wake up. ### sleepmode.loft.sh/timezone[​](#sleepmode-loft-sh-timezone "Direct link to sleepmode.loft.sh/timezone") **Type:** Annotation **Example:** `sleepmode.loft.sh/timezone: "America/New_York"` **Used on:** Namespace, VirtualClusterTemplate or inline VirtualClusterInstance under `spec.template.metadata.annotations` **Set by:** User-configurable Specifies the timezone for scheduled sleep and wakeup operations. Accepts IANA timezone names. Defaults to UTC. ### sleepmode.loft.sh/force[​](#sleepmode-loft-sh-force "Direct link to sleepmode.loft.sh/force") **Type:** Annotation **Example:** `sleepmode.loft.sh/force: "true"` **Used on:** Namespace, VirtualClusterInstance **Set by:** User-configurable Forces the namespace or tenant cluster to sleep immediately, regardless of activity. ### sleepmode.loft.sh/force-duration[​](#sleepmode-loft-sh-force-duration "Direct link to sleepmode.loft.sh/force-duration") **Type:** Annotation **Example:** `sleepmode.loft.sh/force-duration: "3600"` **Used on:** Namespace, VirtualClusterInstance **Set by:** User-configurable Forces sleep for a specific duration in seconds. After this period, normal activity tracking resumes. Set to `0` for indefinite sleep until manually woken. ### sleepmode.loft.sh/scope[​](#sleepmode-loft-sh-scope "Direct link to sleepmode.loft.sh/scope") **Type:** Annotation **Example:** `sleepmode.loft.sh/scope: "workloads-only"` **Used on:** VirtualClusterInstance **Set by:** User-configurable For vCluster 0.33.0 or later, keeps the tenant cluster control plane running and lets the tenant cluster manage workload sleep. The platform agent doesn't take over auto sleep for this instance. ### sleepmode.loft.sh/exclude[​](#sleepmode-loft-sh-exclude "Direct link to sleepmode.loft.sh/exclude") **Type:** Annotation **Example:** `sleepmode.loft.sh/exclude: "true"` **Used on:** Deployment, StatefulSet, ReplicaSet, Pod **Set by:** User-configurable Excludes this workload from auto sleep. When the namespace sleeps, its existing Pods continue running. The sleep quota prevents Kubernetes from creating a replacement if an excluded Pod stops while the namespace is asleep. ### sleepmode.loft.sh/ignore-all[​](#sleepmode-loft-sh-ignore-all "Direct link to sleepmode.loft.sh/ignore-all") **Type:** Annotation **Example:** `sleepmode.loft.sh/ignore-all: "true"` **Used on:** Namespace, VirtualClusterTemplate or inline VirtualClusterInstance under `spec.template.metadata.annotations` **Set by:** User-configurable Ignores all activity when determining whether the namespace or tenant cluster should sleep. ### sleepmode.loft.sh/ignore-ingresses[​](#sleepmode-loft-sh-ignore-ingresses "Direct link to sleepmode.loft.sh/ignore-ingresses") **Type:** Annotation **Example:** `sleepmode.loft.sh/ignore-ingresses: "true"` **Used on:** Namespace, VirtualClusterTemplate or inline VirtualClusterInstance under `spec.template.metadata.annotations` **Set by:** User-configurable Ignores supported Ingress, HTTPRoute, and Istio traffic when determining activity. Useful when automated web requests should not prevent sleep. ### sleepmode.loft.sh/ignore-groups[​](#sleepmode-loft-sh-ignore-groups "Direct link to sleepmode.loft.sh/ignore-groups") **Type:** Annotation **Example:** `sleepmode.loft.sh/ignore-groups: "apps,batch"` **Used on:** Namespace **Set by:** User-configurable Ignores requests from users in specific groups when determining activity. Provide a comma-separated list of user group names. This annotation has no effect on tenant clusters. Use it for namespace sleep mode only. ### sleepmode.loft.sh/ignore-vclusters[​](#sleepmode-loft-sh-ignore-vclusters "Direct link to sleepmode.loft.sh/ignore-vclusters") **Type:** Annotation **Example:** `sleepmode.loft.sh/ignore-vclusters: "true"` **Used on:** Namespace, VirtualClusterTemplate or inline VirtualClusterInstance under `spec.template.metadata.annotations` **Set by:** User-configurable Ignores tenant cluster requests when determining namespace activity. ### sleepmode.loft.sh/ignore-resources[​](#sleepmode-loft-sh-ignore-resources "Direct link to sleepmode.loft.sh/ignore-resources") **Type:** Annotation **Example:** `sleepmode.loft.sh/ignore-resources: "pods,configmaps"` **Used on:** Namespace **Set by:** User-configurable Ignores requests to specific Kubernetes resources when determining activity. Provide a comma-separated list of `resource` or `resource.group` values. This annotation has no effect on tenant clusters. Use it for namespace sleep mode only. ### sleepmode.loft.sh/ignore-verbs[​](#sleepmode-loft-sh-ignore-verbs "Direct link to sleepmode.loft.sh/ignore-verbs") **Type:** Annotation **Example:** `sleepmode.loft.sh/ignore-verbs: "get,list,watch"` **Used on:** Namespace **Set by:** User-configurable Ignores requests with specific Kubernetes API verbs when determining activity. Provide a comma-separated list. This annotation has no effect on tenant clusters. Use it for namespace sleep mode only. ### sleepmode.loft.sh/ignore-resource-verbs[​](#sleepmode-loft-sh-ignore-resource-verbs "Direct link to sleepmode.loft.sh/ignore-resource-verbs") **Type:** Annotation **Example:** `sleepmode.loft.sh/ignore-resource-verbs: "pods=get list,deployments.apps=get"` **Used on:** Namespace **Set by:** User-configurable Ignores specific Kubernetes API verb combinations for specific resources. Format: `resource[.group]=verb1 verb2,resource2[.group]=verb3`. This annotation has no effect on tenant clusters. Use it for namespace sleep mode only. ### sleepmode.loft.sh/ignore-resource-names[​](#sleepmode-loft-sh-ignore-resource-names "Direct link to sleepmode.loft.sh/ignore-resource-names") **Type:** Annotation **Example:** `sleepmode.loft.sh/ignore-resource-names: "pods=monitoring-pod,configmaps=config1"` **Used on:** Namespace **Set by:** User-configurable Ignores requests to specific named resources. Format: `resource[.group]=name1 name2`. This annotation has no effect on tenant clusters. Use it for namespace sleep mode only. ### sleepmode.loft.sh/ignore-active-connections[​](#sleepmode-loft-sh-ignore-active-connections "Direct link to sleepmode.loft.sh/ignore-active-connections") **Type:** Annotation **Example:** `sleepmode.loft.sh/ignore-active-connections: "true"` **Used on:** Namespace, VirtualClusterTemplate or inline VirtualClusterInstance under `spec.template.metadata.annotations` **Set by:** User-configurable Ignores active connections (such as `kubectl exec` or `kubectl port-forward`) when determining whether to sleep. Allows sleep even with open connections. ### sleepmode.loft.sh/ignore-user-agents[​](#sleepmode-loft-sh-ignore-user-agents "Direct link to sleepmode.loft.sh/ignore-user-agents") **Type:** Annotation **Example:** `sleepmode.loft.sh/ignore-user-agents: "kube-probe/*,prometheus/*"` **Used on:** Namespace, VirtualClusterTemplate or inline VirtualClusterInstance under `spec.template.metadata.annotations` **Set by:** User-configurable Ignores requests from specific user agents. Supports trailing wildcards. Comma-separated list. ### sleepmode.loft.sh/ignore-activity[​](#sleepmode-loft-sh-ignore-activity "Direct link to sleepmode.loft.sh/ignore-activity") **Type:** Label **Example:** `sleepmode.loft.sh/ignore-activity: "true"` **Used on:** AccessKey **Set by:** Platform Marks an access key whose API requests should not count as activity for sleep mode purposes. The ArgoCD integration sets this on its access keys so ArgoCD reconcile traffic does not keep a tenant cluster awake. ### sleepmode.loft.sh/disable-ingress-wakeup[​](#sleepmode-loft-sh-disable-ingress-wakeup "Direct link to sleepmode.loft.sh/disable-ingress-wakeup") **Type:** Annotation **Example:** `sleepmode.loft.sh/disable-ingress-wakeup: "true"` **Used on:** Namespace, VirtualClusterInstance **Set by:** User-configurable Disables automatic wakeup from supported Ingress, HTTPRoute, and Istio traffic. When set, the namespace or tenant cluster remains asleep when it receives that traffic. ### sleepmode.loft.sh/disable-metrics-tracking[​](#sleepmode-loft-sh-disable-metrics-tracking "Direct link to sleepmode.loft.sh/disable-metrics-tracking") **Type:** Annotation **Example:** `sleepmode.loft.sh/disable-metrics-tracking: "true"` **Used on:** Cluster **Set by:** User-configurable Disables Istio metrics-based activity tracking for the connected cluster. Other configured activity sources, such as proxied Kubernetes API requests and HTTP request mirrors, can still update activity. ### sleepmode.loft.sh/request-mirror-controller-allowlist[​](#sleepmode-loft-sh-request-mirror-controller-allowlist "Direct link to sleepmode.loft.sh/request-mirror-controller-allowlist") **Type:** Annotation **Example:** `sleepmode.loft.sh/request-mirror-controller-allowlist: "example.com/controller"` **Used on:** Cluster **Set by:** User-configurable Comma-separated list of `GatewayClass.spec.controllerName` values to treat as request-mirror-supporting for HTTPRoutes on this connected cluster. Use it when a compatible controller doesn't advertise the `HTTPRouteRequestMirror` feature in `GatewayClass.status.supportedFeatures`. ## Auto sleep status annotations[​](#sleep-mode-status-annotations "Direct link to Auto sleep status annotations") These annotations are set by the platform to indicate auto sleep status. They are read-only. ### sleepmode.loft.sh/last-activity[​](#sleepmode-loft-sh-last-activity "Direct link to sleepmode.loft.sh/last-activity") **Type:** Annotation (read-only) **Example:** `sleepmode.loft.sh/last-activity: "1706745600"` **Used on:** Namespace, VirtualClusterInstance **Set by:** Platform Unix timestamp of the last detected activity. Set automatically by the platform. ### sleepmode.loft.sh/sleeping-since[​](#sleepmode-loft-sh-sleeping-since "Direct link to sleepmode.loft.sh/sleeping-since") **Type:** Annotation (read-only) **Example:** `sleepmode.loft.sh/sleeping-since: "1706745600"` **Used on:** Namespace, VirtualClusterInstance **Set by:** Platform Unix timestamp of when the namespace or tenant cluster entered auto sleep. Present only when sleeping. ### sleepmode.loft.sh/sleep-type[​](#sleepmode-loft-sh-sleep-type "Direct link to sleepmode.loft.sh/sleep-type") **Type:** Annotation (read-only) **Example:** `sleepmode.loft.sh/sleep-type: "inactivitySleep"` **Used on:** Namespace, VirtualClusterInstance **Set by:** Platform Indicates how sleep was triggered. Values: `inactivitySleep`, `forcedSleep`, `forcedDurationSleep`, `scheduledSleep`. ### sleepmode.loft.sh/scheduled-sleep[​](#sleepmode-loft-sh-scheduled-sleep "Direct link to sleepmode.loft.sh/scheduled-sleep") **Type:** Annotation (read-only) **Example:** `sleepmode.loft.sh/scheduled-sleep: "1706832000"` **Used on:** Namespace, VirtualClusterInstance **Set by:** Platform Unix timestamp of the next scheduled sleep based on the sleep schedule. ### sleepmode.loft.sh/scheduled-wakeup[​](#sleepmode-loft-sh-scheduled-wakeup "Direct link to sleepmode.loft.sh/scheduled-wakeup") **Type:** Annotation (read-only) **Example:** `sleepmode.loft.sh/scheduled-wakeup: "1706774400"` **Used on:** Namespace, VirtualClusterInstance **Set by:** Platform Unix timestamp of the next scheduled wakeup based on the wakeup schedule. ### sleepmode.loft.sh/endpoint-slices[​](#sleepmode-loft-sh-endpoint-slices "Direct link to sleepmode.loft.sh/endpoint-slices") **Type:** Annotation (read-only) **Example:** `sleepmode.loft.sh/endpoint-slices: '{"endpoints":[{"addresses":["10.0.0.1"]}]}'` **Used on:** EndpointSlice **Set by:** Platform Stores the original endpoint slice configuration before sleep mode modifications for restoration on wakeup. ### sleepmode.loft.sh/endpoints-subsets[​](#sleepmode-loft-sh-endpoints-subsets "Direct link to sleepmode.loft.sh/endpoints-subsets") **Type:** Annotation (read-only) **Example:** `sleepmode.loft.sh/endpoints-subsets: '{"addresses":[{"ip":"10.0.0.1"}]}'` **Used on:** Endpoints **Set by:** Platform Stores the original endpoints subsets before sleep mode modifications for restoration on wakeup. ### sleepmode.loft.sh/service-selector[​](#sleepmode-loft-sh-service-selector "Direct link to sleepmode.loft.sh/service-selector") **Type:** Annotation (read-only) **Example:** `sleepmode.loft.sh/service-selector: '{"app":"nginx"}'` **Used on:** Service **Set by:** Platform Stores the original service selector before sleep mode modifications for restoration on wakeup. ### sleepmode.loft.sh/service-ports[​](#sleepmode-loft-sh-service-ports "Direct link to sleepmode.loft.sh/service-ports") **Type:** Annotation (read-only) **Example:** `sleepmode.loft.sh/service-ports: '[{"port":80,"targetPort":8080}]'` **Used on:** Service **Set by:** Platform Stores the original service ports before sleep mode modifications for restoration on wakeup. ### sleepmode.loft.sh/target-service-name[​](#sleepmode-loft-sh-target-service-name "Direct link to sleepmode.loft.sh/target-service-name") **Type:** Annotation **Example:** `sleepmode.loft.sh/target-service-name: "nginx-service"` **Used on:** Ingress **Set by:** Platform Identifies the target service for sleep mode ingress wakeup functionality. ### sleepmode.loft.sh/target-service-namespace[​](#sleepmode-loft-sh-target-service-namespace "Direct link to sleepmode.loft.sh/target-service-namespace") **Type:** Annotation **Example:** `sleepmode.loft.sh/target-service-namespace: "production"` **Used on:** Ingress **Set by:** Platform Identifies the target service namespace when the service is in a different namespace than the ingress. ### sleepmode.loft.sh/target-service-port[​](#sleepmode-loft-sh-target-service-port "Direct link to sleepmode.loft.sh/target-service-port") **Type:** Annotation **Example:** `sleepmode.loft.sh/target-service-port: "8080"` **Used on:** Ingress **Set by:** Platform Identifies the target service port for sleep mode ingress wakeup. Can be a port name or number. ### sleepmode.loft.sh/istio-virtual-service-http-routes[​](#sleepmode-loft-sh-istio-virtual-service-http-routes "Direct link to sleepmode.loft.sh/istio-virtual-service-http-routes") **Type:** Annotation (read-only) **Example:** `sleepmode.loft.sh/istio-virtual-service-http-routes: '[{"route":[{"destination":{"host":"nginx"}}]}]'` **Used on:** VirtualService (Istio) **Set by:** Platform Stores the original Istio virtual service HTTP routes before sleep mode modifications for restoration on wakeup. ### sleepmode.loft.sh/istio-virtual-service-sleeping[​](#sleepmode-loft-sh-istio-virtual-service-sleeping "Direct link to sleepmode.loft.sh/istio-virtual-service-sleeping") **Type:** Annotation (read-only) **Example:** `sleepmode.loft.sh/istio-virtual-service-sleeping: "true"` **Used on:** VirtualService (Istio) **Set by:** Platform Indicates that the Istio virtual service should continue reconciling to sleep or be restored when removed. ### sleepmode.loft.sh/http-route-request-mirror[​](#sleepmode-loft-sh-http-route-request-mirror "Direct link to sleepmode.loft.sh/http-route-request-mirror") **Type:** Annotation (read-only) **Example:** `sleepmode.loft.sh/http-route-request-mirror: "true"` **Used on:** HTTPRoute **Set by:** Platform Indicates that the platform owns and manages request mirror filters on this HTTPRoute for sleep mode functionality. ### sleepmode.loft.sh/http-route-sleeping[​](#sleepmode-loft-sh-http-route-sleeping "Direct link to sleepmode.loft.sh/http-route-sleeping") **Type:** Annotation (read-only) **Example:** `sleepmode.loft.sh/http-route-sleeping: "true"` **Used on:** HTTPRoute **Set by:** Platform Indicates that the HTTPRoute should continue reconciling to sleep mode configuration if present or be restored to normal operation when removed. ## User and team management[​](#user-team-management "Direct link to User and team management") These labels and annotations are used on user and team resources. ### loft.sh/user[​](#loft-sh-user "Direct link to loft.sh/user") **Type:** Label **Example:** `loft.sh/user: "john-doe"` **Used on:** Namespace, VirtualClusterInstance, SpaceInstance, AccessKey **Set by:** Platform Identifies the user that owns this resource. ### loft.sh/team[​](#loft-sh-team "Direct link to loft.sh/team") **Type:** Label **Example:** `loft.sh/team: "platform-team"` **Used on:** Namespace, VirtualClusterInstance, SpaceInstance, AccessKey **Set by:** Platform Identifies the team that owns this resource. ### loft.sh/last-activity[​](#loft-sh-last-activity "Direct link to loft.sh/last-activity") **Type:** Annotation **Example:** `loft.sh/last-activity: "1706745600"` **Used on:** User **Set by:** Platform Unix timestamp of the user's last activity in the platform. ### loft.sh/custom-data[​](#loft-sh-custom-data "Direct link to loft.sh/custom-data") **Type:** Annotation **Example:** `loft.sh/custom-data: '{"department":"engineering"}'` **Used on:** User **Set by:** User-configurable Custom JSON data attached to a user. Can be used for external integrations. ### loft.sh/create-account[​](#loft-sh-create-account "Direct link to loft.sh/create-account") **Type:** Annotation **Example:** `loft.sh/create-account: "true"` **Used on:** User **Set by:** User-configurable When `true`, automatically creates an account for this user. ### loft.sh/previous-email[​](#loft-sh-previous-email "Direct link to loft.sh/previous-email") **Type:** Annotation **Example:** `loft.sh/previous-email: "old@example.com"` **Used on:** User **Set by:** Platform Stores the user's previous email address after an email change. ### loft.sh/notification-email[​](#loft-sh-notification-email "Direct link to loft.sh/notification-email") **Type:** Annotation **Example:** `loft.sh/notification-email: "alerts@example.com"` **Used on:** User **Set by:** User-configurable Alternate email address for platform notifications. ### loft.sh/notification-email-change-time[​](#loft-sh-notification-email-change-time "Direct link to loft.sh/notification-email-change-time") **Type:** Annotation **Example:** `loft.sh/notification-email-change-time: "1706745600"` **Used on:** User **Set by:** Platform Unix timestamp when the notification email was last changed. ## SSO and authentication[​](#sso-authentication "Direct link to SSO and authentication") These annotations relate to single sign-on and authentication. ### loft.sh/single-sign-on[​](#loft-sh-single-sign-on "Direct link to loft.sh/single-sign-on") **Type:** Annotation **Example:** `loft.sh/single-sign-on: "true"` **Used on:** User, Team **Set by:** Platform Indicates that this user or team was created through SSO. ### loft.sh/sso-provider[​](#loft-sh-sso-provider "Direct link to loft.sh/sso-provider") **Type:** Annotation **Example:** `loft.sh/sso-provider: "github"` **Used on:** User, Team **Set by:** Platform Identifies the SSO provider that created this user or team. ## RBAC and access control[​](#rbac-access-control "Direct link to RBAC and access control") These labels and annotations control role-based access. ### loft.sh/admin[​](#loft-sh-admin "Direct link to loft.sh/admin") **Type:** Label **Example:** `loft.sh/admin: "true"` **Used on:** ClusterRoleBinding **Set by:** Platform Marks a ClusterRoleBinding as granting admin privileges. ### loft.sh/aggregate-to-admin[​](#loft-sh-aggregate-to-admin "Direct link to loft.sh/aggregate-to-admin") **Type:** Label **Example:** `loft.sh/aggregate-to-admin: "true"` **Used on:** ClusterRole **Set by:** User-configurable Aggregates this ClusterRole's permissions into the admin role. ### loft.sh/aggregate-to-view[​](#loft-sh-aggregate-to-view "Direct link to loft.sh/aggregate-to-view") **Type:** Label **Example:** `loft.sh/aggregate-to-view: "true"` **Used on:** ClusterRole **Set by:** User-configurable Aggregates this ClusterRole's permissions into the view role. ### loft.sh/default-template[​](#loft-sh-default-template "Direct link to loft.sh/default-template") **Type:** Label **Example:** `loft.sh/default-template: "true"` **Used on:** VirtualClusterTemplate, SpaceTemplate, ClusterAccountTemplate **Set by:** User-configurable Marks this template as the default when no template is specified. ### loft.sh/default-role[​](#loft-sh-default-role "Direct link to loft.sh/default-role") **Type:** Label **Example:** `loft.sh/default-role: "true"` **Used on:** ClusterRole **Set by:** User-configurable Marks this ClusterRole as the default role assigned to new users. ### loft.sh/management-default-role[​](#loft-sh-management-default-role "Direct link to loft.sh/management-default-role") **Type:** Label **Example:** `loft.sh/management-default-role: "true"` **Used on:** ClusterRole **Set by:** User-configurable Marks this ClusterRole as the default management role. ### loft.sh/management-namespace[​](#loft-sh-management-namespace "Direct link to loft.sh/management-namespace") **Type:** Label **Example:** `loft.sh/management-namespace: "loft"` **Used on:** Namespace **Set by:** Platform Identifies the namespace containing platform management resources. ### rbac.loft.sh/auto-update[​](#rbac-loft-sh-auto-update "Direct link to rbac.loft.sh/auto-update") **Type:** Annotation **Example:** `rbac.loft.sh/auto-update: "true"` **Used on:** ClusterRole, ClusterRoleBinding **Set by:** Platform When `true`, allows the platform to automatically update this RBAC resource. ### rbac.loft.sh/generation[​](#rbac-loft-sh-generation "Direct link to rbac.loft.sh/generation") **Type:** Annotation **Example:** `rbac.loft.sh/generation: "5"` **Used on:** ClusterRole, ClusterRoleBinding **Set by:** Platform Tracks the generation number for RBAC reconciliation. ## Access keys[​](#access-keys "Direct link to Access keys") These labels identify access key purposes and associations. ### loft.sh/cluster[​](#loft-sh-cluster-access-key "Direct link to loft.sh/cluster") **Type:** Label **Example:** `loft.sh/cluster: "production"` **Used on:** AccessKey **Set by:** Platform Associates this access key with a specific cluster agent. ### loft.sh/vcluster[​](#loft-sh-vcluster-access-key "Direct link to loft.sh/vcluster") **Type:** Label **Example:** `loft.sh/vcluster: "my-vcluster"` **Used on:** AccessKey **Set by:** Platform Associates this access key with a specific vCluster. ### loft.sh/runner[​](#loft-sh-runner-access-key "Direct link to loft.sh/runner") **Type:** Label **Example:** `loft.sh/runner: "ci-runner"` **Used on:** AccessKey **Set by:** Platform Associates this access key with a specific runner. ### loft.sh/control-plane-access-key[​](#loft-sh-control-plane-access-key "Direct link to loft.sh/control-plane-access-key") **Type:** Label **Example:** `loft.sh/control-plane-access-key: "true"` **Used on:** AccessKey **Set by:** Platform Identifies this access key as used for control plane communication. ### loft.sh/vcluster-node[​](#loft-sh-vcluster-node "Direct link to loft.sh/vcluster-node") **Type:** Label **Example:** `loft.sh/vcluster-node: "true"` **Used on:** AccessKey **Set by:** Platform Identifies this access key as used for vCluster node registration. ### platform.vcluster.com/cooldown-seconds[​](#platform-vcluster-com-cooldown-seconds "Direct link to platform.vcluster.com/cooldown-seconds") **Type:** Label **Example:** `platform.vcluster.com/cooldown-seconds: "300"` **Used on:** AccessKey **Set by:** Platform Specifies a custom cooldown duration in seconds for this access key, overriding the default cooldown period. ### platform.vcluster.com/shell-pod-uid[​](#platform-vcluster-com-shell-pod-uid "Direct link to platform.vcluster.com/shell-pod-uid") **Type:** Label **Example:** `platform.vcluster.com/shell-pod-uid: "a1b2c3d4-e5f6-7890-abcd-ef1234567890"` **Used on:** AccessKey **Set by:** Platform Associates this access key with a specific shell pod by its UID for session management. ### platform.vcluster.com/metrics-writer[​](#platform-vcluster-com-metrics-writer "Direct link to platform.vcluster.com/metrics-writer") **Type:** Label **Example:** `platform.vcluster.com/metrics-writer: "true"` **Used on:** AccessKey **Set by:** Platform Identifies a metrics-writer access key issued for a tenant cluster to write observability data to the platform's metrics backend. ## Integrations[​](#integrations "Direct link to Integrations") These annotations and labels configure external integrations. ### loft.sh/import-argocd[​](#loft-sh-import-argocd "Direct link to loft.sh/import-argocd") **Type:** Label **Example:** `loft.sh/import-argocd: "true"` **Used on:** VirtualClusterInstance, Cluster **Set by:** User-configurable Part of the legacy project-level Argo CD integration. When set to `"true"`, the platform automatically registers this tenant cluster or cluster with Argo CD using the project-level configuration. For new setups, use the connector-based integration instead. See [Connect to Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md). ### loft.sh/user-managed-destinations[​](#loft-sh-user-managed-destinations "Direct link to loft.sh/user-managed-destinations") **Type:** Annotation **Example:** `loft.sh/user-managed-destinations: '["https://kubernetes.default.svc"]'` **Used on:** AppProject (ArgoCD) **Set by:** Platform Part of the legacy project-level Argo CD integration. Tracks which Argo CD AppProject destinations are managed by the project controller. This prevents the controller from removing destinations managed by tenant clusters when syncing project specifications. ### loft.sh/argocd-connector[​](#loft-sh-argocd-connector "Direct link to loft.sh/argocd-connector") **Type:** Annotation **Example:** `loft.sh/argocd-connector: "my-argocd-connector"` **Used on:** ArgoCD application resources **Set by:** Platform Tracks the connector name last used to sync this ArgoCD application. A change in this value triggers cleanup on the old backend before the application is created on the new connector. ### loft.sh/argocd-akuity-connector[​](#loft-sh-argocd-akuity-connector "Direct link to loft.sh/argocd-akuity-connector") **Type:** Annotation **Example:** `loft.sh/argocd-akuity-connector: "akuity-connector:small"` **Used on:** VirtualClusterInstance **Set by:** Platform Stores the Akuity connector name and agent size (format: "connectorName:agentSize") for the currently installed Akuity agent. A change in either component triggers full cleanup and re-registration of the ArgoCD integration. ### loft.sh/argocd-akuity-manifest-hash[​](#loft-sh-argocd-akuity-manifest-hash "Direct link to loft.sh/argocd-akuity-manifest-hash") **Type:** Annotation **Example:** `loft.sh/argocd-akuity-manifest-hash: "a1b2c3d4e5f67890abcdef1234567890abcdef1234567890abcdef1234567890"` **Used on:** VirtualClusterInstance **Set by:** Platform The SHA-256 hash of the last applied Akuity agent manifest. Re-application is skipped when the hash is unchanged, preventing overwrites of manual or operator-driven patches. ### loft.sh/argocd-application-id[​](#loft-sh-argocd-application-id "Direct link to loft.sh/argocd-application-id") **Type:** Label **Example:** `loft.sh/argocd-application-id: "my-app"` **Used on:** Resources associated with ArgoCD applications **Set by:** Platform Identifies the ArgoCD application resource name associated with this resource. ### loft.sh/argocd-registered-app-name[​](#loft-sh-argocd-registered-app-name "Direct link to loft.sh/argocd-registered-app-name") **Type:** Annotation **Example:** `loft.sh/argocd-registered-app-name: "my-app"` **Used on:** VirtualClusterInstance **Set by:** Platform Stores the ArgoCD application name that was last successfully registered. Used internally to track registration state. ### loft.sh/argocd-registered-cluster-name[​](#loft-sh-argocd-registered-cluster-name "Direct link to loft.sh/argocd-registered-cluster-name") **Type:** Annotation **Example:** `loft.sh/argocd-registered-cluster-name: "vcluster-prod-abc123"` **Used on:** VirtualClusterInstance **Set by:** Platform Stores the ArgoCD cluster name that was last successfully registered. Used to detect and migrate name changes when a platform license is added and the cluster name gains an instance-ID hash suffix. ### loft.sh/connector-type[​](#loft-sh-connector-type "Direct link to loft.sh/connector-type") **Type:** Label **Example:** `loft.sh/connector-type: "rancher"` **Used on:** Connector **Set by:** Platform Identifies the type of external connector (rancher, etc.). ### loft.sh/made-by-connector[​](#loft-sh-made-by-connector "Direct link to loft.sh/made-by-connector") **Type:** Annotation **Example:** `loft.sh/made-by-connector: "rancher-connector"` **Used on:** Cluster **Set by:** Platform Indicates this cluster was imported by an external connector. ### loft.sh/is-imported[​](#loft-sh-is-imported "Direct link to loft.sh/is-imported") **Type:** Annotation **Example:** `loft.sh/is-imported: "true"` **Used on:** Cluster, VirtualClusterInstance **Set by:** Platform Indicates this resource was imported into the platform rather than created by it. ### platform.vcluster.com/made-by-connector[​](#platform-vcluster-com-made-by-connector "Direct link to platform.vcluster.com/made-by-connector") **Type:** Annotation **Example:** `platform.vcluster.com/made-by-connector: "database-connector"` **Used on:** Database resources **Set by:** Platform Identifies the connector that was used to create the database. ## Observability[​](#observability "Direct link to Observability") These labels are used for platform observability features including fleet metrics and monitoring. ### platform.vcluster.com/fleet-observability-connector[​](#platform-vcluster-com-fleet-observability-connector "Direct link to platform.vcluster.com/fleet-observability-connector") **Type:** Label **Example:** `platform.vcluster.com/fleet-observability-connector: "true"` **Used on:** Connector **Set by:** Platform Identifies observability connectors that provide data to the platform-wide fleet metrics Grafana instance. ### platform.vcluster.com/self-signed[​](#platform-vcluster-com-self-signed "Direct link to platform.vcluster.com/self-signed") **Type:** Label **Example:** `platform.vcluster.com/self-signed: "true"` **Used on:** Secret **Set by:** Platform Marks a gateway serving certificate Secret as issued by the platform controller. Distinguishes platform-managed certificates from user-provided certificates. ## Networking[​](#networking "Direct link to Networking") These annotations configure network-related features. ### loft.sh/network-peer-type[​](#loft-sh-network-peer-type "Direct link to loft.sh/network-peer-type") **Type:** Annotation **Example:** `loft.sh/network-peer-type: "tailscale"` **Used on:** NetworkPeer **Set by:** Platform Identifies the type of network peer connection. ### loft.sh/network-peer-tags[​](#loft-sh-network-peer-tags "Direct link to loft.sh/network-peer-tags") **Type:** Annotation **Example:** `loft.sh/network-peer-tags: "tag:production,tag:us-east"` **Used on:** NetworkPeer **Set by:** User-configurable Tailscale tags for this network peer. ### loft.sh/network-peer-routes[​](#loft-sh-network-peer-routes "Direct link to loft.sh/network-peer-routes") **Type:** Annotation **Example:** `loft.sh/network-peer-routes: "10.0.0.0/8,172.16.0.0/12"` **Used on:** NetworkPeer **Set by:** User-configurable Routes to advertise for this network peer. ### loft.sh/machine-key[​](#loft-sh-machine-key "Direct link to loft.sh/machine-key") **Type:** Label **Example:** `loft.sh/machine-key: "mkey1234567890abcdefghijklmnopqrstuvwxyz0123456789ab"` **Used on:** NetworkPeer **Set by:** Platform Indexes a NetworkPeer by its Tailscale machine public key, encoded as 52 base32 characters to satisfy Kubernetes' label value length limits. ### loft.sh/allowed-hostname[​](#loft-sh-allowed-hostname "Direct link to loft.sh/allowed-hostname") **Type:** Annotation **Example:** `loft.sh/allowed-hostname: "cluster.internal"` **Used on:** AccessKey **Set by:** User-configurable Restricts this access key to connections from specific hostnames. ### loft.sh/allowed-peers[​](#loft-sh-allowed-peers "Direct link to loft.sh/allowed-peers") **Type:** Annotation **Example:** `loft.sh/allowed-peers: "peer1,peer2"` **Used on:** AccessKey **Set by:** User-configurable Restricts this access key to connections from specific network peers. ### loft.sh/coordinator-instance-id[​](#loft-sh-coordinator-instance-id "Direct link to loft.sh/coordinator-instance-id") **Type:** Annotation **Example:** `loft.sh/coordinator-instance-id: "coord-123"` **Used on:** NetworkPeer **Set by:** Platform Identifies the coordination instance for distributed networking. ### loft.sh/ingress-mirror[​](#loft-sh-ingress-mirror "Direct link to loft.sh/ingress-mirror") **Type:** Annotation **Example:** `loft.sh/ingress-mirror: "true"` **Used on:** Ingress **Set by:** Platform Indicates that this ingress is a mirror of another ingress resource for management purposes. ### loft.sh/network-peer-persistence[​](#loft-sh-network-peer-persistence "Direct link to loft.sh/network-peer-persistence") **Type:** Annotation **Example:** `loft.sh/network-peer-persistence: "true"` **Used on:** NetworkPeer, AccessKey **Set by:** User-configurable Marks a network peer as persistent to exclude it from garbage collection. When set on an access key, prevents garbage collection of network peers created by that access key, enabling compatibility with vanilla Tailscale clients. ### loft.sh/ssh-keys[​](#loft-sh-ssh-keys "Direct link to loft.sh/ssh-keys") **Type:** Annotation **Example:** `loft.sh/ssh-keys: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB...,ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA..."` **Used on:** AccessKey **Set by:** User-configurable Specifies a comma-separated list of SSH public keys that are allowed for network peers created by this access key. Used to restrict network peer authentication. ## Shared and project secrets[​](#shared-project-secrets "Direct link to Shared and project secrets") These annotations and labels are used for secret management. ### loft.sh/encrypted[​](#loft-sh-encrypted "Direct link to loft.sh/encrypted") **Type:** Annotation **Example:** `loft.sh/encrypted: "true"` **Used on:** Secret **Set by:** Platform Marks a Secret whose data is encrypted at rest. Paired with encryption on write and decryption on read operations. ### loft.sh/tenant-config-secret[​](#loft-sh-tenant-config-secret "Direct link to loft.sh/tenant-config-secret") **Type:** Label **Example:** `loft.sh/tenant-config-secret: "true"` **Used on:** Secret **Set by:** Platform Marks a Secret as a tenant's backing config Secret. Platform backup selects Secrets by this label to collect per-tenant configuration (SSO client secrets, UI settings) alongside the Tenant resource. ### loft.sh/sharedsecret-name[​](#loft-sh-sharedsecret-name "Direct link to loft.sh/sharedsecret-name") **Type:** Label **Example:** `loft.sh/sharedsecret-name: "database-creds"` **Used on:** Secret **Set by:** Platform The name of the shared secret this secret was created from. ### loft.sh/sharedsecret-namespace[​](#loft-sh-sharedsecret-namespace "Direct link to loft.sh/sharedsecret-namespace") **Type:** Label **Example:** `loft.sh/sharedsecret-namespace: "loft-default-p-default-s-default"` **Used on:** Secret **Set by:** Platform The namespace where the source shared secret is stored. ### loft.sh/disable-sync[​](#loft-sh-disable-sync "Direct link to loft.sh/disable-sync") **Type:** Annotation **Example:** `loft.sh/disable-sync: "true"` **Used on:** Secret **Set by:** User-configurable When set, prevents the platform from syncing this secret from a shared secret. ### loft.sh/project-secret[​](#loft-sh-project-secret "Direct link to loft.sh/project-secret") **Type:** Label **Example:** `loft.sh/project-secret: "true"` **Used on:** Secret **Set by:** Platform Marks this secret as a synced instance of a project secret. ### loft.sh/project-secret-name[​](#loft-sh-project-secret-name "Direct link to loft.sh/project-secret-name") **Type:** Annotation **Example:** `loft.sh/project-secret-name: "api-keys"` **Used on:** Secret **Set by:** Platform The name of the project secret this secret was created from. ### loft.sh/project-secret-description[​](#loft-sh-project-secret-description "Direct link to loft.sh/project-secret-description") **Type:** Annotation **Example:** `loft.sh/project-secret-description: "API keys for external services"` **Used on:** ProjectSecret **Set by:** User-configurable Human-readable description of the project secret. ### loft.sh/project-secret-displayname[​](#loft-sh-project-secret-displayname "Direct link to loft.sh/project-secret-displayname") **Type:** Annotation **Example:** `loft.sh/project-secret-displayname: "External API Keys"` **Used on:** ProjectSecret **Set by:** User-configurable Display name for the project secret shown in the UI. ### loft.sh/project-secret-owner[​](#loft-sh-project-secret-owner "Direct link to loft.sh/project-secret-owner") **Type:** Annotation **Example:** `loft.sh/project-secret-owner: "user:john-doe"` **Used on:** ProjectSecret **Set by:** Platform Identifies the owner of this project secret. ### loft.sh/project-secret-access[​](#loft-sh-project-secret-access "Direct link to loft.sh/project-secret-access") **Type:** Annotation **Example:** `loft.sh/project-secret-access: "project"` **Used on:** ProjectSecret **Set by:** User-configurable Access scope for the project secret. ## Stack management[​](#stack-management "Direct link to Stack management") These annotations and labels are used for StackInstance management and task tracking. ### platform.vcluster.com/stack[​](#platform-vcluster-com-stack "Direct link to platform.vcluster.com/stack") **Type:** Label **Example:** `platform.vcluster.com/stack: "my-stack-instance"` **Used on:** Resources created by StackInstance tasks **Set by:** Platform Identifies the StackInstance that created this resource. The stack controller uses this label to list all child resources belonging to a specific stack instance. ### platform.vcluster.com/stack-config-name[​](#platform-vcluster-com-stack-config-name "Direct link to platform.vcluster.com/stack-config-name") **Type:** Label **Example:** `platform.vcluster.com/stack-config-name: "postgres-db"` **Used on:** Stack custom resources **Set by:** Platform Stores the name a stack was configured under in vcluster.yaml. The resource name is derived from this value but cannot be turned back into it. ### platform.vcluster.com/stack-task[​](#platform-vcluster-com-stack-task "Direct link to platform.vcluster.com/stack-task") **Type:** Annotation **Example:** `platform.vcluster.com/stack-task: "deploy-database"` **Used on:** Resources created by StackInstance tasks **Set by:** Platform Records the task name that created this resource. Used to link a child resource to its task definition and to detect orphaned resources when tasks are removed from the stack. ### platform.vcluster.com/stack-retry[​](#platform-vcluster-com-stack-retry "Direct link to platform.vcluster.com/stack-retry") **Type:** Annotation **Example:** `platform.vcluster.com/stack-retry: "deploy-database"` **Used on:** StackInstance **Set by:** User-configurable Requests a retry of a specific task by name, or all applicable tasks when set to `"all"`. A failed AppInstance can't be re-triggered this way; edit its Stack task instead. The Stack controller processes the request and then removes the annotation. See [Retry a stack](https://vcluster.com/docs/platform/troubleshoot/stacks.md#retry-a-stack) for the procedure and limitations. ## Applications[​](#applications "Direct link to Applications") These labels are used for application management. ### loft.sh/app[​](#loft-sh-app "Direct link to loft.sh/app") **Type:** Label **Example:** `loft.sh/app: "nginx"` **Used on:** Helm release resources **Set by:** Platform Identifies resources belonging to a platform-managed application. ### loft.sh/app-instance[​](#loft-sh-app-instance "Direct link to loft.sh/app-instance") **Type:** Annotation **Example:** `loft.sh/app-instance: "default/my-app-instance"` **Used on:** HelmRelease **Set by:** Platform Stamped onto every Helm release deployed by an AppInstance to attribute the release to the AppInstance that manages it. The value format is `/`. ### loft.sh/system-app[​](#loft-sh-system-app "Direct link to loft.sh/system-app") **Type:** Label **Example:** `loft.sh/system-app: "true"` **Used on:** Application resources **Set by:** Platform Marks this application as a system application managed by the platform. ### loft.sh/extra-recommended-apps[​](#loft-sh-extra-recommended-apps "Direct link to loft.sh/extra-recommended-apps") **Type:** Annotation **Example:** `loft.sh/extra-recommended-apps: "prometheus,grafana"` **Used on:** Cluster **Set by:** User-configurable Comma-separated list of additional recommended applications for this cluster. ### loft.sh/app-name[​](#loft-sh-app-name "Direct link to loft.sh/app-name") **Type:** Annotation **Example:** `loft.sh/app-name: "nginx"` **Used on:** HelmRelease **Set by:** Platform Indicates that the Helm release was deployed via the platform app store and identifies the app name. ### loft.sh/app-version[​](#loft-sh-app-version "Direct link to loft.sh/app-version") **Type:** Annotation **Example:** `loft.sh/app-version: "1.2.3"` **Used on:** HelmRelease **Set by:** Platform The version of the platform app that was deployed. ### loft.sh/url[​](#loft-sh-url "Direct link to loft.sh/url") **Type:** Annotation **Example:** `loft.sh/url: "https://charts.example.com/stable"` **Used on:** HelmRelease **Set by:** Platform The Helm repository URL from which the release was deployed. ### loft.sh/insecure-skip-tls[​](#loft-sh-insecure-skip-tls "Direct link to loft.sh/insecure-skip-tls") **Type:** Annotation **Example:** `loft.sh/insecure-skip-tls: "true"` **Used on:** HelmRelease **Set by:** User-configurable When `true`, TLS certificate verification is skipped during Helm operations for this release. ## Cleanup and finalizers[​](#cleanup-finalizers "Direct link to Cleanup and finalizers") These finalizers and labels control resource cleanup behavior. ### loft.sh/cleanup[​](#loft-sh-cleanup "Direct link to loft.sh/cleanup") **Type:** Finalizer **Example:** `finalizers: ["loft.sh/cleanup"]` **Used on:** Various resources **Set by:** Platform General cleanup finalizer ensuring proper resource deletion. ### loft.sh/cleanup-management[​](#loft-sh-cleanup-management "Direct link to loft.sh/cleanup-management") **Type:** Finalizer **Example:** `finalizers: ["loft.sh/cleanup-management"]` **Used on:** Cluster, Project **Set by:** Platform Ensures management resources are cleaned up when the parent resource is deleted. ### loft.sh/cleanup-argocd-integration[​](#loft-sh-cleanup-argocd-integration "Direct link to loft.sh/cleanup-argocd-integration") **Type:** Finalizer **Example:** `finalizers: ["loft.sh/cleanup-argocd-integration"]` **Used on:** VirtualClusterInstance **Set by:** Platform Blocks VirtualClusterInstance deletion until managed ArgoCD applications and cluster registrations are deleted through the ArgoCD API. ### platform.vcluster.com/cleanup-stacks[​](#platform-vcluster-com-cleanup-stacks "Direct link to platform.vcluster.com/cleanup-stacks") **Type:** Finalizer **Example:** `finalizers: ["platform.vcluster.com/cleanup-stacks"]` **Used on:** VirtualClusterInstance **Set by:** Platform Blocks VirtualClusterInstance deletion until every managed StackInstance is deleted, independently of the Argo CD integration finalizer. ### loft.sh/cleanup-workload[​](#loft-sh-cleanup-workload "Direct link to loft.sh/cleanup-workload") **Type:** Finalizer **Example:** `finalizers: ["loft.sh/cleanup-workload"]` **Used on:** VirtualClusterInstance, SpaceInstance **Set by:** Platform Ensures workload resources are cleaned up when deleted. ### loft.sh/cleanup-rancher[​](#loft-sh-cleanup-rancher "Direct link to loft.sh/cleanup-rancher") **Type:** Finalizer **Example:** `finalizers: ["loft.sh/cleanup-rancher"]` **Used on:** Cluster **Set by:** Platform Ensures Rancher integration resources are cleaned up. ### loft.sh/cleanup-connectors[​](#loft-sh-cleanup-connectors "Direct link to loft.sh/cleanup-connectors") **Type:** Finalizer **Example:** `finalizers: ["loft.sh/cleanup-connectors"]` **Used on:** Cluster **Set by:** Platform Ensures connector resources are cleaned up. ### loft.sh/cleanup-nodes[​](#loft-sh-cleanup-nodes "Direct link to loft.sh/cleanup-nodes") **Type:** Finalizer **Example:** `finalizers: ["loft.sh/cleanup-nodes"]` **Used on:** Cluster **Set by:** Platform Ensures dynamically provisioned nodes are cleaned up. ### loft.sh/cleanup-cloud-resources[​](#loft-sh-cleanup-cloud-resources "Direct link to loft.sh/cleanup-cloud-resources") **Type:** Finalizer **Example:** `finalizers: ["loft.sh/cleanup-cloud-resources"]` **Used on:** Cluster **Set by:** Platform Ensures cloud provider resources are cleaned up. ### loft.sh/cleanup-identity-provider[​](#loft-sh-cleanup-identity-provider "Direct link to loft.sh/cleanup-identity-provider") **Type:** Finalizer **Example:** `finalizers: ["loft.sh/cleanup-identity-provider"]` **Used on:** SSO configuration **Set by:** Platform Ensures identity provider resources are cleaned up. ### platform.vcluster.com/cleanup-observability-connector[​](#platform-vcluster-com-cleanup-observability-connector "Direct link to platform.vcluster.com/cleanup-observability-connector") **Type:** Finalizer **Example:** `finalizers: ["platform.vcluster.com/cleanup-observability-connector"]` **Used on:** Connector **Set by:** Platform Blocks connector deletion until the observability datasource is cleaned up. ### platform.vcluster.com/cleanup-observability-gateway[​](#platform-vcluster-com-cleanup-observability-gateway "Direct link to platform.vcluster.com/cleanup-observability-gateway") **Type:** Finalizer **Example:** `finalizers: ["platform.vcluster.com/cleanup-observability-gateway"]` **Used on:** Secret **Set by:** Platform Blocks connector secret deletion until the observability gateway is cleaned up. ## Drift detection[​](#drift-detection "Direct link to Drift detection") These annotations control drift detection behavior. ### drift.loft.sh/force-check[​](#drift-loft-sh-force-check "Direct link to drift.loft.sh/force-check") **Type:** Annotation **Example:** `drift.loft.sh/force-check: "true"` **Used on:** VirtualClusterInstance, SpaceInstance **Set by:** User-configurable Forces an immediate drift check on this resource. ## Node providers[​](#node-providers "Direct link to Node providers") These annotations control node provider behavior. ### vcluster.com/pause-deploy[​](#vcluster-com-pause-deploy "Direct link to vcluster.com/pause-deploy") **Type:** Annotation **Example:** `vcluster.com/pause-deploy: "true"` **Used on:** NodeProvider **Set by:** User-configurable Pauses Helm install, upgrade, and cleanup of the node provider's deployed components, such as Metal3/Ironic, Multus, and the DHCP server. Node type sync and provider startup keep running, and existing Helm releases are left untouched. Remove the annotation to resume normal deployment reconciliation. ## Miscellaneous[​](#miscellaneous "Direct link to Miscellaneous") These annotations are used for various platform features. ### loft.sh/version[​](#loft-sh-version "Direct link to loft.sh/version") **Type:** Annotation **Example:** `loft.sh/version: "4.0.0"` **Used on:** Platform configuration **Set by:** Platform The platform version that last modified this resource. ### loft.sh/managed-by[​](#loft-sh-managed-by "Direct link to loft.sh/managed-by") **Type:** Label **Example:** `loft.sh/managed-by: "project-controller"` **Used on:** Various generated resources **Set by:** Platform Names the controller that manages a generated resource. Used for resource ownership tracking and lifecycle management. ### loft.sh/warn-deletion[​](#loft-sh-warn-deletion "Direct link to loft.sh/warn-deletion") **Type:** Annotation **Example:** `loft.sh/warn-deletion: "true"` **Used on:** Cluster, Project, VirtualClusterInstance **Set by:** User-configurable Enables a deletion warning in the UI for this resource. ### loft.sh/non-deletable[​](#loft-sh-non-deletable "Direct link to loft.sh/non-deletable") **Type:** Annotation **Example:** `loft.sh/non-deletable: "true"` **Used on:** Various resources **Set by:** User-configurable Prevents deletion of this resource through the platform API and UI. ### loft.sh/platform-db-applied-time[​](#loft-sh-platform-db-applied-time "Direct link to loft.sh/platform-db-applied-time") **Type:** Annotation **Example:** `loft.sh/platform-db-applied-time: "1706745600"` **Used on:** Platform database resources **Set by:** Platform Timestamp of when database migrations were last applied. ### platform.vcluster.com/is-browser-shell-ns[​](#platform-vcluster-com-is-browser-shell-ns "Direct link to platform.vcluster.com/is-browser-shell-ns") **Type:** Annotation **Example:** `platform.vcluster.com/is-browser-shell-ns: "true"` **Used on:** Namespace **Set by:** Platform Indicates and confirms that a namespace was created for the browser shell feature. --- # Platform CLI hub This page is a Platform-focused entry point to the `vcluster platform` CLI. It groups commands by task and links each one to its full flag reference. The canonical, flag-level reference for every command lives in the [`vcluster` CLI docs](https://vcluster.com/docs/vcluster/cli/vcluster_platform). This page shows the Platform UI equivalent next to each CLI command. ## Common tasks and CLI equivalents[​](#common-tasks-and-cli-equivalents "Direct link to Common tasks and CLI equivalents") ### Start or install the platform[​](#start-or-install-the-platform "Direct link to Start or install the platform") | Task | Platform UI | CLI command | | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | [Install Platform](https://vcluster.com/docs/platform/install/quick-start-guide.md) | Not available in the UI. Platform installs with Helm or the CLI. | [`vcluster platform start`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_start) | See the [quickstart guide](https://vcluster.com/docs/platform/install/quick-start-guide.md) or the [Helm installation guide](https://vcluster.com/docs/platform/install/helm.md) for full install flows. ### Log in and access keys[​](#log-in-and-access-keys "Direct link to Log in and access keys") | Task | Platform UI | CLI command | | ----------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | [Log in to Platform](https://vcluster.com/docs/platform/install/quick-start-guide.md#login) | Sign in at the Platform URL | [`vcluster platform login`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_login) | | Log out of Platform | Profile picture and name (bottom of the sidebar) > `Logout` | [`vcluster platform logout`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_logout) | | [Create an access key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md#create-an-access-key) | `Access & Secrets` > `Users & Roles` > select a user > `Access Keys` > `Create Access Key` > configure it in the configuration sheet > `Create Access Key` | [`vcluster platform create accesskey`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_create_accesskey) | ### Connect a cluster[​](#connect-a-cluster "Direct link to Connect a cluster") | Task | Platform UI | CLI command | | ------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | [Register a cluster with Platform](https://vcluster.com/docs/platform/administer/clusters/connect-cluster.md) | `Infrastructure` > `Control Plane Clusters` > `Connect Cluster via CLI` > follow the configuration sheet's generated command | [`vcluster platform add cluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_add_cluster) | | Create a local kubeconfig context for a connected cluster | `Infrastructure` > `Control Plane Clusters` > three-dot menu for the desired cluster > `Access`, then copy the generated command | [`vcluster platform connect cluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_connect_cluster) | ### Create, list, and delete tenant clusters[​](#create-list-and-delete-tenant-clusters "Direct link to Create, list, and delete tenant clusters") | Task | Platform UI | CLI command | | ------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | [Create a tenant cluster](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-with-template.md) | project dropdown > `Tenant Clusters` > `Create Tenant Cluster` | [`vcluster platform create vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_create_vcluster) | | [Add an existing tenant cluster](https://vcluster.com/docs/platform/use-platform/virtual-clusters/add-virtual-clusters.md) | `Tenant Clusters` > `Add Existing Cluster` > `Add existing tenant cluster` | [`vcluster platform add vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_add_vcluster) | | List tenant clusters | project dropdown > `Tenant Clusters` | [`vcluster platform list vclusters`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_list_vclusters) | | Delete a tenant cluster | `Tenant Clusters` > three-dot menu (`More actions`) for the desired cluster > `Delete` | [`vcluster platform delete vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_delete_vcluster) | ### Create, list, and delete namespaces[​](#create-list-and-delete-namespaces "Direct link to Create, list, and delete namespaces") | Task | Platform UI | CLI command | | -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | [Create a namespace](https://vcluster.com/docs/platform/use-platform/namespaces/create/create-with-template.md) | `Projects` > `Namespaces` > `Create Namespace` > optional template > `Create Namespace` | [`vcluster platform create namespace`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_create_namespace) | | List namespaces | `Projects` > `Namespaces` | [`vcluster platform list namespaces`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_list_namespaces) | | Delete a namespace | `Namespaces` view, hover the row, click the delete icon in the Actions column | [`vcluster platform delete namespace`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_delete_namespace) | See [retrieve a namespace kubeconfig](https://vcluster.com/docs/platform/use-platform/namespaces/retrieve-kube-config.md) for related namespace commands. ### Sleep, wake up, and share resources[​](#sleep-wake-up-and-share-resources "Direct link to Sleep, wake up, and share resources") | Task | Platform UI | CLI command | | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | [Put a namespace to sleep](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/sleep-mode.md#put-a-namespace-to-sleep) | `Namespaces` > actions for the desired namespace > `Sleep` | [`vcluster platform sleep namespace`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_sleep_namespace) | | [Wake up a namespace](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/sleep-mode.md#wake-a-namespace) | `Namespaces` > actions for the desired namespace > `Wake Up` | [`vcluster platform wakeup namespace`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_wakeup_namespace) | | [Put a tenant cluster to sleep](https://vcluster.com/docs/platform/use-platform/virtual-clusters/key-features/sleep-mode.md#put-a-tenant-cluster-to-sleep) | `Tenant Clusters` > `More actions` for the desired cluster > `Sleep` | [`vcluster platform sleep vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_sleep_vcluster) | | [Wake up a tenant cluster](https://vcluster.com/docs/platform/use-platform/virtual-clusters/key-features/sleep-mode.md#wake-a-tenant-cluster) | `Tenant Clusters` > `More actions` for the desired cluster > `Wake Up`, shown only while sleeping | [`vcluster platform wakeup vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_wakeup_vcluster) | | [Share a tenant cluster](https://vcluster.com/docs/platform/use-platform/virtual-clusters/manage-access.md) | `Tenant Clusters` > `Edit` > `Permissions` > `Add Permission` > `Save Changes` | [`vcluster platform share vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_share_vcluster) | | [Share a namespace](https://vcluster.com/docs/platform/administer/users-permissions/permissions/namespace.md) | `Namespaces` > `Edit` > `Permissions` > `Save Changes` | [`vcluster platform share namespace`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_share_namespace) | ### Backup, destroy, and reset password[​](#backup-destroy-and-reset-password "Direct link to Backup, destroy, and reset password") | Task | Platform UI | CLI command | | -------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | [Back up Platform](https://vcluster.com/docs/platform/maintenance/backup-restore/backup-restore-platform.md) | Not available in the UI, CLI/Helm only | [`vcluster platform backup management`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_backup_management) | | [Destroy a Platform instance](https://vcluster.com/docs/platform/maintenance/uninstall.md#delete-the-platform) | Not available in the UI, CLI/Helm only | [`vcluster platform destroy`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_destroy) | | [Reset a user's password](https://vcluster.com/docs/platform/administer/users-permissions/managing-users/reset-password.md) | `Access & Secrets` > `Users & Roles` > select a user > `Reset Password` > confirm > `Copy & Close` | [`vcluster platform reset password`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_reset_password) | Resetting the admin password without UI access is CLI/kubectl only, see [reset the admin password](https://vcluster.com/docs/platform/administer/authentication/reset-admin-password.md). ## All platform CLI commands[​](#all-platform-cli-commands "Direct link to All platform CLI commands") ### Start[​](#start "Direct link to Start") | Command | Description | | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------ | | [`vcluster platform start`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_start) | Start a vCluster platform instance and connect via port-forwarding | ### Login[​](#login "Direct link to Login") | Command | Description | | ------------------------------------------------------------------------------------------------ | ------------------------------------- | | [`vcluster platform login`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_login) | Login to a vCluster platform instance | ### Logout[​](#logout "Direct link to Logout") | Command | Description | | -------------------------------------------------------------------------------------------------- | --------------------------------------- | | [`vcluster platform logout`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_logout) | Log out of a vCluster platform instance | ### Access keys[​](#access-keys "Direct link to Access keys") | Command | Description | | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | | [`vcluster platform access-key`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_access-key) | Prints the access token to a vCluster platform instance | ### Connect[​](#connect "Direct link to Connect") | Command | Description | | -------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | | [`vcluster platform connect`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_connect) | connects to vCluster platform resources | | [`vcluster platform connect cluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_connect_cluster) | Creates a kube context for the given cluster | | [`vcluster platform connect management`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_connect_management) | Creates a kube context to the vCluster platform Management API | | [`vcluster platform connect namespace`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_connect_namespace) | Creates a kube context for the given vCluster platform namespace | | [`vcluster platform connect vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_connect_vcluster) | Connect to a virtual cluster | ### Add[​](#add "Direct link to Add") | Command | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- | | [`vcluster platform add`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_add) | Adds a cluster to vCluster platform | | [`vcluster platform add cluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_add_cluster) | add current cluster to vCluster platform | | [`vcluster platform add standalone`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_add_standalone) | Adds an existing vCluster Standalone cluster to the vCluster platform | | [`vcluster platform add vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_add_vcluster) | Adds an existing vCluster to the vCluster platform | ### Create[​](#create "Direct link to Create") | Command | Description | | ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | | [`vcluster platform create`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_create) | Creates vCluster platform resources | | [`vcluster platform create accesskey`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_create_accesskey) | Creates a new access key for the current user | | [`vcluster platform create namespace`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_create_namespace) | Creates a new vCluster platform namespace in the given cluster | | [`vcluster platform create vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_create_vcluster) | Creates a new virtual cluster | ### Delete[​](#delete "Direct link to Delete") | Command | Description | | ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | | [`vcluster platform delete`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_delete) | Deletes vCluster platform resources | | [`vcluster platform delete namespace`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_delete_namespace) | Deletes a vCluster platform namespace from a cluster | | [`vcluster platform delete vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_delete_vcluster) | Deletes a virtual cluster | ### List[​](#list "Direct link to List") | Command | Description | | -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | | [`vcluster platform list`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_list) | Lists configuration | | [`vcluster platform list clusters`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_list_clusters) | Lists the loft clusters you have access to | | [`vcluster platform list namespaces`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_list_namespaces) | Lists the vCluster platform namespaces you have access to | | [`vcluster platform list projects`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_list_projects) | Lists the loft projects you have access to | | [`vcluster platform list secrets`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_list_secrets) | Lists all the shared secrets you have access to | | [`vcluster platform list teams`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_list_teams) | Lists the vCluster platform teams you are member of | | [`vcluster platform list vclusters`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_list_vclusters) | Lists all virtual clusters that are connected to the current platform | ### Get[​](#get "Direct link to Get") | Command | Description | | ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | | [`vcluster platform get`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_get) | Retrieves and display informations | | [`vcluster platform get cluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_get_cluster) | Prints the current cluster | | [`vcluster platform get cluster-access-key`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_get_cluster-access-key) | Retrieve the network peer cluster access key | | [`vcluster platform get current-user`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_get_current-user) | Retrieves the current logged in user | | [`vcluster platform get secret`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_get_secret) | Returns the key value of a project / shared secret | ### Set[​](#set "Direct link to Set") | Command | Description | | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | | [`vcluster platform set`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_set) | Set configuration | | [`vcluster platform set secret`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_set_secret) | Sets the key value of a project / shared secret | ### Share[​](#share "Direct link to Share") | Command | Description | | -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | [`vcluster platform share`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_share) | Shares a virtual cluster / namespace with another vCluster platform user or team | | [`vcluster platform share namespace`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_share_namespace) | Shares a vCluster platform namespace with another platform user or team | | [`vcluster platform share vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_share_vcluster) | Shares a virtual cluster with another vCluster platform user or team | ### Sleep[​](#sleep "Direct link to Sleep") | Command | Description | | -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | | [`vcluster platform sleep`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_sleep) | Put a virtual cluster / namespace to sleep | | [`vcluster platform sleep namespace`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_sleep_namespace) | Put a vCluster platform namespace to sleep | | [`vcluster platform sleep vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_sleep_vcluster) | Put a virtual cluster to sleep | ### Wake up[​](#wake-up "Direct link to Wake up") | Command | Description | | ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | | [`vcluster platform wakeup`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_wakeup) | Wake up a virtual cluster / namespace | | [`vcluster platform wakeup namespace`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_wakeup_namespace) | Wakes up a vCluster platform namespace | | [`vcluster platform wakeup vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_wakeup_vcluster) | Wake up a virtual cluster after it was put to sleep. | ### Backup[​](#backup "Direct link to Backup") | Command | Description | | ------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------- | | [`vcluster platform backup`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_backup) | Backup subcommands | | [`vcluster platform backup management`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_backup_management) | Create a vCluster platform management plane backup | ### Reset[​](#reset "Direct link to Reset") | Command | Description | | ------------------------------------------------------------------------------------------------------------------ | ----------------------------- | | [`vcluster platform reset`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_reset) | Reset configuration | | [`vcluster platform reset password`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_reset_password) | Resets the password of a user | ### Destroy[​](#destroy "Direct link to Destroy") | Command | Description | | ---------------------------------------------------------------------------------------------------- | ------------------------------------ | | [`vcluster platform destroy`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_destroy) | Destroy a vCluster Platform instance | --- # Search vCluster docs with Algolia MCP AI coding assistants that support the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) can query the vCluster documentation directly from your editor or chat interface. This uses Algolia's [hosted MCP server](https://www.algolia.com/doc/guides/model-context-protocol/public-mcp), which connects to the same search index that powers the vCluster docs site search. ## Available tools[​](#available-tools "Direct link to Available tools") The MCP server exposes two tools: * **algolia\_search\_vcluster** - Full-text search across vCluster documentation. * **algolia\_search\_for\_facet\_values** - Filter results by facets such as version or section. Access is read-only. No data is modified through this integration. ## Configure your AI tool[​](#configure-your-ai-tool "Direct link to Configure your AI tool") Add the vCluster docs MCP server to your AI assistant of choice. * Claude Code * Claude.ai * Cursor * VS Code Add to your project's `.mcp.json` or global `~/.claude.json`: ``` { "mcpServers": { "vcluster-docs": { "url": "https://mcp.eu.algolia.com/1/VR12oimCRMyH3BKzGU2jVw/mcp" } } } ``` 1. Go to **Settings** > **Integrations** > **Add MCP Server**. 2. Enter the server URL: ``` https://mcp.eu.algolia.com/1/VR12oimCRMyH3BKzGU2jVw/mcp ``` Add to `~/.cursor/mcp.json`: ``` { "mcpServers": { "vcluster-docs": { "url": "https://mcp.eu.algolia.com/1/VR12oimCRMyH3BKzGU2jVw/mcp" } } } ``` Add to `~/.vscode/mcp.json`: ``` { "servers": { "vcluster-docs": { "type": "http", "url": "https://mcp.eu.algolia.com/1/VR12oimCRMyH3BKzGU2jVw/mcp" } } } ``` No API keys or authentication tokens are needed. The server URL is the only configuration required. ## Usage tips[​](#usage-tips "Direct link to Usage tips") Once configured, your AI assistant can search the vCluster docs when answering questions. Most MCP-aware tools call the search automatically based on your prompt. You can also reference the tool directly, for example by asking your assistant to "search the vCluster docs for sleep mode configuration." To search within a specific docs version, ask your assistant to filter by version facet. For example: "Search vCluster docs for pod syncing in version 0.22." --- # Contribute to Platform Docs This guide is for anyone who wants to improve the vCluster and vCluster Platform docs, and it's written for contributors outside vCluster Labs. Everything it describes uses public resources: this repository, the published docs site, and the public [loft-sh/vcluster](https://github.com/loft-sh/vcluster) and [loft-sh/api](https://github.com/loft-sh/api) repositories. A few steps, such as adding backport labels or requesting an AI review, need a maintainer. Ask for them in your PR description. vCluster Labs staff have an internal guide with more detail on tickets, branches, and private previews. This docs website is built using [Docusaurus](https://docusaurus.io/) v3, a modern static website generator. ## Getting started[​](#getting-started "Direct link to Getting started") Fork the [vCluster docs repository](https://github.com/loft-sh/vcluster-docs) and clone your fork locally ``` git clone https://github.com//vcluster-docs ``` then run the following command to install dependencies: ``` npm install ``` This repository requires Node.js 22 or later. The `.nvmrc` file pins the version that CI uses. ### Local development[​](#local-development "Direct link to Local development") Run the following command to start a local development server: ``` npm run start ``` This command opens `http://localhost:3000/docs` in your default browser. Most changes are reflected live without having to restart the server. Note that this command doesn't always catch build errors. ### Build[​](#build "Direct link to Build") After finishing your changes, run the following command to test production build locally: ``` npm run build ``` This command generates static content into the `build` directory and can be served using any static contents hosting service. You can run the following command to serve the built website locally: ``` npm run serve ``` Before making a pull request, it's recommended to run this command to fix any broken links that may have been introduced. A build doesn't catch every kind of broken link, so also see [Checking links before you push](#checking-links-before-you-push). ### Preview a stacked PR[​](#preview-a-stacked-pr "Direct link to Preview a stacked PR") Netlify builds a deploy preview only for PRs whose base branch is `main` or another branch Netlify deploys, such as `next`. A stacked PR, one that targets another PR's branch, gets no preview from Netlify. The `stacked-pr-preview` workflow fills the gap. For a PR whose base isn't `main`, `next`, or a `vcluster-v*` or `platform-v*` branch, it builds the site and deploys a Netlify draft at the alias `pr-`. It then posts the URL, `https://pr---vcluster-docs-site.netlify.app/docs/`, as a PR comment and updates both on every push. PRs from forks get no preview, because GitHub doesn't pass repository secrets to them. A native GitHub stack created with `gh stack` on top of `main` already gets Netlify Deploy Previews, so the workflow skips it. #### Deploy a preview by hand[​](#deploy-a-preview-by-hand "Direct link to Deploy a preview by hand") Use the manual steps when the workflow can't run, for example on a PR from a fork. If you have access to the docs project in Netlify, you can deploy a draft preview yourself with the Netlify CLI. Otherwise, ask someone who has access. The build needs about 11 GB of memory. ``` npx netlify-cli login npx netlify-cli link npx netlify-cli deploy --context deploy-preview --dir=public --alias=pr- ``` `netlify link` prompts you to choose the project. The `deploy` command runs the site build first, so you don't need a separate `netlify build`. The deploy is a draft, so it doesn't change the live site, and the CLI prints its URL when it finishes. Rerun the `deploy` command to update it after you push changes. Never use `main`, `next`, or another deployed branch name as the alias. Netlify serves an alias at the same subdomain as a branch deploy with that name. Omit `--prod`, which publishes to the live site. As with any preview, put `/next/` after the product path to see unreleased changes, for example `/docs/platform/next/...`. ## Where your change goes[​](#where-your-change-goes "Direct link to Where your change goes") Start by finding the page you want to change. Most changes update a page that already exists. Search the docs site, or search this repository for a config key, flag, or resource name. If a feature extends something that's already documented, update that page instead of adding a new one. The site is built from three folders, each with its own sidebar: | Folder | Covers | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `vcluster/` | The cluster vCluster creates: its control plane, worker nodes, `vcluster.yaml`, resource sync, and the `vcluster` CLI. | | `platform/` | vCluster Platform: tenants and projects, templates, users and permissions, apps, fleet operations, Platform install and administration, and the API. vMetal, the machine layer, lives in `platform/vmetal/`. | | `docs/` | Pages that span products, such as the services a provider can offer and end-to-end guides. | Within each folder, sections are organized by what the reader is doing, not by where the code lives. These are the placement questions that come up most: * **vCluster: Deploy, Configure, or Operate?** Deploy (`vcluster/deploy/`) is for creating a cluster or attaching its nodes. Configure (`vcluster/configure/`) is for changing a `vcluster.yaml` setting to turn on or tune a capability. Operate (`vcluster/operate/`) is for actions on a cluster that's already running, such as upgrades and backups. * **Platform: Deliver services to tenants, or Use the platform?** If an admin sets something up for others to consume, such as a template or a permission, it goes in `platform/tenants/`. If a project user works with what they were given, such as creating a cluster from a template, it goes in `platform/use-platform/`. * **vMetal: machine model or driver?** Behavior that's the same for every provisioning driver goes in `platform/vmetal/concepts/`. Anything specific to one driver, such as Metal3 or KubeVirt, goes with that driver in `platform/vmetal/drivers/`. * **A feature that spans products.** Document each part where its owner lives, and link between them, instead of writing the same procedure twice. If you're not sure where a page belongs, say so in the PR description. A maintainer will place it. ### Moving or renaming a page[​](#moving-or-renaming-a-page "Direct link to Moving or renaming a page") A move needs more than a `git mv`. Links into the page, imports, and redirects all have to follow it. The scripts in `hack/move-docs/` move the files, rewrite the links, and write the redirects. Their README lists what they don't catch and the checks to run before you push. Keep each move in its own PR. ### Generated pages[​](#generated-pages "Direct link to Generated pages") Some reference pages are generated from source on each release, so an edit to them is overwritten. Change the source instead, or describe the problem in an issue or PR. | Content | Folder | Generated from | | ----------------------------- | ---------------------------- | ---------------------------------------------------------------------------------------- | | `vcluster.yaml` key reference | `vcluster/_partials/config/` | The schema field descriptions in [loft-sh/vcluster](https://github.com/loft-sh/vcluster) | | vCluster CLI reference | `vcluster/cli/vcluster_*.md` | The command help text in [loft-sh/vcluster](https://github.com/loft-sh/vcluster) | | Platform API reference | `platform/api/` | The Go doc comments in [loft-sh/api](https://github.com/loft-sh/api) | To add prose next to a generated field, see [Adding prose to generated partials](#adding-prose-to-generated-partials). ## Versions and branches[​](#versions-and-branches "Direct link to Versions and branches") ### Versioned and unversioned folders[​](#versioned-and-unversioned-folders "Direct link to Versioned and unversioned folders") | Folder | What it is | Published at | | ---------------------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | `vcluster/`, `platform/` | The docs for the next, unreleased version. Make your changes here. | `/docs/vcluster/next/...` and `/docs/platform/next/...` | | `vcluster_versioned_docs/version-X.Y.Z/` | A frozen copy of each released vCluster version. The newest is the default. | `/docs/vcluster/...` for the newest, `/docs/vcluster/X.Y.Z/...` for older ones | | `platform_versioned_docs/version-X.Y.Z/` | A frozen copy of each released Platform version. | `/docs/platform/...` for the newest, `/docs/platform/X.Y.Z/...` for older ones | | `docs/` | The root docs. Not versioned. | `/docs/...` | At each release, the unversioned folders are copied into a new version folder. Don't edit the versioned folders directly. Change the page in `vcluster/` or `platform/`, and use backport labels to carry the fix into released versions. ### Which branch to target[​](#which-branch-to-target "Direct link to Which branch to target") Open your PR against `main`. Everything merged to `main` appears on the public site right away, under the unreleased `/next/` URLs. The repository also has a `next` branch. It isn't the same as the `/next/` URLs. Maintainers use the `next` branch only for major releases, to prepare docs that must not appear on the public site before the release, such as a large reorganization. Don't target it unless a maintainer asks you to. ### Backport changes to released versions[​](#backport-changes-to-released-versions "Direct link to Backport changes to released versions") To carry a fix into released versions, add a `backport-vX.XX` label to the PR, one per version. After the PR merges, a workflow copies each changed file under `vcluster/` or `platform/` to the same path in that version's folder and opens a backport PR. > \[!WARNING] The copy only works when the page has the same path in the older version. Platform 5.0 and vCluster 0.38 reorganized most of the docs, so a page's current path often doesn't exist in Platform 4.12 or earlier, or vCluster 0.37 or earlier. For those pages, don't add a backport label. Make the change by hand in each older version's folder, at that version's own path, and say so in the PR description. Pages that kept their path still backport normally. This applies until Platform 4.12 and vCluster 0.37 reach end of support. Until Platform 5.0 and vCluster 0.38 are released, the unversioned folders on `main` still use the Platform 4.12 and vCluster 0.37 layout, so backport labels on PRs into `main` work normally. vCluster labels use the minor version, such as `backport-v0.37`. Platform labels include the patch version, such as `backport-v4.12.0`. Only maintainers can add labels, so if you're contributing from outside vCluster Labs, list the versions the fix applies to in your PR description. ## After you open a PR[​](#after-you-open-a-pr "Direct link to After you open a PR") * **Checks.** If you opened the PR from a fork, GitHub waits for a maintainer to approve the workflow runs before any checks start. The checks include the Lint check (see [Style guide automation](#style-guide-automation)), link validators, glossary and feature table checks, and a redirect check when a page moves or is deleted. * **Preview.** Netlify builds a deploy preview and comments with its link. Your changes to `vcluster/` or `platform/` appear in the unreleased version, so add `/next/` after `/docs/vcluster` or `/docs/platform` in the preview URL. * **Review.** A maintainer reviews the PR and may suggest edits for style, placement, or accuracy. ## AI-assisted PR review[​](#ai-assisted-pr-review "Direct link to AI-assisted PR review") Maintainers can request an AI review from Claude by mentioning `@claude` in a PR comment. Only members of the `loft-sh` organization can trigger it, so if you're contributing from outside vCluster Labs, ask a maintainer. Example commands: * `@claude review this PR` - get a focused review * `@claude fact check this page` - check the technical claims against the code Claude reads the PR and posts review comments. It doesn't push changes. The review checks documentation style, validates vCluster YAML configurations, and identifies broken links. The AI review is meant to assist, not replace, human review. ## SEO and crawler policy[​](#seo-and-crawler-policy "Direct link to SEO and crawler policy") The docs site exposes multiple versions (current stable, prior supported, EOS/EOL, and the `next` preview). Only the current stable version should compete for search engine and AI crawler attention; older versions remain reachable for users who need them, but must not outrank or dilute the current docs. See DOC-1325 for background. ### What the current setup does[​](#what-the-current-setup-does "Direct link to What the current setup does") * **`lastVersion`** in `docusaurus.config.js` makes unversioned URLs (for example `/docs/vcluster/deploy/...`) serve the current stable release. Those unversioned URLs are the canonical, indexable entry points. * **noindex on non-current versions** is emitted by `src/theme/DocItem/Layout/index.js`. The swizzle reads `useActivePluginAndVersion()` and injects `` whenever `isLast` is false. That covers EOS, EOL, any prior stable version still in `onlyIncludeVersions`, and the `current` (next/main) preview. * **Sitemap** (`docusaurus.config.js`, sitemap plugin) lists only unversioned URLs. Any path matching `/vcluster//` or `/platform//` — plus the `next` previews — is stripped by both `ignorePatterns` and a post-filter in `createSitemapItems`. * **`llms.txt`** already excludes versioned docs via `includeVersionedDocs: false` on the `docusaurus-plugin-llms-txt` plugin. That stays as-is. * **`static/robots.txt`** is intentionally permissive. `Disallow` would block crawlers from fetching the page and therefore from ever seeing the `noindex` meta tag, which is the opposite of what we want. Note that the production host `vcluster.com/robots.txt` is served by the main site repo, not this one; this file applies to the Netlify preview domain. * **Canonical tags** are not emitted automatically. Only add a `` when the content at a versioned URL is effectively identical to the current version. Do not point version-specific content (release notes, deprecated APIs, migration guides) at the current docs. ### When adding or removing a version[​](#when-adding-or-removing-a-version "Direct link to When adding or removing a version") 1. Update `onlyIncludeVersions` in `docusaurus.config.js` for the affected docs plugin (vcluster or platform). No SEO code changes are required — the swizzle derives `isLast` from the plugin config. 2. When bumping `lastVersion`, the previous stable version automatically starts emitting `noindex,follow`. No per-version list to edit. 3. Regenerate the lifecycle JSON (`npm run generate-lifecycle-json`) so the supported-versions partial stays in sync with the config. ## Style guide[​](#style-guide "Direct link to Style guide") The full style guide is [style-guide.md](https://github.com/loft-sh/vcluster-docs/blob/main/.claude/skills/vcluster-docs-writer/references/style-guide.md), with product names and terminology in [vcluster-terms.md](https://github.com/loft-sh/vcluster-docs/blob/main/.claude/skills/vcluster-docs-writer/references/vcluster-terms.md). Both are part of the docs-writer skill in this repository, so AI tools that work in the repository follow the same rules. For a new feature, task, or troubleshooting page, start from the matching [page template](https://github.com/loft-sh/vcluster-docs/blob/main/.claude/skills/vcluster-docs-writer/references/templates/README.md). The sections below summarize the rules contributors run into most. The Vale linter enforces most of the rules. See the [style guide automation](#style-guide-automation) section for more information. The vCluster docs use the following style guides in this order: 1. [Google developer documentation style guide](https://developers.google.com/style) 2. Kubernetes [Documentation Style Guide](https://kubernetes.io/docs/contribute/style/style-guide/) for Kubernetes terms. Page titles follow **sentence** case. Capitalize Kubernetes objects [according to the K8s style guide](https://kubernetes.io/docs/contribute/style/style-guide/#documentation-formatting-standards). ### Core principles[​](#core-principles "Direct link to Core principles") Here are some core principles to keep in mind when writing documentation: * Active voice creates stronger documentation and helps with translations - write "deploy the configuration" instead of "the configuration was deployed". * Using sentence case in headings makes content more approachable - write "Getting started with providers" instead of "Getting Started with Providers". * Keep sentences under 25 words to improve readability and SEO performance - break down complex ideas into digestible chunks. * Write in present tense - say "this command installs" rather than "this command will install". Docs refer to actions that happen in the present as users read them. * Use contractions (don't, isn't, can't) for better clarity - "don't" stands out more than "do not" when expressing negatives. * Avoid Latin phrases (e.g., i.e., etc.) - choose plain English alternatives that all readers can understand easily. * Never use terms like "easy," "simple," or "obvious" as they can undermine reader confidence and create unnecessary pressure. * Create descriptive link text - instead of "click here," explain where the link leads, such as "view the installation guide". * Spell out acronyms on first use unless they're universally known in technical contexts. ### Consistency and document flow[​](#consistency-and-document-flow "Direct link to Consistency and document flow") When creating new documentation, review similar existing documents to maintain consistent flow and structure. For example, see how platform deployment documentation follows a standard pattern in the [Quick Start Guide](https://www.vcluster.com/docs/platform/install/quick-start-guide). This helps readers navigate documentation intuitively since they know what to expect. Documents of the same kind should use consistent header levels and structure. All installation guides should use the same header hierarchy and naming conventions to help users navigate between different installation options. Docusaurus admonitions should be used consistently throughout the documentation to highlight important information. ``` :::note[Additional context: The `vcluster create` command automatically creates] a new namespace if it doesn't exist. ::: :::tip[Use `vcluster` CLI to quickly deploy a cluster. :::] :::info[The default configuration uses minimal resources suitable for testing.] ::: :::warning[Ensure your Kubernetes cluster has sufficient available memory before] deploying vCluster. ::: :::danger[Do not delete the vCluster pod. :::] ``` ### Oxford comma[​](#oxford-comma "Direct link to Oxford comma") Use the Oxford comma (serial comma) before the last item when listing 3 or more items in a sentence. * **Do Not**: vCluster requires a, b and c. * **Do**: vCluster requires a, b, and c. See Scribbr's [article](https://www.scribbr.com/commas/oxford-comma/) on the Oxford comma for a detailed explanation and examples. ### Version numbers[​](#version-numbers "Direct link to Version numbers") Version numbers for vCluster Labs products carry a lowercase `v` prefix in prose. That covers vCluster, vCluster Platform, vNode, vMetal, and the vCluster CLI. * **Do Not**: Upgrade to vCluster 0.38 before you install Platform 5.0. * **Do**: Upgrade to vCluster v0.38 before you install Platform v5.0. Third-party versions, such as Kubernetes 1.34 or etcd 3.5, follow their upstream style. Code, commands, flags, YAML values, URLs, and paths don't change, so Helm's `--version 0.38.0` stays as it is. The `Loft.version-prefix` Vale rule flags a product name or "version" followed by an unprefixed number. See "Version Numbers" in the [style guide](https://github.com/loft-sh/vcluster-docs/blob/main/.claude/skills/vcluster-docs-writer/references/style-guide.md) for the full rule. ### Code blocks[​](#code-blocks "Direct link to Code blocks") Use `<>` to indicate placeholders in code blocks. For example: ``` kubectl get pods ``` #### Dynamic version tokens[​](#dynamic-version-tokens "Direct link to Dynamic version tokens") Instead of hardcoding version numbers that go stale, use these tokens: | Token | Renders as | | ---------------------------- | ------------------------------------------------ | | `__PLATFORM_VERSION__` | Latest stable Platform patch release, as `X.Y.Z` | | `__VCLUSTER_VERSION__` | Latest stable vCluster patch release, as `X.Y.Z` | | `__PLATFORM_VERSION_MINOR__` | Minor version of that Platform release, as `X.Y` | | `__VCLUSTER_VERSION_MINOR__` | Minor version of that vCluster release, as `X.Y` | ``` export PLATFORM_VERSION=__PLATFORM_VERSION__ helm install vcluster --version __VCLUSTER_VERSION__ ``` The values come from `src/data/latest-versions.json`, which a daily workflow (`sync-latest-versions.yml`) keeps at the latest stable patch of each product's `lastVersion`. Don't edit that file by hand. A remark plugin replaces tokens at build time in text, inline code, code blocks, and string JSX attributes, and `InterpolatedCodeBlock` replaces them in its `code` prop. Both read the same file, so they agree. In versioned docs, tokens resolve to that folder's version instead. A page in `platform_versioned_docs/version-4.11.0/` renders `__PLATFORM_VERSION__` as `4.11.0`, not the latest 4.11 patch. Use tokens only for "the current version". For a fixed minimum, such as "requires vCluster v0.34 or later", write the number. A token would render the current version on every page and in every versioned copy. Tokens render without the `v` prefix, because most of them sit in code. When one appears in prose, write `v__VCLUSTER_VERSION__`. #### Interactive code blocks[​](#interactive-code-blocks "Direct link to Interactive code blocks") For code blocks that contain values users need to customize, use the `InterpolatedCodeBlock` component instead of regular code blocks. This allows users to edit values directly in the documentation. **Using local variables** (user can edit each instance): ``` import InterpolatedCodeBlock from '@site/src/components/InterpolatedCodeBlock'; ``` **Using global variables** (define once, use everywhere on the page): ``` import PageVariables from '@site/src/components/PageVariables'; import InterpolatedCodeBlock from '@site/src/components/InterpolatedCodeBlock'; ``` This eliminates repeated export statements across multiple code blocks. Use `PageVariables` when the same values (like version numbers or registry URLs) appear in multiple code examples on a page. Notes: * Place `PageVariables` before any code blocks that use those variables * Can be placed anywhere on the page (doesn't have to be at the top) * Multiple `PageVariables` components will merge together **Adding titles to code blocks** (optional): The `InterpolatedCodeBlock` component supports an optional `title` attribute to display a descriptive title above the code block: ``` ``` The `title` attribute is completely optional and backwards compatible. Existing code blocks without a title will continue to work as before. #### Icon component[​](#icon-component "Direct link to Icon component") Use the `Icon` component to display checkmarks, crosses or warnings for feature availability, comparison tables, or inline indicators. **Usage:** ``` import Icon from '@site/src/components/Icon'; | Feature | Supported | |---------|-----------| | Connection pooling | | | SSL encryption | | | Legacy auth | | ``` **Inline usage:** ``` Connection pooling is available in all tiers. ``` **With custom tooltips:** ``` ``` **Icon types:** * `type="check"` or `type="checkmark"` - Orange checkmark (light mode), green (dark mode) * `type="cross"` or `type="x"` - Gray X symbol * `type="warning"` or `type="!"` - Yellow ! symbol The component automatically adapts to dark/light theme changes. #### Formatting and variables[​](#formatting-and-variables "Direct link to Formatting and variables") To make code blocks easier to work with, consider adding variables and use [here docs](https://en.wikipedia.org/wiki/Here_document) to make it easier to copy and paste code snippets. Using a variable in a code block: ``` $POD_NAME = pod-name kubectl get pods $POD_NAME ``` Creating a file using a here doc: ``` cat < vcluster.yaml controlPlane: # Deploy etcd instead of using the embedded SQLite backingStore: etcd: deploy: enabled: true statefulSet: highAvailability: replicas: 3 # Deploy vCluster with 3 replicas statefulSet: highAvailability: replicas: 3 EOF ``` Applying a Kubernetes manifest using docs: ``` cat << EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: example-pod labels: app: nginx spec: containers: - name: nginx image: nginx:1.21 ports: - containerPort: 80 EOF ``` > \[!NOTE] here docs support is available in most shells, including bash, zsh, and fish, however in shells other than bash, the syntax may vary. #### Highlight lines of code[​](#highlight-lines-of-code "Direct link to Highlight lines of code") Use [inline comments](https://docusaurus.io/docs/markdown-features/code-blocks#highlighting-with-comments) in the code to highlight lines. ### Links and imports[​](#links-and-imports "Direct link to Links and imports") Link to another page in the same docs section with a relative path to the source file, including the `.mdx` suffix. The suffix is what tells Docusaurus to rewrite the link at build time. Without it, the raw string survives into the compiled bundle and the browser resolves it against the current page URL when clicked, which produces a different path. The server-rendered HTML still looks correct, so this kind of break is invisible to `curl`, `git grep`, and static inspection. It only shows up on click. * Correct: `[Projects](../understand/what-are-projects.mdx)` * Correct, for a directory target: `[Networking](../networking/README.mdx)` * Broken: `[Projects](../understand/what-are-projects)` * Broken: `[Networking](../networking/)` Relative paths also work on GitHub, survive slug changes, and track file moves, so prefer them over `/docs/` URL paths inside a section. Link across sections, such as from Platform to vCluster, with a `/docs/` absolute path. The two sections are separate Docusaurus plugin instances, so relative file paths don't resolve between them. * `[vCluster docs](/docs/vcluster)` * `[Sleep mode](/docs/vcluster/configure/vcluster-yaml/sleep-mode)` #### Links inside partials and fragments[​](#links-inside-partials-and-fragments "Direct link to Links inside partials and fragments") Calculate the path from the partial or fragment file itself, not from a page that imports it. Reusable content gets imported from pages at different depths, so a path that works from one importer breaks from another. Markdown link syntax isn't parsed inside a JSX expression. When reusable content needs to link from within JSX, use the `VersionAwareDocLink` component so the link keeps the reader's docs version. ``` import VersionAwareDocLink from '@site/src/components/VersionAwareDocLink'; security baseline ``` A cross-product link targets the other product's stable docs, because Platform and vCluster version independently. #### Partials and fragments[​](#partials-and-fragments "Direct link to Partials and fragments") The two folder names mean different things: | Folder | Contains | Edit it? | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | `_partials/` | Generated content: the vcluster.yaml reference (`vcluster/_partials/config/`), the Platform API reference (`platform/api/_partials/resources/`), and the Platform CLI index | No. A generator overwrites it. Change the source, as described in [Generated pages](#generated-pages) | | `_fragments/` | Hand-written content that more than one page imports, such as admonitions, shared procedures, and prerequisites | Yes | Put a new reusable snippet in `_fragments/`, in the same product tree as the pages that import it, or in `docs/_fragments/` for root and global content. Don't add a hand-written file to `_partials/`. Many existing hand-written snippets still live in `_partials/` from before this rule. Import them where they are, and don't use them as a model for new files. The "Check partials" CI check enforces this. It fails a PR that adds a new file under `_partials/` outside the generated paths. Editing or moving an existing file passes. #### Import paths[​](#import-paths "Direct link to Import paths") Choose the import path based on what you're importing and whether that content should freeze with a docs version. | Importing | Use | Why | | ----------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | | React or theme components | `@site/src/` or `@theme/` | Not version-specific | | Same-product partials and fragments | a relative path | Versioning copies the docs tree, so each snapshot stays self-contained | | Intentionally global content | `@site/docs/_fragments/` for new content, `@site/docs/_partials/` for existing files | Updates every docs version at once | | Cross-product content | `@site//` | Tracks the other product's current content, which is the only option because the version lines don't match up | A same-product `@site/` import keeps resolving to the live tree even from a versioned page, so an archived page ends up showing current content. CI rejects those. #### Checking links before you push[​](#checking-links-before-you-push "Direct link to Checking links before you push") A production build catches links that point at nothing. It doesn't catch a link that resolves at build time but breaks on click, so run the link checks too. ``` npm run build # links to missing pages npm run validate-mdx-links # click-time breaks in live docs npm run validate-reusable-imports # same-product live imports ``` ### vCluster terms[​](#vcluster-terms "Direct link to vCluster terms") vCluster Labs is the company. Do not use "Loft", "LoftLabs", or "Loft Platform" to refer to vCluster products. "vCluster" is a trademark. There are strict legal frameworks around how to use a trademark, for example, it cannot be used in plural. **Do not use "vClusters"**. Never use vCluster or vClusters when talking about a cluster that vCluster creates. Use **clusters**. Both "virtual cluster" and "tenant cluster" are retired: the Tenant primitive claims the word "tenant" for the customer organization, so the cluster gives it back. ### The hierarchy[​](#the-hierarchy "Direct link to The hierarchy") 1. A **system admin** installs Platform onto a Kubernetes cluster, which becomes the **control plane cluster**. 2. A **platform admin** creates **Tenants**. 3. A **tenant admin** divides the Tenant boundary across **Projects**. 4. A **project user** creates a **cluster**. A **Tenant** is a customer organization, never a cluster. Capitalize it only when naming the API resource, the same way "Project" is. **Tenant isolation** is the term for separation at every layer, from the Tenant boundary in the management plane down to dedicated nodes. Don't coin "cluster isolation", and don't convert existing "tenant isolation" prose. When a sentence needs to say *how* separation happens, name the mechanism: private nodes, vNode, API-level and namespace separation. "Multi-Tenancy" stays only as the name of the licensed feature, on the Platform License page and in `loft-sh/plans`. As a descriptor, "multi-tenancy" and "multitenancy" are retired. Describe what the product does instead. ### Products[​](#products "Direct link to Products") * vCluster: open source project that provisions and manages clusters * vCluster Platform: the management platform and UI for managing clusters and Tenants across one or more control plane clusters * vMetal: infrastructure orchestrator for bare-metal and VM provisioning beneath clusters * vNode: tenant-isolation container runtime "vCluster Pro" is not a product and should not be used. License-gated features are "Enterprise" features, unlocked via a vCluster Platform Enterprise plan (Dev/Prod/Scale), not a separate "Pro" product or tier. The `vcluster-pro` container image, the `loft-sh/vcluster-pro` repository name, and the `sidebar_class_name: pro` frontmatter value are code/engineering identifiers and are unaffected by this rule — that sidebar class renders as an "ENTERPRISE" badge. Name vCluster Platform once per page, then call it "the platform". Keep the formal name on first mention and in install, licensing, support, release, and API contexts. Call the interface "the platform UI", not "vCluster Platform UI". The `Loft.no-vcluster-platform` Vale rule warns when a page repeats the formal name. Use `vcluster.yaml` for all cluster configuration. `values.yaml` only means the Helm values file for installing vCluster Platform. ### CLI[​](#cli "Direct link to CLI") The command line interface name is `vcluster`. ### Abbreviations for Kubernetes distros[​](#abbreviations-for-kubernetes-distros "Direct link to Abbreviations for Kubernetes distros") * [Lightweight Kubernetes](https://k3s.io/): K3s * [Kubernetes](https://kubernetes.io/): K8s * [Zero Friction Kubernetes](https://k0sproject.io/): k0s Note that k0s is the only Kubernetes distro to use a lower case 'k' * [AWS Elastic Kubernetes Service](https://aws.amazon.com/eks/): EKS ### Other product terms[​](#other-product-terms "Direct link to Other product terms") * AWS [EKS](https://aws.amazon.com/eks/) * [CoreDNS](https://coredns.io/) * [etcd](https://etcd.io/) ### Positioning shared nodes[​](#positioning-shared-nodes "Direct link to Positioning shared nodes") Shared nodes are a supported tenancy boundary for trusted tenants, not a security boundary for untrusted tenants with Kubernetes access or arbitrary workload execution. Frame the boundary by trust and tenant access, not by internal versus external. * Recommend shared nodes for trusted tenants such as development, testing, CI/CD, and internal engineering teams. * Route external, resale, regulated, or otherwise untrusted tenant offerings to private nodes, optionally with vNode for runtime isolation. * Don't soften this to "strongly not recommended". Shared-node mitigations such as NetworkPolicy don't replace data-plane isolation for untrusted tenants. * Recommend NetworkPolicy as an added isolation layer for all tenants, including trusted ones. vCluster can create it through `policies.networkPolicy`, and the CNI enforces it. * Frame the risk as an architectural property (shared kernel and nodes) plus a configuration responsibility, not as a vCluster defect. Don't reference specific customers or security incidents. * Reuse the `vcluster/_partials/admonitions/shared-nodes-suitability.mdx` partial rather than rewriting the caveat. There is one exception. A provider serving its own trusted models behind an API (no tenant Kubernetes access) is application-level multitenancy and can serve external customers. See the inference provider production guide. See the shared nodes positioning section in `CLAUDE.md` for the full guidance. ## Style guide automation[​](#style-guide-automation "Direct link to Style guide automation") To maintain quality and consistency in our technical documentation, we use [vale](https://vale.sh/) as a linter to automatically enforce style guidelines. > \[!NOTE] The Lint check runs Vale on pull requests into `main` that change `.md` or `.mdx` files under `platform/`, `vcluster/`, and the versioned docs folders. It posts its findings as review comments on your PR. Only Vale errors on lines your PR adds fail the check. Warnings appear as comments but don't block the merge. Root `docs/` pages and the generated CLI pages aren't checked. > > Fix what Vale flags in the prose. Don't change file paths, commands, or config keys to satisfy it. ### What is Vale?[​](#what-is-vale "Direct link to What is Vale?") [Vale](https://vale.sh/) is an open-source, command-line linter that helps you enforce style and grammar rules in written documentation. It’s highly configurable, allowing you to define custom rules that suit your project’s needs. By integrating it into our CI pipeline, contributors can receive real-time feedback on their documentation during pull requests. ### Running Vale locally[​](#running-vale-locally "Direct link to Running Vale locally") Vale offers multiple installation methods suitable for different operating systems: * **macOS**: `brew install vale` * **Windows**: `choco install vale` * **Linux**: `brew install vale` or use another [package manager](https://repology.org/project/vale/versions) * **Docker**: `docker pull jdkato/vale` Individual files or folders can be linted * To check all files (this may take some time): `vale .` * To check a specific file: `vale path/to/file.mdx` * To check a specific folder including subfolders: `vale /path/to/folder` Running `vale` locally allows us to check our documentation before submitting a pull request. This helps catch style issues early and streamlines the review process. ### Plugins[​](#plugins "Direct link to Plugins") VSCode and Neovim have `vale` plugins that can be installed to lint files as you write them. * VS Code [Vale plugin](https://github.com/errata-ai/vale-vscode). * Neovim setup: * Install [mason.nvim](https://github.com/williamboman/mason.nvim) and add vale\_ls LSP. Configure and use like any other LSP. * Install `vale_ls` on start with `lazy.nvim` ``` return { "williamboman/mason.nvim", optional = true, opts = function(_, opts) if type(opts.ensure_installed) == "table" then vim.list_extend(opts.ensure_installed, { "vale_ls" }) end end, } ``` Run `vale` from the repository root so it picks up the repository's `.vale.ini`. CI uses Vale 3.13.0, so a newer local version can report slightly differently. > \[!NOTE] The repository's `.vale.ini` sets `MinAlertLevel = warning`, so Vale hides suggestions. To see them too, run `vale --minAlertLevel=suggestion `. ### Adding terms to the vocabulary[​](#adding-terms-to-the-vocabulary "Direct link to Adding terms to the vocabulary") If Vale flags a legitimate product or technical term, add it to `.github/styles/config/vocabularies/Loft/accept.txt` in the same PR. Each line is a regular expression. Terms that Vale should always flag go in `reject.txt` in the same folder. Don't add a word only to silence one finding. ### Controlling Vale rules[​](#controlling-vale-rules "Direct link to Controlling Vale rules") Use HTML comments to turn Vale off around a block, in both `.md` and `.mdx` files. The repository's `.vale.ini` treats `.mdx` files as Markdown, so JSX comments such as `{/* vale off */}` don't suppress Vale. ``` This content won't be checked by Vale. ``` ### Suppressing the release drift check[​](#suppressing-the-release-drift-check "Direct link to Suppressing the release drift check") On every vCluster release, `hack/cli-drift` and `hack/config-drift` compare prose code blocks against the regenerated CLI reference and `vcluster.yaml` schema partials, and drift findings open an automated fix PR. Some pages show outdated config or removed commands on purpose: migration guides, before/after comparisons, changelog examples. Opt such a block out by placing a `drift-ignore` comment on the line directly above the opening fence: ```` {/* drift-ignore */} ```yaml # deliberately shows the pre-0.24 syntax exportKubeConfig: secret: name: my-secret ``` ```` In `.md` files use `` instead. The marker applies only to the single fenced block below it. Both scanners honor it, for YAML blocks and for CLI examples alike. Do not use it to silence a finding on current-state docs; fix the drift instead. `InterpolatedCodeBlock` components are scanned the same way as fences: `[[VAR:...]]` placeholders are resolved to their defaults before validation, and the marker on the line directly above the component tag opts it out. A `yaml` block or component titled `vcluster.yaml` is always validated, even when its top-level keys no longer match any current schema root (that is what a block looks like after a release removes the field it documents). ## Adding prose to generated partials[​](#adding-prose-to-generated-partials "Direct link to Adding prose to generated partials") The vCluster config partials under `vcluster/_partials/config/` are overwritten on every release by `hack/vcluster/partials/main.go`. Editing those files in place was the original workflow and the original failure mode: the next regeneration silently dropped the edit. Two routes exist now, both designed to survive regeneration. ### Use the Extras map when prose belongs with a schema field[​](#use-the-extras-map-when-prose-belongs-with-a-schema-field "Direct link to Use the Extras map when prose belongs with a schema field") If the prose is intrinsically tied to one schema path — a security warning that only makes sense alongside `controlPlane.standalone.joinNode`, a migration tip that belongs at the bottom of `controlPlane.distro` — add a `pathExtras` entry in `hack/vcluster/partials/main.go`: ``` var pathExtras = map[string]Extras{ "controlPlane/standalone/joinNode": { Before: "\n:::warning Security consideration\n...\n:::\n\n", }, } ``` `Before` lands above the rendered schema content, `After` below. Links in this prose follow the same rules as any other partial, so use a relative path with the `.mdx` suffix, calculated from where the generated file lands. That depth is the same for live and versioned output. The map keys are validated against the live schema before any output is written; a rename or removal upstream panics the generator with the full list of dangling keys, so the prose can be migrated explicitly instead of silently orphaned. ### Use a fragment when prose stands alone[​](#use-a-fragment-when-prose-stands-alone "Direct link to Use a fragment when prose stands alone") If the prose is conceptual (an overview page, a how-to, a comparison) and only happens to live near generated content, write it as a fragment under `vcluster/_fragments/` and import it where needed. The generator doesn't modify it. This is the right choice when: * The prose can be read on its own without a specific schema field next to it. * The prose references multiple schema paths. * The prose has its own headings, examples, and lifecycle (changes for reasons unrelated to schema bumps). ### Quick rule of thumb[​](#quick-rule-of-thumb "Direct link to Quick rule of thumb") * One schema path with one or two paragraphs of prose → `pathExtras`. * Multi-paragraph prose, multiple references, own structure → a fragment, imported manually. --- # Troubleshoot connected cluster connection failures A [connected cluster](https://vcluster.com/docs/platform/understand/what-are-clusters.md) is a Kubernetes cluster running the vCluster Platform agent, registered with the platform as a `Cluster` resource (`clusters.management.loft.sh`). This runbook covers failures in that agent-to-platform connection, whether a cluster never finishes connecting, an already-connected cluster goes offline or flaps, or an agent gets stuck mid-upgrade. This is a different mechanism from the TSNet tunnel a tenant cluster's own control plane pod uses to reach the platform. If a tenant cluster runs successfully but shows as not ready in the platform UI, see [Resolve vCluster TSNet connection failures](https://vcluster.com/docs/platform/troubleshoot/tsnet-connectivity.md) instead. If the connected cluster is healthy but a request fails with a quota-related `Forbidden` error, see [Troubleshoot quota and admission denials](https://vcluster.com/docs/platform/troubleshoot/quota-admission.md). ## Check the connection status[​](#check-the-connection-status "Direct link to Check the connection status") For how a connected cluster's agent communicates with the platform, see [Platform communication and connectivity](https://vcluster.com/docs/platform/understand/platform_communication.md). Start by reading the `Cluster` resource's status. Run this against a kubectl context for Platform's management API, such as one created with `vcluster platform connect management`, not the connected cluster's own context: Modify the following with your specific values to generate a copyable command: CONNECTED\_CLUSTER\_NAMEmy-connected-cluster ``` kubectl get cluster.management.loft.sh my-connected-cluster \ -o jsonpath='{.status.phase}{"\n"}{.status.reason}{"\n"}{.status.message}{"\n"}' ``` `status.phase` is one of three values: * empty (still initializing) * `Initialized` (the reconcile loop completed a pass without a hard error) * `Failed` (the reconcile loop hit a hard error) There is no separate `Ready` phase, and `Initialized` doesn't necessarily mean healthy. A reconcile pass that finds the agent pod unhealthy still completes without a hard error, so `status.phase` becomes `Initialized` with a non-empty `status.reason` describing the pod problem. Always read `status.reason` alongside `status.phase` rather than treating `Initialized` alone as a clean bill of health. `status.reason` holds a short machine-readable code, and `status.message` carries the underlying error, if any. The tables on this page are keyed off that `reason` value. The platform UI's **Infrastructure > Control Plane Clusters** view also reports an `Online` state for the cluster. That state only reflects reality for clusters connected through the agent's network tunnel (`spec.networkPeer: true`). It has no meaning for a cluster registered with a direct kubeconfig secret (`spec.config`) or for the cluster the platform itself runs on (`spec.local: true`), so don't rely on it in those two cases. ## Where the failure happens[​](#where-the-failure-happens "Direct link to Where the failure happens") 1. **Before the agent ever connects**: the `Cluster` resource stays in its initial, empty phase, or the agent chart itself fails to install. Go to [Failures before the first connection](#failures-before-the-first-connection). 2. **The `Cluster` resource has a `status.reason` set**, whether or not the agent has ever connected, and regardless of whether `status.phase` shows empty, `Initialized`, or `Failed`. Go to [Failures during reconciliation](#failures-during-reconciliation) and match the reason. 3. **After a successful connection**: the cluster was `Initialized`, then goes offline or its connection flaps. `status.phase` can regress back to empty while this happens. Go to [Connection loss after a successful connect](#connection-loss-after-a-successful-connect). 4. **During an agent version upgrade**: the cluster is stuck reporting an upgrade is in progress. Go to [Agent stuck mid-upgrade](#agent-stuck-mid-upgrade). ## Failures before the first connection[​](#failures-before-the-first-connection "Direct link to Failures before the first connection") These happen before the platform ever sees the agent register. `status.phase` stays empty throughout this wait. `status.reason` itself is also blank during the earliest part of the wait, then becomes `Offline` once the platform can confirm there's no online agent tunnel yet. Either is expected briefly for a new cluster; the chart install itself never completing is the case to check for if it persists. Check the agent pod directly. This targets the connected cluster's own API server, not Platform's, so switch `kubectl` context (or add `--context`) before running it: Modify the following with your specific values to generate a copyable command: AGENT\_NAMESPACEvcluster-platform ``` kubectl get pods -n vcluster-platform -l app=loft kubectl logs -n vcluster-platform deploy/loft ``` The agent namespace defaults to `vcluster-platform` (older installs used `loft`). Confirm it with `spec.managementNamespace` on the `Cluster` resource if you're not sure. | Signal | Cause | Resolution | | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Agent logs report the coordinator is unreachable | The agent probes the platform's DERP relay endpoint over HTTPS before starting its tunnel, and logs a message like "Failed to reach the coordinator server. Make sure that the agent can reach the control-plane" when the probe fails. Egress to the platform host is blocked, or a proxy or firewall is dropping the connection. | Confirm outbound HTTPS access from the connected cluster to the platform host. If a firewall, egress policy, or corporate proxy sits in the path (common on managed Kubernetes services), allow outbound HTTPS to the platform's hostname. | | Agent logs report `token is empty` or `url is empty` | The Helm values the install command set (`token`, `url`) never reached the agent, usually because the generated connect command was edited, truncated, or applied against the wrong values file. | Regenerate the connect command from the platform UI or CLI and reapply it without editing the `token` or `url` values. | | Registration fails with `no auth key provided` | Same root cause as the previous row, but seen from the platform's side instead of the agent's. The token value was empty at registration time. | Regenerate and reapply the connect command, same as the previous row. | | Registration fails on an invalid access key | The access key was deleted, disabled, or expired on the platform after being issued, but before, or while, the agent tried to use it. | Regenerating the connect command reuses the existing access key unless its scope is malformed, so a disabled or expired key comes back unchanged. Re-enable the access key, or delete it so the platform creates a fresh one on the next connect command, then reapply the command. | | `Cluster` creation is rejected for missing a connection mode | The `Cluster` object was created directly (for example, applied by hand or by automation) without `spec.config`, `spec.local`, or `spec.networkPeer` set. | Use the platform UI, CLI, or the generated Helm command to create the `Cluster` object, which sets `spec.networkPeer: true` for agent-based connections. | | `Cluster` creation is rejected for a second `spec.local` cluster | Something tried to mark a second cluster as the one the platform itself runs on, and only one `Cluster` object may have `spec.local: true`. | Remove `spec.local` from the new object. Use `spec.networkPeer: true` instead. | | `Cluster` creation is rejected for exceeding the license limit | Connecting this cluster would exceed the license's connected-cluster limit. | Disconnect an unused cluster, or contact vCluster Labs to adjust the license. | To check outbound HTTPS access directly, rather than waiting for the agent to log a failure, run this from a pod in the connected cluster: Modify the following with your specific values to generate a copyable command: PLATFORM\_HOSTplatform.example.com ``` kubectl run curl-test --rm -it --restart=Never --image=curlimages/curl -- \ curl -v https://platform.example.com/derp/probe ``` A `200` response confirms the path is open, matching what the agent itself expects. A timeout, connection reset, or non-200 response points to a firewall, egress policy, or corporate proxy blocking or intercepting the path. ## Failures during reconciliation[​](#failures-during-reconciliation "Direct link to Failures during reconciliation") This table covers the `status.reason` values you're most likely to see. Most describe a step that only runs after the agent has connected at least once, but `Offline` and `LocalClient` are checked on every pass and can appear before the first connection too. `status.phase` splits three ways depending on the reason: `PeerHealthCheck`, `InitializeCluster`, `DeployProxyOnlyAgent`, `ClusterUID`, `HealthCheckCluster`, and `LocalClient` come from a hard error, so they set `status.phase` to `Failed`. `Offline` and `NoOnlinePeers` requeue for a retry instead, which leaves `status.phase` empty, the same as before a first connection. The pod-health reasons below complete reconciliation without an error, so they set `status.phase` to `Initialized`. | `status.reason` | Cause | Resolution | | ------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Offline` | No online agent tunnel and no direct kubeconfig secret configured. This is expected briefly while a new cluster waits for its first connection. It's only a problem if it persists. | If this persists, work through [Failures before the first connection](#failures-before-the-first-connection) above. | | `NoOnlinePeers` | The agent registered at least once, but none of its tunnel sessions currently respond. | The agent pod likely crashed or restarted after registering. Check the agent pod's status and logs in its namespace. | | `PeerHealthCheck` | The platform hit an error health-checking the connected cluster's agent tunnel sessions. | Check the vCluster Platform pod's own logs. This is a platform-side error, not an agent-side one. | | `InitializeCluster` | Covers several distinct steps that all share this reason: building a client to reach the connected cluster, creating Platform's management namespace on it, installing or upgrading the agent's CRDs and Helm release, and installing its RBAC roles. `status.message` is the only way to tell which step actually failed. | Read `status.message` first to identify the failing step. A client-build failure usually means the connected cluster's API server is unreachable or its credentials are invalid. A namespace, CRD, Helm, or RBAC failure usually means a permissions problem, often a Kubernetes `Forbidden` response. For a direct-kubeconfig cluster (`spec.config`), the client-build case is the credential the platform itself holds. | | `DeployProxyOnlyAgent` | Deploying or refreshing the temporary bridge agent used during an agent version upgrade hit an actual error, for example building or creating the bridge deployment, or listing agent tunnel sessions. This is different from the bridge simply waiting to come online, which doesn't set a reason. | See [Agent stuck mid-upgrade](#agent-stuck-mid-upgrade). | | `ClusterUID` | The token installed on this cluster was copied from a different cluster's install command, or the same physical cluster was connected twice under two different `Cluster` names. The platform reads the target's `kube-system` namespace UID once it has a working connection to the cluster, and refuses to let two `Cluster` objects share one underlying cluster. For an agent-connected cluster, this can only be detected once the tunnel is online, so it never appears before the first connection. | Use a unique, freshly generated connect command per physical cluster. If the same cluster was connected twice by accident, delete the duplicate `Cluster` object rather than reusing its token. | | `HealthCheckCluster` | The platform failed to even query the agent pod's status, for example an error listing pods in the agent namespace. | Read `status.message` for the underlying error, then check whether the platform can still reach the connected cluster's API. | | A pod condition such as `NotFound`, `Pending`, `Terminating`, `NotReady`, `CrashLoopBackOff`, or `ImagePullBackOff` | The agent pod health check itself succeeded but found the pod unhealthy, and that condition is reported directly as `status.reason`. `status.phase` still shows `Initialized` in this case. Checking the pod condition through `status.reason` is what tells you it's not actually healthy. | Treat this as a standard pod health issue. Check events and logs for the agent pod in its namespace, and resolve the underlying scheduling, image, or crash cause. | | `LocalClient` | The platform's own local tailscale control-plane client, used to reach the agent tunnel mesh, is unavailable. This only surfaces for regular connected clusters (`spec.local: false`); for the cluster the platform itself runs on, the same failure is retried silently instead. | Check the vCluster Platform pod's own health and logs. This is a platform-side problem, not something wrong with the connected cluster. | ## Connection loss after a successful connect[​](#connection-loss-after-a-successful-connect "Direct link to Connection loss after a successful connect") A connected cluster that already reported `Initialized` can still go offline or flap. Existing tenant clusters keep running through this. The platform just loses management, monitoring, and proxying until the connection returns. When this happens, the reconcile loop also resets the `Cluster` resource's `status.phase` back to empty and sets `status.reason` to `Offline` or `NoOnlinePeers`, the same reasons covered in [Failures during reconciliation](#failures-during-reconciliation) above. | Signal | Cause | Resolution | | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The cluster repeatedly flips online and offline in the platform UI | The agent's own liveness loop is failing its periodic check every 10 seconds and logs a message like "Agent cannot reach platform" with an increasing failure counter. | A single occurrence is a transient blip. Repeated occurrences over time point to an unstable network path (intermittent egress, a proxy doing TLS inspection, or a flaky load balancer in front of the platform) rather than a one-time failure. Escalate to whoever owns network egress for the cluster. | | The cluster shows offline, but the agent pod looks healthy | The platform marks a connection offline if the internal platform component that owned that session stops reporting for about two minutes, independent of the agent's own health. | Check the vCluster Platform pods themselves, not just the agent, especially after a platform restart or rollout. | | A connected cluster that was offline for a long time reconnects, but tenant clusters on it briefly show as unmanaged before recovering | The platform garbage-collects its internal tracking of an idle connection after about 15 minutes of no activity, then recreates it once the agent reconnects. | No action needed. This is expected after an extended network partition. The platform resumes full management once the new connection is established. | ## Agent stuck mid-upgrade[​](#agent-stuck-mid-upgrade "Direct link to Agent stuck mid-upgrade") When the platform upgrades an agent, it first deploys a temporary bridge deployment, `proxy-only-agent`, so the connected cluster's API server stays reachable while the primary agent restarts. The common stall, the bridge waiting for its own tunnel session to register or come online, requeues silently without writing a new status. `status.reason` and `status.phase` just keep whatever they showed before the upgrade started, so don't expect them to read as empty. A lingering `proxy-only-agent` deployment in the agent namespace is the real signal to check for, regardless of what the `Cluster` resource's status currently shows. As with the pod check above, this runs against the connected cluster's own context: Modify the following with your specific values to generate a copyable command: AGENT\_NAMESPACEvcluster-platform ``` kubectl get deployments -n vcluster-platform loft proxy-only-agent ``` | Signal | Cause | Resolution | | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `proxy-only-agent` deployment exists, but `status.reason` never changes | This is the common, non-error stall described above. | Give it a few minutes. If it doesn't clear, check the bridge deployment's own pod status and the agent tunnel path in [Enable verbose agent and platform logs](#enable-verbose-agent-and-platform-logs). | | `proxy-only-agent` deployment never becomes ready | The bridge deployment reuses the primary agent's service account and image, so it fails for the same reasons the primary agent would. Common causes are insufficient resources in the namespace or a service account missing required permissions. | Check `proxy-only-agent`'s pod events and logs the same way you would the primary agent. | | `status.reason` is `DeployProxyOnlyAgent` | Deploying or refreshing the bridge hit an actual error, rather than just waiting for it to come online. | Read `status.message` for the underlying error and resolve it directly. | | The upgrade seems to loop indefinitely, repeatedly trying to reinstall the agent | The agent's chart version must match the platform's own version exactly, not just meet a minimum. A stuck or partially applied Helm release, or drift in the agent's Extra Agent Values from what the platform expects, can also make every reconcile decide an upgrade is still needed. | Check the agent's Helm release history and Extra Agent Values in its namespace for a stuck `pending-upgrade` state or drifted values, and avoid manually editing the chart version or Extra Agent Values the platform manages. Editing `token` or `url` doesn't cause this particular loop. It breaks agent authentication or connectivity instead, covered in the rows above. | | The upgrade loops specifically with errors about a Helm release that doesn't exist | `spec.managementNamespace` on the `Cluster` object doesn't match the namespace the agent Helm release actually lives in. The platform automatically checks both the `loft` and `vcluster-platform` defaults for a legacy install, but doesn't fall back for any other custom namespace name. | Update `spec.managementNamespace` to match where the agent deployment actually runs. | ## Enable verbose agent and platform logs[​](#enable-verbose-agent-and-platform-logs "Direct link to Enable verbose agent and platform logs") If the tables above don't explain what you're seeing, turn on verbose TSNet logging. The platform and the agent each read their own environment, so set the variable on whichever side you're debugging. For the platform side, add to `values.yaml`: Platform values.yaml ``` env: LOFT_LOG_TSNET_CONTROL_PLANE: true logging: level: debug ``` For the agent side, go to **Infrastructure > Control Plane Clusters**, click **Edit** for the cluster, open the **Agent** tab, and add under **Extra Agent Values**. See [Set agent values using the UI](https://vcluster.com/docs/platform/administer/clusters/advanced/agent-config.md#set-agent-values-using-the-ui). Agent extra values ``` env: LOFT_LOG_TSNET_AGENT: true ``` `LOFT_LOG_TSNET_CONTROL_PLANE` turns on verbose logging for the platform's TSNet control-plane component, and for its DERP relay and mesh traffic, since all three share the same toggle. `LOFT_LOG_TSNET_AGENT` turns on the equivalent verbose logging for the agent's own TSNet client. ## Prevent repeat failures[​](#prevent-repeat-failures "Direct link to Prevent repeat failures") ### Generate a fresh connect command per physical cluster[​](#generate-a-fresh-connect-command-per-physical-cluster "Direct link to Generate a fresh connect command per physical cluster") Never copy a connect command, Helm values file, or agent secret from one cluster's install to another. Each connected cluster's access token is tied to that specific cluster's identity, and reusing one is the most common cause of the duplicate or mismatched `ClusterUID` failures described above. ### Verify egress before installing the agent[​](#verify-egress-before-installing-the-agent "Direct link to Verify egress before installing the agent") Confirm outbound HTTPS access to the platform host from the target cluster before running the install command, using the same check described in [Failures before the first connection](#failures-before-the-first-connection), particularly on managed Kubernetes services or behind a corporate proxy. Diagnosing a blocked connection after installation takes longer than checking it up front. ### Treat agent-managed Helm values as read-only[​](#treat-agent-managed-helm-values-as-read-only "Direct link to Treat agent-managed Helm values as read-only") Don't hand-edit the `token`, `url`, chart version, or Extra Agent Values the platform sets on the agent release. Drift in the chart version or Extra Agent Values is a common cause of upgrade loops. Drift in `token` or `url` breaks agent authentication or connectivity instead, without necessarily triggering a reinstall. ### Keep `managementNamespace` in sync after manual namespace changes[​](#keep-managementnamespace-in-sync-after-manual-namespace-changes "Direct link to keep-managementnamespace-in-sync-after-manual-namespace-changes") If you move the agent to a different namespace outside of the platform's own upgrade flow, update `spec.managementNamespace` on the `Cluster` object to match. A mismatch here surfaces as a confusing upgrade loop rather than a clear error. --- # Troubleshoot Platform Database The platform database is a local PostgreSQL instance provisioned only in air-gapped environments. For background on what it is and how it is configured, see [Platform Database](https://vcluster.com/docs/platform/install/air-gapped/platform-database.md). If the platform UI shows an **Action Required: Usage History Database Down** banner, the postgres pod is unhealthy. Start with the diagnostics below. ## Diagnose a postgres pod that fails to start[​](#diagnose-a-postgres-pod-that-fails-to-start "Direct link to Diagnose a postgres pod that fails to start") If `kubectl describe cluster.storage.loft.sh ` shows `EmbeddedPostgresAvailable=False`, the postgres pod is not healthy. Check the pod status and events in the platform namespace: ``` kubectl get pods -n -l loft.sh/system-app=platform-db kubectl describe pod -n -l loft.sh/system-app=platform-db ``` Common causes are storage provisioning failures (the PVC cannot bind to a PersistentVolume) and image pull errors. Resolve the underlying pod issue and the condition should clear on the next reconcile. ## Repeated "error sending usage snapshot" in logs[​](#repeated-error-sending-usage-snapshot-in-logs "Direct link to Repeated \"error sending usage snapshot\" in logs") This error appears every 15 minutes when the platform cannot reach the database. It does not affect platform operation, but usage data is not recorded during the outage. Investigate the postgres pod health as described above. ## Recover from a deleted PVC[​](#recover-from-a-deleted-pvc "Direct link to Recover from a deleted PVC") The PersistentVolumeClaim named `platform-db` backs the platform database and should never be deleted. Deleting it permanently destroys any usage snapshots that have not yet been transmitted to the vCluster license server. Those snapshots cannot be reconstructed, which creates a gap in the billing record for that period. Snapshots already transmitted to the license server are not affected. In the case that the PVC is deleted, follow these steps to return the platform database to a working state: 1. Set $PLATFORM\_NS, $PLATFORM\_NAME, and $CLUSTER\_NAME based on the name and namespace of the vCluster Platform Deployment. ``` export PLATFORM_NS= export PLATFORM_NAME= export CLUSTER_NAME= ``` 2. Get current vCluster Platform replicas. ``` export DESIRED_REPLICAS=$(kubectl get deployment -n $PLATFORM_NS $PLATFORM_NAME -o json | jq -r '.spec.replicas') ``` 3. Scale down vCluster Platform Deployment to 0. ``` kubectl scale deployment -n $PLATFORM_NS $PLATFORM_NAME --replicas 0 ``` 4. Edit the cluster that vCluster Platform is installed in and change the label `loft.sh/platform-db-applied-hash` to a different value, for example add a random letter to the string. Deleting the label will also work. ``` kubectl edit cluster.storage.loft.sh $CLUSTER_NAME ``` 5. Scale up the platform Deployment to its original replica count. ``` kubectl scale deployment -n $PLATFORM_NS $PLATFORM_NAME --replicas $DESIRED_REPLICAS ``` 6. Confirm that vCluster Platform database becomes available. ``` kubectl rollout status deployment/$PLATFORM_NAME -n $PLATFORM_NS ``` --- # Troubleshoot quota and admission denials Quota errors can come from two different enforcement paths. A **management denial** comes from the vCluster Platform management API before it accepts a platform-managed resource, such as a tenant cluster, namespace, or Machine. A **Kubernetes admission denial** comes from the connected cluster after Kubernetes admission evaluates a resource, such as a pod, service, persistent volume claim (PVC), CPU, memory, or storage request. Use this runbook when a request fails with a quota or admission error, or when a tenant reports that a workload cannot be created even though the project looks healthy. First, use the error message to identify the denial type. Then follow the matching diagnostic path. ## Common error messages[​](#common-error-messages "Direct link to Common error messages") Start by finding the part of the error that names the quota scope and resource expression. That tells you whether the request failed against the whole project, a per-user limit, a per-team limit, or a generated ClusterQuota on a connected cluster. ### Management denials[​](#management-denials "Direct link to Management denials") Management denials come from the vCluster Platform management API before the requested management resource is accepted. They apply to platform-owned objects, such as tenant cluster instances, namespace instances, and Machines. These denials answer the question: "Can this project, user, or team create or wake another platform-managed resource?" Management resource quota denials follow this pattern: ``` metadata.name: Forbidden: cannot create virtualclusterinstance, because project virtualclusterinstances limit of 5 would be exceeded metadata.name: Forbidden: cannot create spaceinstance, because per-user spaceinstances.active limit of 2 would be exceeded metadata.name: Forbidden: cannot create nodeclaim, because per-team nodeclaims.provider=metal limit of 10 would be exceeded metadata.name: Forbidden: cannot update virtualclusterinstance, because project virtualclusterinstances.active limit of 3 would be exceeded ``` Read these messages as follows: | Message part | Meaning | | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cannot create` or `cannot update` | The operation that vCluster Platform evaluated. Wake-up requests appear as updates because the platform changes the sleeping state. | | `virtualclusterinstance`, `spaceinstance`, or `nodeclaim` | The management resource that consumes quota. Tenant clusters consume `virtualclusterinstances`, namespaces consume `spaceinstances`, and Machines consume `nodeclaims`. | | `project`, `per-user`, or `per-team` | The quota scope that blocked the request. Per-user/team quotas apply to each owner separately. | | `virtualclusterinstances.active`, `spaceinstances.active`, or `nodeclaims.provider=` | The specific quota expression that matched the denied resource. Expressions can include filters such as active state, template, or node provider. | | `limit of 5 would be exceeded` | The request would put usage above the configured limit. The existing usage might already equal or exceed the limit. | ### Kubernetes admission denials[​](#kubernetes-admission-denials "Direct link to Kubernetes admission denials") Kubernetes admission denials come from the connected cluster after a request reaches Kubernetes admission. They apply to Kubernetes resources created in the namespace that hosts a tenant cluster or namespace instance, such as pods, services, PVCs, CPU, memory, and storage. vCluster Platform creates ClusterQuota objects on the connected cluster so Kubernetes can enforce the project's quota there. Kubernetes admission quota denials usually come from a generated ClusterQuota on the connected cluster: ``` Forbidden: exceeded quota: project-quota-my-project, requested: requests.cpu=1, used: requests.cpu=9, limited: requests.cpu=10 Forbidden: exceeded quota: user-quota-alice-my-project, requested: count/pods=1, used: count/pods=20, limited: count/pods=20 Forbidden: status unknown for quota: project-quota-my-project, resources: requests.memory Forbidden: insufficient quota to consume: requests.cpu Forbidden: failed quota: project-quota-my-project: must specify requests.cpu,limits.cpu Forbidden: caches not synchronized ``` Read these messages as follows: | Message part | Meaning | | -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | `project-quota-...` | A project-wide ClusterQuota denied the request. | | `user-quota-...` or `team-quota-...` | A per-user or per-team ClusterQuota denied the request. | | `requested`, `used`, and `limited` | The denied request, current usage, and hard limit for the resource that exceeded quota. | | `requests.cpu`, `limits.memory`, `requests.storage`, or `count/pods` | The Kubernetes resource expression that admission evaluated. | | `status unknown for quota` | The quota exists, but admission doesn't have current usage for one or more tracked resources. | | `insufficient quota to consume` | The resource requires a covering quota, but no matching quota covers this request. | | `must specify requests.*` or `must specify limits.*` | The workload is missing resource requests or limits required by the quota. | | `caches not synchronized` | The platform agent admission cache wasn't ready when the request arrived. | If the response only says `Forbidden`, check Kubernetes events or platform agent logs for the full admission message. ## Identify the actual problem[​](#identify-the-actual-problem "Direct link to Identify the actual problem") Select the tab for the type of denial you are seeing, then follow the steps to identify the problem. Commands that inspect projects and management resources run against the vCluster Platform management API. Commands that inspect namespaces or ClusterQuota objects run against the connected cluster where the denied Kubernetes resource was created. Make sure to run the commands against the correct Kubernetes context. * Management denial * Kubernetes admission denial 1. **Retrieve the denied management resource** Use the resource type from the denial message to retrieve the denied management object. For tenant clusters and namespaces, the namespace where the management object lives is the project namespace. ``` kubectl get virtualclusterinstances.management.loft.sh -A kubectl get spaceinstances.management.loft.sh -A kubectl get nodeclaims.management.loft.sh -A ``` 2. **Inspect the resource context** Inspect the object that failed. Check the owner, template, active state, provider, and target cluster. ``` kubectl -n loft-p-my-project get virtualclusterinstance my-vcluster -o yaml kubectl -n loft-p-my-project get spaceinstance my-space -o yaml kubectl -n loft-p-my-project get nodeclaim my-machine -o yaml ``` Use these fields to determine which limits can apply: | Field or state | Why it matters | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | Project namespace, such as `loft-p-my-project` | Identifies the project that owns the resource. | | `spec.owner.user` or `spec.owner.team` | Identifies whether a per-user or per-team quota can apply. Team-owned resources count against the team, not each member. | | `spec.templateRef.name` | Matches template-specific quota expressions such as `virtualclusterinstances.template=`. | | Sleeping state | Matches `.active` quota expressions. A wake-up request can fail when active usage is already at the limit. | | `spec.providerRef` on a Machine | Matches provider-specific Machine expressions such as `nodeclaims.provider=`. | | Target cluster | Identifies where to inspect generated ClusterQuota objects for Kubernetes resource denials. | 3. **Match the project or owner quota** Inspect the project quota limits and usage: ``` kubectl get projects.management.loft.sh my-project -o yaml ``` Look under `spec.quotas` for configured limits. Look under `status.quotas` for current usage. Match the expression from the error message, such as `virtualclusterinstances.active`, `spaceinstances.template=`, or `nodeclaims.provider=metal`. 1) **Identify the connected-cluster namespace** Start with the namespace where the denied Kubernetes resource was created. If the denial happened from inside a tenant cluster or namespace instance, get the connected-cluster namespace from the instance's `spec.clusterRef.namespace` field. ``` kubectl -n loft-p-my-project get virtualclusterinstance my-vcluster \ -o jsonpath='{.spec.clusterRef.namespace}{"\n"}' kubectl -n loft-p-my-project get spaceinstance my-space \ -o jsonpath='{.spec.clusterRef.namespace}{"\n"}' ``` Use that value as `` in the connected-cluster commands: ``` kubectl get namespace -o yaml ``` 2) **Check project and owner labels** Inspect the namespace labels on the connected cluster. ``` kubectl get namespace \ -o jsonpath='{.metadata.labels.loft\.sh/project}{"\n"}{.metadata.labels.loft\.sh/user}{"\n"}{.metadata.labels.loft\.sh/team}{"\n"}' ``` The `loft.sh/project`, `loft.sh/user`, and `loft.sh/team` labels determine which project, per-user, or per-team ClusterQuota objects apply. 3) **Match the project quota limit and usage** Inspect the project quota limits and usage: ``` kubectl get projects.management.loft.sh my-project -o yaml ``` Look under `spec.quotas` for configured limits. Look under `status.quotas` for current usage. Match the expression from the error message, such as `requests.cpu`, `limits.memory`, `requests.storage`, or `count/pods`. 4) **Inspect generated ClusterQuota objects** For connected-cluster admission denials, inspect generated ClusterQuota objects on the cluster where the denied resource was created. ``` kubectl get clusterquotas.storage.loft.sh \ -l loft.sh/project-cluster-quota=my-project kubectl get clusterquotas.storage.loft.sh \ -l loft.sh/project-user-cluster-quota=my-project kubectl get clusterquota.storage.loft.sh project-quota-my-project -o yaml ``` Project-wide ClusterQuota objects use the `loft.sh/project-cluster-quota` label. Per-user/team ClusterQuota objects use the `loft.sh/project-user-cluster-quota` label. They also include either `loft.sh/user` or `loft.sh/team`. ## Resolve the denial[​](#resolve-the-denial "Direct link to Resolve the denial") Choose the resolution that matches the problem you found: | Problem | Resolution | | --------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | Expected usage reached the project limit | Increase the project quota, or delete unused resources until `used` is below `limited`. | | One user or team reached their owner limit | Increase the per-user/team quota, move incorrectly owned resources to the right owner, or delete resources owned by that user or team. | | An `.active` quota blocked a wake-up request | Increase the active-resource quota, or put other inactive tenant clusters or namespaces to sleep before retrying the wake-up. | | A template-specific quota blocked creation | Use a different allowed template, increase the template-specific quota, or delete unused resources created from that template. | | A `nodeclaims.provider=` quota blocked a Machine | Use a different allowed node provider, increase the provider-specific Machine quota, or delete unused Machines from that provider. | | A Kubernetes resource quota exceeded `requests.*`, `limits.*`, `count/*`, storage, PVC, or service limits | Increase the relevant project or owner quota, or reduce workloads that consume that resource on the connected cluster. | | The error says `must specify requests.*` or `must specify limits.*` | Add the missing resource requests or limits to the workload. Kubernetes quota admission requires those fields when a quota tracks them. | | The error says `status unknown for quota` | Wait for quota status to reconcile, then retry. If the error persists, inspect platform agent logs and the generated ClusterQuota status. | | The error says `insufficient quota to consume` | Add or adjust the project quota so a matching ClusterQuota covers the requested resource. | | The error says `caches not synchronized` | Retry after the platform agent finishes startup. If it persists, check the platform agent pod and logs on the connected cluster. | After changing a quota, verify both the project status and generated ClusterQuota status: ``` kubectl get project.management.loft.sh my-project -o jsonpath='{.status.quotas}' kubectl get clusterquota.storage.loft.sh project-quota-my-project -o jsonpath='{.status.total}' ``` For persistent `caches not synchronized` errors, find the platform agent namespace and inspect the agent logs on the connected cluster: ``` kubectl get cluster.management.loft.sh \ -o jsonpath='{.spec.managementNamespace}{"\n"}' kubectl get pods -A -l app=loft kubectl -n logs deploy/loft ``` ## Prevent repeat failures[​](#prevent-repeat-failures "Direct link to Prevent repeat failures") ### Set limits above expected peak usage[​](#set-limits-above-expected-peak-usage "Direct link to Set limits above expected peak usage") Set quota limits above normal peak usage when the project is expected to grow. For per-user/team quotas, leave enough headroom for one user's normal burst without allowing one owner to consume the whole project quota. ### Check current usage before lowering limits[​](#check-current-usage-before-lowering-limits "Direct link to Check current usage before lowering limits") Before lowering limits, compare the new value with current `status.quotas` usage. vCluster Platform doesn't delete existing resources when a quota is reduced. The project can remain over quota until usage is cleaned up. ### Leave headroom for multi-cluster projects[​](#leave-headroom-for-multi-cluster-projects "Direct link to Leave headroom for multi-cluster projects") For multi-cluster projects, avoid setting generic Kubernetes resource quotas exactly at current usage. vCluster Platform splits these quotas across connected clusters from reported usage on the other clusters. Simultaneous resource creation on multiple clusters can briefly exceed the intended aggregate quota. ### Add tenant-cluster storage guardrails[​](#add-tenant-cluster-storage-guardrails "Direct link to Add tenant-cluster storage guardrails") For storage limits inside tenant clusters, combine project quotas with per-tenant-cluster ResourceQuota objects or policy enforcement. See [Enforce storage quotas per tenant cluster](https://vcluster.com/docs/platform/how-to/enforce-storage-quotas.md). --- # Troubleshoot Stacks Start here: 1. Run the status command below and note the aggregate `phase`, the Ready condition's `reason`, and each task's phase. 2. Follow the path for the aggregate phase: * For `Pending` or `Degraded`, start with [Resolve top-level reasons](#resolve-top-level-reasons). * For `Progressing`, inspect the task that is `Pending`, `Waiting`, or `Progressing` under [Diagnose a task](#diagnose-a-task). * For `Deleting`, wait for deletion to finish. If it doesn't, inspect the remaining owned applications and their deletion status. * For `Healthy`, use [Diagnose outputs](#diagnose-outputs) or the integration-specific checks if the deployed workload still isn't working. 3. Apply the resolution, then re-run the status command. `Healthy` means the Stack's own orchestration succeeded, not that everything it deployed is fully working. For some integrations, confirm the deployed workload's own readiness too. See [Troubleshoot NVIDIA Run:ai certified stacks](#troubleshoot-nvidia-runai-certified-stacks) for an example. ## Collect stack status[​](#collect-stack-status "Direct link to Collect stack status") Connect to the vCluster Platform management API: ``` vcluster platform connect management ``` List StackInstances and inspect one instance: ``` kubectl get stackinstances -A kubectl describe stackinstance -n kubectl get stackinstance -n -o yaml ``` Print a compact summary, including the Ready condition's reason and message: ``` kubectl get stackinstance -n \ -o jsonpath='{.status.phase}{"\n"}{range .status.conditions[?(@.type=="Ready")]}{.reason}{"\t"}{.message}{"\n"}{end}{range .status.tasks[*]}{.name}{"\t"}{.type}{"\t"}{.phase}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}' ``` Compare `status.observedGeneration` with `metadata.generation`. A lower observed generation means the controller hasn't reconciled the latest specification yet. ### Understand phases[​](#understand-phases "Direct link to Understand phases") The StackInstance reports one aggregate phase across all of its tasks: | StackInstance phase | Meaning | | ------------------- | --------------------------------------------------------- | | `Pending` | A missing or invalid prerequisite requires manual action. | | `Progressing` | At least one task is still rolling out. | | `Healthy` | Every resolved task is healthy. | | `Degraded` | At least one task failed. | | `Deleting` | The controller is removing owned children. | Each task also reports its own phase, which the aggregate phase summarizes: | Task phase | Meaning | | ------------- | --------------------------------------------------------------------------- | | `Pending` | The task hasn't materialized a child yet. | | `Waiting` | The task is waiting for dependencies. | | `Progressing` | Its AppInstance or ArgoCDApplication is rolling out. | | `Blocked` | A prerequisite needs manual action. | | `Healthy` | The child reached readiness. | | `Failed` | The controller, generated child, or task timeout produced a terminal error. | ## Resolve top-level reasons[​](#resolve-top-level-reasons "Direct link to Resolve top-level reasons") A `Degraded` or `Pending` StackInstance carries a reason on its Ready condition. Where that reason comes from differs, so start with the `Degraded` row or the instance-level table below, and only treat a reason as a blocked task's own reason (further down) once you've ruled those out. | Signal | Cause | Resolution | | ---------- | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | `Degraded` | At least one task failed. This reason is always exactly `Degraded`, regardless of which task or why. | Go to [Diagnose a task](#diagnose-a-task) and find the task in phase `Failed`. | | `NoTasks` | The resolved Stack definition has zero tasks. | Add at least one task to the referenced StackTemplate or inline `spec.template`. | ### Instance-level reasons[​](#instance-level-reasons "Direct link to Instance-level reasons") These reasons can identify an instance-level problem that blocks the current reconciliation. Existing child applications can remain, and the controller can mark existing task entries as `Blocked`. Because `TemplateNotFound` and `DestinationNotFound` are also used by child tasks, use the Ready condition message and the checks below to identify the source; task presence alone isn't enough. | Signal | Cause | Resolution | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `TemplateNotFound` | Either `spec.templateRef.name` on the StackInstance doesn't resolve to a StackTemplate, or an Argo CD Application task can't resolve its own template reference. Existing task entries can be `Blocked` in either case. | If the StackInstance uses `spec.templateRef`, run `kubectl get stacktemplate ` first. Create the missing StackTemplate or fix the StackInstance reference. If that StackTemplate exists, inspect the blocked ArgoCDApplication and fix its ArgoCDApplicationTemplate reference. | | `TemplateInvalid` | The template can't render: a missing parameter, an invalid Go template expression, a dependency cycle, or an invalid output reference. | Inspect the Ready condition message, then compare the StackInstance parameters with the StackTemplate declarations. Tasks read parameters as `{{ .Values. }}` and outputs as `{{ .Outputs.. }}`; a task that consumes an output must depend on the source task. | | `FeatureNotAllowed` | At least one task requires a feature outside the active plan. App tasks require the Apps feature. Argo CD Application tasks require the Argo CD Integration feature. Both are available on every plan, including the free plan. | Remove the unlicensed task, or use a plan that includes the missing feature. | | `DestinationNotFound` | Either the StackInstance's own destination tenant cluster or control plane cluster doesn't exist, or an `app` task's generated AppInstance reports the same reason for its own destination. Existing task entries can be `Blocked` in either case. | If `status.tasks[]` is empty or every task is `Blocked` with this reason, it's the StackInstance's own destination: create one with the same name. The controller looks up the destination by name on every reconciliation, so the existing StackInstance resolves on its own once it exists. Create a new StackInstance only to target a different destination, since `spec.destination` is immutable. If only one task is `Blocked`, inspect that task's own AppInstance destination instead. | | `DestinationDeleting` | The destination is being deleted. | Wait for the deletion to finish. As with `DestinationNotFound`, the existing StackInstance resolves once a destination with the same name is available again. | | `OutputsConflict` | Another object already owns the Secret the controller needs for captured outputs. | Read the Ready condition message and inspect Secrets in the project namespace. Don't delete a Secret until you confirm the StackInstance doesn't own it. | | `OutputsRejected` | The API server rejected the outputs Secret. | Read the Ready condition message for the underlying API error. | ### Reasons that can come from a blocked task[​](#reasons-that-can-come-from-a-blocked-task "Direct link to Reasons that can come from a blocked task") A `Pending` StackInstance can also report the first blocked task's reason directly. `TemplateNotFound` and `DestinationNotFound` can originate at either the instance or task level, so read the Ready condition message and inspect `status.tasks[]` rather than relying on the reason alone. Then go to [Diagnose a task](#diagnose-a-task): * An `app` task blocks with `AppNotFound`, `DestinationNotFound`, or `DestinationNotReady`. * An `argoCDApplication` task blocks with `InvalidConnector`, `CreateArgoClientFailed`, `GetClusterFailed`, or `TemplateNotFound`. Recreating a connector with the same name resolves `InvalidConnector` on its own. The controller watches connectors and re-enqueues affected tasks as soon as a matching name exists again. Pointing an existing connector at a different Argo CD server doesn't clean up the application it already created on the previous server. To confirm a fix worked, re-run the status command above and check that the aggregate phase reads `Healthy`. ## Diagnose a task[​](#diagnose-a-task "Direct link to Diagnose a task") If the top-level reason doesn't point to a fix, or every task reports `Healthy` but the deployed workload still isn't right, move to the task level. Use `status.tasks[].type` and `applicationName` to find the generated child. An `app` task creates an AppInstance. An `argoCDApplication` task creates an ArgoCDApplication. ``` # Replace the resource kind and generated name with values from task status. kubectl describe appinstance -n kubectl describe argocdapplication -n ``` | Signal | Cause | Resolution | | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Task phase `Pending` | The task is ready to start, but the controller hasn't materialized its child yet. | Recheck `status.observedGeneration`, then inspect StackInstance events if the task remains pending across reconciliations. | | Task phase `Waiting` | The task's `dependsOn` lists a task that isn't healthy yet. | Diagnose that dependency first. Independent tasks keep progressing while one branch waits or fails. | | Task phase `Progressing` | The generated AppInstance or ArgoCDApplication is still rolling out, or the controller is capturing its outputs. | Inspect the generated child and the task's reason and message. If the task is capturing outputs, continue with [Diagnose outputs](#diagnose-outputs). | | Task phase `Blocked` | An instance-level reason stopped reconciliation, or the task is missing a prerequisite. An `app` child can report `AppNotFound`, `DestinationNotFound`, or `DestinationNotReady`. An `argoCDApplication` child can report `InvalidConnector`, `CreateArgoClientFailed`, `GetClusterFailed`, or `TemplateNotFound`. | Compare the task reason with the Ready condition and resolve any instance-level problem first. Otherwise, inspect the generated AppInstance or ArgoCDApplication. Create the missing App or ArgoCDApplicationTemplate, fix the child destination, or fix the Argo CD connector. | | Task phase `Failed`, reason `MissingOwner` | AppInstances require an owner. vCluster Platform fills in `spec.owner` automatically for any authenticated caller creating the StackInstance through the management API, whether that's the UI, a script, or a GitOps controller. A StackInstance admitted by writing `storage.loft.sh/v1` directly bypasses that defaulting and can end up without an owner. | Set `spec.owner` on the StackInstance. The controller propagates owner and access settings to children. | | Task phase `Failed`, message includes `release name is invalid` | The generated child name fits the 63-character Kubernetes object-name limit, but Helm's own release-name limit is 53 characters. A long StackInstance name, task name, or both can cross it. | Shorten the StackInstance name, the task name, or both, then reconcile again. | | Task phase `Failed`, any other reason | The controller couldn't validate, render, build, write, or read the child; the child reported failure; or the task exceeded its deployment or output-capture timeout. A task uses its own `timeout`, then `defaults.taskTimeout`, or 10 minutes by default. | Read the task reason and message first. If `applicationName` is empty, fix the reported task definition, dependency, ownership conflict, or apply error. Otherwise, inspect the child. For an AppInstance, check Helm status, hook Jobs, Pods, events, and required credentials. For an ArgoCDApplication, check connector health, sync state, and application health. | To confirm a fix worked, re-run the compact task summary above and check that the task reads `Healthy`. ## Retry a stack[​](#retry-a-stack "Direct link to Retry a stack") Set the `platform.vcluster.com/stack-retry` annotation on the StackInstance to a task name or `all` to retry without recreating the instance: ``` metadata: annotations: platform.vcluster.com/stack-retry: "all" ``` The controller clears the annotation once it acts on it, so the annotation's presence is what "a retry is in flight" means. There's no status field for it. Requesting a retry from the UI needs the `update` verb on the StackInstance. An Argo CD Application task retries by re-triggering its child directly. A Stack retry can't re-trigger a failed `app` task because its AppInstance runs again only when the task's definition changes. The controller reports a `RetryNotApplicable` event for that task. Edit the task instead. A retry of `all` can still retry other eligible tasks in the Stack. ## Diagnose outputs[​](#diagnose-outputs "Direct link to Diagnose outputs") An output becomes available only after its source task is healthy. If a task never gets an output value, work through these constraints in order: * `fromSecret` names an existing key in a Secret. * `fromResource` selects one scalar value with kubectl JSONPath syntax, and can't read a cluster-scoped resource. * The source must be in a namespace this Stack has already deployed an App or Argo CD Application into. The controller rejects a source in any other namespace before reading it, even when RBAC and the resource lookup would otherwise succeed. * Output names and the names of tasks that declare outputs contain only lowercase letters and digits. Other task names can also contain hyphens. * The consuming task lists the source task in `dependsOn`. Published values are served through `stackinstances/outputs`, not through StackInstance status. See the [StackInstanceOutputs reference](https://vcluster.com/docs/platform/api/resources/stackinstanceoutputs.md) for the response schema. Confirm the reader has `get` permission on that subresource. Treat published credentials as sensitive even when the UI can display them. See [Stack permissions](https://vcluster.com/docs/platform/administer/users-permissions/permissions/stacks.md). Use each output's `state` and `reason` to distinguish a value that is still being captured from a definition or task failure: | State and reason | Meaning | | ------------------------------ | ---------------------------------------------------------------------------------- | | `Pending` / `NotCaptured` | The task hasn't captured a current value. | | `Pending` / `SourceChanged` | The stored value came from an older source definition and is being captured again. | | `Failed` / `OutputNotDeclared` | `publishedOutputs` references an output the task doesn't declare. | | `Failed` / `NoSource` | The task output doesn't declare a source. | | `Failed` / `TaskFailed` | The producing task failed before a current value was available. | | `Available` / `TaskFailed` | The producing task failed after capture, so the API serves the last good value. | ## Handle removed tasks[​](#handle-removed-tasks "Direct link to Handle removed tasks") When a referenced StackTemplate removes a task, inspect `status.orphanedApplications`. * `prunePolicy: Retain`, the default, keeps the child resource. * `prunePolicy: Prune` deletes removed children in reverse dependency order. Before switching to `Prune`, confirm that no retained application contains data that must survive. ## Troubleshoot NVIDIA Run:ai certified stacks[​](#troubleshoot-nvidia-runai-certified-stacks "Direct link to troubleshoot-nvidia-runai-certified-stacks") | Signal | Cause | Resolution | | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Every task is `Healthy`, but NVIDIA Run:ai itself isn't ready | A healthy Helm task only means the chart installed. The operator's own dependency checks and scheduler startup happen after that. | Verify the NVIDIA Run:ai operator's dependency checks and scheduler Pods directly. Confirm the Prometheus operator has its expected fixed name and that the GPU components are ready. | | Ingress or certificate setup fails | The ingress LoadBalancer has no address yet, or the ingress-provider setting doesn't match how the LoadBalancer surfaces its address. An IPv4 address supports a derived `nip.io` domain; AWS LoadBalancers return hostnames instead. A supplied certificate's DNS names may not cover the configured domain, or the central model's host and tenant private-CA settings may disagree. | Confirm the LoadBalancer address and matching ingress-provider setting. For a supplied certificate, confirm its DNS names cover the domain. In the central model, confirm the host and tenant agree on whether a private certificate authority is required. | | A central-model tenant waits for Secrets | The tenant's `discover` task waits for registration facts, registry credentials, and the optional certificate authority, synced from the shared host. | Check the registration StackInstance in the project namespace first. Then confirm `runai-tenant-facts`, `runai-reg-creds-host`, and (when a private CA is enabled) `runai-ca-cert` exist in the tenant cluster's `runai` namespace. Don't copy registration credentials into Stack parameters; fix registration or Secret sync instead, so rotated values keep propagating. | | A central-model tenant has no GPU nodes | No control plane cluster node matches the tenant's node-label selector. The template has no all-node fallback. | Confirm matching nodes with `kubectl get nodes -l runai.vcluster.com/tenant=`. Each tenant needs at least one matching node, and a node must not match multiple tenants. | | GPU utilization is empty in the central model | Known limitation: DCGM Exporter runs on the control plane cluster, outside each tenant's API. | None. Synced node capacity still supports allocation; per-tenant utilization views aren't available. | ### Recover from an incomplete Run:ai removal[​](#recover-from-an-incomplete-run-removal "Direct link to recover-from-an-incomplete-run-removal") Helm uninstall can leave namespaces, persistent volume claims, completed hook Jobs, and custom resource definitions. Inspect leftovers before reinstalling. For the central model, delete resources in this order: 1. Delete each tenant cluster. 2. Delete its `runai-reg-` registration StackInstance. 3. Confirm that NVIDIA Run:ai lists no remaining clusters. 4. Delete the shared host StackInstance. danger Deleting the shared host StackInstance removes the shared control plane and its PostgreSQL, NATS, and Thanos Receive data. Every remaining tenant loses service. See [Manage Certified Stacks](https://vcluster.com/docs/platform/administer/templates/certified-stacks.md) for prerequisites and supported deployment sequences. --- # Troubleshoot template and app sync failures This runbook covers three separate sync mechanisms, each with its own status conditions and its own section below. Use it when a tenant cluster, namespace, or app instance created from a template fails to come up, shows an outdated configuration, or doesn't pick up a template edit the way you expected. 1. **VirtualClusterInstance and SpaceInstance template sync**, reported through `TemplateResolved`/`TemplateSynced` conditions. Covered in [Template sync failures](#template-sync-failures). "Template sync" means the process that resolves a [template](https://vcluster.com/docs/platform/understand/what-are-templates.md), and for versioned templates a specific version, into the configuration applied to a tenant cluster or namespace. It also covers keeping that configuration up to date as the template changes. 2. **[Apps](https://vcluster.com/docs/platform/understand/what-are-apps.md)** (AppInstance), which report `Synced`/`Deployed` conditions with reasons like `AppNotFound` and `DeployFailed`. Covered in [App sync failures](#app-sync-failures). 3. **Argo CD-deployed applications** (ArgoCDApplication/ArgoCDApplicationTemplate, declared through `deploy.argoCD.applications` in `vcluster.yaml`), which report their own `Synced` condition. A tenant cluster's `Ready` condition doesn't reflect this condition's health. Covered in [Argo CD ApplicationTemplate sync failures](#argo-cd-applicationtemplate-sync-failures). A healthy tenant cluster or namespace doesn't mean every app or Argo CD deployment it hosts is healthy too. Each mechanism's conditions are independent, and a failure in one doesn't necessarily surface in another. Every table below follows the same layout: **Signal** is what you observe, **Cause** is why it happens, **Resolution** is what to do about it. note This is a different mechanism from the vCluster resource syncer (`sync.toHost`/`sync.fromHost` in `vcluster.yaml`), which moves individual Kubernetes objects between the tenant cluster and the control plane cluster. See the [vCluster sync configuration reference](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync) for that mechanism. If you're chasing a missing or duplicated Kubernetes object rather than a template not applying, you likely want that page instead. ## Template sync failures[​](#template-sync-failures "Direct link to Template sync failures") VirtualClusterInstance and SpaceInstance report template sync state through `TemplateResolved` and `TemplateSynced` conditions. A failure shows up in three different places, depending on when it occurs. ### Where the failure happens[​](#where-the-failure-happens "Direct link to Where the failure happens") Find the case that matches what you're seeing, then jump to that section. If you're troubleshooting an app instance instead, skip to [App sync failures](#app-sync-failures). For an application deployed through the Argo CD integration, see [Argo CD ApplicationTemplate sync failures](#argo-cd-applicationtemplate-sync-failures). 1. **At creation or update time**: `kubectl apply` or the platform UI rejects the request outright. Nothing is created or changed. Go to [Failures at creation or update time](#failures-at-creation-or-update-time). 2. **After creation, during reconciliation**: the instance exists, but its status conditions show the template never resolved, or resolved but never synced. Go to [Failures during reconciliation](#failures-during-reconciliation) and match the condition reason. 3. **Inside the tenant cluster itself**: both status conditions report healthy, but the resulting `vcluster.yaml` fails validation or requests a feature the license doesn't cover, so the control plane pod never starts. Go to [Failures inside the tenant cluster](#failures-inside-the-tenant-cluster). For case 2, check the instance's status conditions first to confirm which reason it's reporting: Modify the following with your specific values to generate a copyable command: PROJECT\_NAMESPACEloft-p-my-project VCLUSTER\_INSTANCE\_NAMEmy-vcluster SPACE\_INSTANCE\_NAMEmy-space ``` kubectl -n loft-p-my-project get virtualclusterinstance my-vcluster -o yaml kubectl -n loft-p-my-project get spaceinstance my-space -o yaml ``` ### Failures at creation or update time[​](#failures-at-creation-or-update-time "Direct link to Failures at creation or update time") These are rejected before any object is created or changed, so there's no instance to inspect. The error text comes back directly from the `kubectl apply`, `helm` command, or platform UI action. | Signal | Cause | Resolution | | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Template not allowed in project | The referenced template isn't in the project's [allowed templates](https://vcluster.com/docs/platform/administer/projects/allowed/templates.md) list. | Add the template to the project's allowed list, or reference an already-allowed template. | | `spec.templateRef.name` must not be blank unless authorized | The request omits a template, but the project's `requireTemplate` setting blocks template-less creation for this user. | Select an allowed template, or have a project/platform admin create the resource, or disable `requireTemplate` for the project (see [security considerations](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-no-template.md#create-without-template)). | | Both `spec.template` and `spec.templateRef` set, request `Forbidden` | An instance can use an inline template or a template reference, not both. | Remove one of the two fields. | | `cluster`, `namespace`, or `virtualCluster` field is immutable | An update tried to change which connected cluster or namespace hosts the instance, or which tenant cluster it's bound to. | Create a new instance instead of trying to move an existing one. | | Chart version is required and must be valid semantic versioning | The template (or an inline template) has an empty or malformed `helmRelease.chart.version`. This only checks the version string's format, not whether that chart version is actually published. | Set `helmRelease.chart.version` to a properly formatted version string. If the version is well-formed but doesn't exist, the failure happens later, when the platform tries to resolve the chart, not here. | | `clusterrole is required` | An inline template's `access.rules` has an entry with no `clusterRole` set. | Set `clusterRole` on that rule. | | Proxy target access rejected | An inline template's `helmRelease.values` references a resource proxy target the caller isn't authorized to use. This is checked separately from `access.rules`, and only applies to inline templates. | Adjust the proxy target configuration in the template's values, or have someone with access to that proxy target make the change. | | `virtualclusterinstances` (or similar) quota would be exceeded, triggered by a template change | Switching an instance to a different template pushed usage over a project quota. | See [Troubleshoot quota and admission denials](https://vcluster.com/docs/platform/troubleshoot/quota-admission.md). | ### Failures during reconciliation[​](#failures-during-reconciliation "Direct link to Failures during reconciliation") Once an instance exists, template sync state lives in two status conditions: `TemplateResolved` and `TemplateSynced`. `TemplateSynced` never controls readiness. Use its reason and severity to judge whether it needs action, rather than treating `False` as a failure on its own. `TemplateResolved` is the hard gate, but its effect on readiness differs by resource type. A namespace (SpaceInstance) always needs `TemplateResolved=True` to report `Ready`. A tenant cluster (VirtualClusterInstance) only needs it while the tenant cluster hasn't finished its initial deployment. Once deployed, a template failure (for example, the template being deleted afterward) no longer flips an already-`Ready` tenant cluster to not-ready. It keeps running its last-applied configuration. Modify the following with your specific values to generate a copyable command: PROJECT\_NAMESPACEloft-p-my-project VCLUSTER\_INSTANCE\_NAMEmy-vcluster SPACE\_INSTANCE\_NAMEmy-space ``` kubectl -n loft-p-my-project get virtualclusterinstance my-vcluster \ -o jsonpath='{range .status.conditions[?(@.type=="TemplateResolved")]}{.type}{" "}{.status}{" "}{.reason}{" "}{.severity}{" "}{.message}{"\n"}{end}{range .status.conditions[?(@.type=="TemplateSynced")]}{.type}{" "}{.status}{" "}{.reason}{" "}{.severity}{" "}{.message}{"\n"}{end}' kubectl -n loft-p-my-project get spaceinstance my-space \ -o jsonpath='{range .status.conditions[?(@.type=="TemplateResolved")]}{.type}{" "}{.status}{" "}{.reason}{" "}{.severity}{" "}{.message}{"\n"}{end}{range .status.conditions[?(@.type=="TemplateSynced")]}{.type}{" "}{.status}{" "}{.reason}{" "}{.severity}{" "}{.message}{"\n"}{end}' ``` warning Several resolutions below mean editing the template or its version. For a versioned template, that change propagates immediately to every other instance whose version pattern also matches, not just the one you're fixing. Validate a fix against a single instance first, using an exact version pin. See [Prevent repeat template sync failures](#prevent-repeat-template-sync-failures). #### `TemplateResolved` reasons[​](#templateresolved-reasons "Direct link to templateresolved-reasons") | Signal | Cause | Resolution | | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `TemplateNotFound` | Neither `spec.template` nor `spec.templateRef` is set, or `spec.templateRef.name` doesn't resolve to an existing template. Usually the template was renamed or deleted. Deleting a template doesn't check for instances still referencing it, so this is a common way to end up here. | Recreate the template under the same name, or update `spec.templateRef.name` on the instance to point at an existing template. | | `VersionError` | `spec.templateRef.version` doesn't have exactly three dot-separated components (`MAJOR.MINOR.PATCH`). This only checks the component count, not that each component is numeric or `X`/`x`. | Fix the version string on the instance so it has three dot-separated components, each numeric or `X`/`x`. | | `VersionNotFound` | The version string has three components, but no version on the template matches it. This also covers a component that isn't numeric or `X`/`x` (for example, `1.foo.3`), an exact pin to a version that was never created, or a major-version pattern with no matching major version. | Use a numeric or `X`/`x` component in the version pattern, add the missing version to the template, or point the instance at a version that exists. See [Template versioning](https://vcluster.com/docs/platform/administer/templates/versioning.md#template-version-strings). | | `ErrorSyncingParameters` (tenant clusters only) | A parameter value failed validation against the template's declared parameters: a missing required value, a regular expression mismatch, an out-of-range number, or the wrong type. Also covers malformed `spec.parameters` YAML and failures applying updated parameter defaults to the instance. SpaceInstance doesn't emit this reason; rendering errors in its supplied parameter data surface as `ErrorRenderingTemplate` instead. | Check the condition message for the specific parameter, then fix the instance's `spec.parameters` or the template's parameter definition. See [Template parameters](https://vcluster.com/docs/platform/administer/templates/parameters.md#parameter-validation-and-error-handling). | | `ErrorRenderingTemplate` | The Go/Helm template failed to execute, for example a syntax error, an explicit `fail` call in the template, or the rendered output couldn't be decoded. | Check the condition message for the templating error, then fix the template. | | `ErrorModifyingTemplate` | The rendered configuration's platform-managed settings failed validation, for example an invalid sleep schedule or timezone, an incomplete snapshot storage configuration, a malformed auto-deletion duration, or an inconsistent Argo CD deploy configuration. | Check the condition message for which setting failed, then fix it in the template or the instance's parameter overrides. | #### `TemplateSynced` reasons[​](#templatesynced-reasons "Direct link to templatesynced-reasons") The first two reasons cover template-change notifications every tenant cluster and namespace can report. The rest are scoped to narrower cases, noted alongside each one. | Signal | Cause | Resolution | | ----------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `NewTemplateVersionAvailable` | A newer version exists on the template than the one matching this instance's version pattern (for example, the instance pins `1.2.0` exactly, and the template now has `1.3.0` too). | No action needed unless you want the instance on the newer version. Widen the version pattern (or update it directly) to pick it up. | | `TemplateChangesAvailable` | The template's rendered configuration has changed, but the instance hasn't picked it up. This happens for **unversioned** templates whose version string isn't exactly `x.x.x` and whose `syncOnce` isn't set. | See [How template changes propagate to existing resources](https://vcluster.com/docs/platform/administer/templates/versioning.md#how-template-changes-propagate-to-existing-resources) to trigger the sync. | | `GetPlatformConfigError` (tenant clusters with `spec.external: true`) | The platform couldn't read the platform-managed settings out of the tenant cluster's existing Helm release. | Check the condition message for the underlying error. | | `PlatformConfigValidationError` (tenant clusters with `spec.external: true`) | The tenant cluster's platform-managed settings (sleep, snapshots, deletion, Argo CD) failed validation. | Check the condition message for which setting failed, then fix it on the tenant cluster. | | `ClusterNotReady` (platform-deployed tenant clusters scheduled to a connected cluster, not standalone ones) | The connected cluster this instance is scheduled to isn't initialized yet, so its configuration may not be synced. | Wait for the connected cluster to finish initializing. | To confirm a fix worked, re-run the condition check above: `TemplateResolved` should read `True`, and `TemplateSynced` should read `True` or a reason that isn't an error. ### Failures inside the tenant cluster[​](#failures-inside-the-tenant-cluster "Direct link to Failures inside the tenant cluster") Both status conditions can report healthy and the tenant cluster still fails to start, because the template resolved into a `vcluster.yaml` that the running vCluster instance rejects. These failures show up as a crashing or never-ready control plane pod, not as a condition on the instance. The commands earlier in this runbook run against the platform management API. The control plane pod itself runs on the connected cluster the tenant cluster is scheduled to, so look up that cluster and its resolved namespace from the instance first, then switch your kubeconfig context there before checking the pod: Modify the following with your specific values to generate a copyable command: PROJECT\_NAMESPACEloft-p-my-project VCLUSTER\_INSTANCE\_NAMEmy-vcluster ``` kubectl -n loft-p-my-project get virtualclusterinstance my-vcluster \ -o jsonpath='{.spec.clusterRef.cluster}{"\n"}{.spec.clusterRef.namespace}{"\n"}' ``` Modify the following with your specific values to generate a copyable command: CONNECTED\_CLUSTER\_CONTEXTmy-connected-cluster HOST\_NAMESPACEloft-my-project-v-my-vcluster CONTROL\_PLANE\_PODmy-vcluster-0 ``` kubectl --context my-connected-cluster -n loft-my-project-v-my-vcluster get pods -l app=vcluster kubectl --context my-connected-cluster -n loft-my-project-v-my-vcluster logs my-vcluster-0 -c syncer ``` | Signal | Cause | Resolution | | --------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Log mentions an Enterprise feature not available in the open source build or disabled in your license | The template enables an Enterprise-only feature (for example, embedded [etcd](https://etcd.io/), patches, central admission), but the tenant cluster's license doesn't grant it. | Remove the feature from the template, or use a license that includes it. | | Error unmarshaling JSON, unknown field | The rendered `vcluster.yaml` has a field the running vCluster version doesn't recognize, usually because the template's chart version and the field it sets are out of sync. | Align the template's `helmRelease.chart.version` with the fields used in its `values`, referencing the [vCluster configuration reference](https://vcluster.com/docs/vcluster/configure/vcluster-yaml) for that version. | | A specific semantic validation message, for example about conflicting `sync.fromHost`/`sync.toHost` settings for the same resource type | The rendered `vcluster.yaml` sets two mutually exclusive options. | Fix the conflicting settings in the template. | A chart schema rejection can happen before a pod is ever scheduled. If the pod isn't crashing but the values were rejected upfront, check the platform's own Helm release status and events for the instance, in addition to the pod logs. To confirm a fix worked, re-run the pod command above and check that the control plane pod reaches `Running` and `Ready`. ### Prevent repeat template sync failures[​](#prevent-repeat-template-sync-failures "Direct link to Prevent repeat template sync failures") **Know whether your template auto-propagates before relying on it.** Versioned templates re-apply matching version changes automatically. An unversioned template needs either a persistent `x.x.x` version string, which keeps auto-syncing every edit, or `syncOnce` to pick up a single edit on demand. Unlike the `X`/`x` wildcards on a versioned template, `x.x.x` on an unversioned one is matched exactly and is case-sensitive: `X.X.X` doesn't work. See [How template changes propagate to existing resources](https://vcluster.com/docs/platform/administer/templates/versioning.md#how-template-changes-propagate-to-existing-resources). **Don't delete a template while instances still reference it.** Deleting a template doesn't check for referencing instances first. Any instance still pointing at it starts reporting `TemplateNotFound` on its next reconcile, and its last-applied configuration is frozen in place rather than cleared. Point instances at a replacement template, or confirm none exist, before deleting. **Match a template's Enterprise features to the license the tenant clusters will run under.** A template that enables an Enterprise-only feature works fine until it's used with a license or image that doesn't cover that feature. At that point, every tenant cluster created from it fails the same way. Verify feature availability against the target license before rolling out a template change broadly. **Test template edits on one instance before a fleet-wide rollout.** Template sync applies automatically for versioned templates. A bad edit to a version that many instances already match propagates to all of them on their next reconcile. Validate a new version against a single instance first, using an exact version pin, before widening other instances' version patterns to include it. ## App sync failures[​](#app-sync-failures "Direct link to App sync failures") [Apps](https://vcluster.com/docs/platform/understand/what-are-apps.md), a Helm chart, Kubernetes manifest, or bash script packaged for reuse, deploy into a cluster, namespace, or tenant cluster through an AppInstance. AppInstance uses a separate reconciler from VirtualClusterInstance and SpaceInstance, with its own conditions, `Synced` and `Deployed`. `status.phase` (`Pending`, `Failed`, or `Ready`) and `status.reason` summarize the same state more directly than the conditions do, and the table below is organized around them. Check `status.phase`/`status.reason` first, then the conditions for a reason like `UnresolvableVersion` that only appears on `Deployed`: Modify the following with your specific values to generate a copyable command: PROJECT\_NAMESPACEloft-p-my-project APP\_INSTANCE\_NAMEmy-app ``` kubectl -n loft-p-my-project get appinstance my-app \ -o jsonpath='{.status.phase}{" "}{.status.reason}{" "}{.status.message}{"\n"}' kubectl -n loft-p-my-project get appinstance my-app \ -o jsonpath='{range .status.conditions[*]}{.type}{" "}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}' ``` Or get the full object, including `status.app` and `status.releaseName`: Modify the following with your specific values to generate a copyable command: PROJECT\_NAMESPACEloft-p-my-project APP\_INSTANCE\_NAMEmy-app ``` kubectl -n loft-p-my-project get appinstance my-app -o yaml ``` | Signal | Cause | Resolution | | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `status.phase: Failed`, reason `MissingOwner` | The instance has no owner, so the platform has no identity to mint the access key a deploy runs under. This normally only happens to an instance created outside the platform UI, for example applied directly with `kubectl`. | Set an owner on the instance, or recreate it through a path that assigns one automatically. | | `status.phase: Pending`, reason `AppNotFound` | `spec.templateRef.name` doesn't resolve to an existing App yet. This is expected, and resolves on its own, when the App and the instance were applied together (for example `kubectl apply -f` on a directory) and the App simply hasn't landed yet. | Confirm the referenced App exists. If both were just applied together, wait for the App to appear. | | `Deployed` condition `False`, reason `UnresolvableVersion` | The instance's `spec.templateRef.version` no longer matches any version on the referenced App, usually because that version was removed from the App after the instance started using it. The instance keeps running its last successful release. | Point the instance at a version that still exists on the App, or re-add the missing version to the App. | | `status.phase: Pending`, reason `DestinationNotFound` | The instance's destination, the cluster, namespace, or tenant cluster it deploys into, doesn't exist. | Create the destination, or fix the destination reference on the instance. | | `status.phase: Pending`, reason `DestinationNotReady` | The destination exists but can't take a deploy yet, for example it's still coming up or being deleted. | Wait for the destination to become ready. This resolves on its own once it does. | | `status.phase: Failed`, reason `DeployFailed` | The Helm install/upgrade, manifest apply, or bash script itself failed. The condition message carries the underlying error. | Check the condition message for the specific error. See [Secret and credential resolution failures](#secret-and-credential-resolution-failures) if it mentions a project secret. | A `DeployFailed` instance retries automatically a bounded number of times with backoff, without any action needed. Once the condition message says no retries are left, only a change to the App or a `spec` change to the instance, including its destination reference, triggers another attempt. The destination merely becoming ready on its own doesn't revive an exhausted instance. To confirm a fix worked, re-run the command above: `status.phase` should read `Ready`, with `Synced` and `Deployed` both `True`. ### App parameter failures[​](#app-parameter-failures "Direct link to App parameter failures") An App declares parameters (`spec.parameters`, each with a `type`, `options`, `min`/`max`, and `required`) that the platform UI renders as a form. See [Use Parameters with Apps](https://vcluster.com/docs/platform/use-platform/apps/use-parameters.md). Unlike a tenant cluster template's parameters, this schema isn't enforced against an AppInstance's own `spec.parameters` values when deploying. A missing required value, wrong type, or out-of-range number doesn't produce a dedicated validation error, the value just flows straight into the deploy. `spec.parameters` on the instance is parsed as YAML, then used to render the App's Helm values before merging back in. Any failure in that process surfaces as `DeployFailed`, with the message wrapped as `apply parameters: ...`: | Signal (in the `DeployFailed` message) | Cause | Resolution | | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | `apply parameters:` wrapping a type-mismatch error, no `unmarshal` in the message | `spec.parameters` isn't a mapping (object), for example a bare scalar, string, or list. `spec.parameters` is stored as raw JSON, so genuinely invalid YAML is rejected by the API server before the instance is ever created; a wrong-shaped value is what actually reaches this check. | Set `spec.parameters` to a mapping of parameter names to values. | | `apply parameters:` wrapping a template execution error | The App's Helm values template references a parameter (for example `.Values.foo.bar`) in a way the supplied `spec.parameters` can't satisfy. | Check the condition message for the specific path, then fix `spec.parameters` or the App's values template. | | `apply parameters:` message that also contains `unmarshal` | The App's values template rendered into text that isn't valid YAML, often because a parameter value's substituted text broke the YAML structure. | Check the rendered content in the condition message, then adjust the parameter value or the values template. | ### Secret and credential resolution failures[​](#secret-and-credential-resolution-failures "Direct link to Secret and credential resolution failures") An App, or an inline `AppInstance.spec.template`, can reference Helm repository credentials as a project secret (`usernameRef`/`passwordRef`, each a `projectSecretRef`) instead of supplying them inline. See [Private Helm Repositories for Apps](https://vcluster.com/docs/platform/administer/templates/external-credentials.md). Resolving that reference happens as part of the deploy, so a bad reference surfaces as `DeployFailed`, with the message describing exactly which check failed: | Signal (literal substring in the `DeployFailed` message) | Cause | Resolution | | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | | `project is empty in app project secret ref` or `project secret name is empty in app project secret ref` | The `projectSecretRef` is missing its `project` or `name` field. | Set both fields on the `projectSecretRef`. | | `retrieve project secret /` followed by a not-found error | No secret with that name exists in the referenced project's namespace. | Create the secret, or fix the `project`/`name` on the ref. | | `project secret / is empty` | The secret exists but has no data keys at all. | Add the credential data to the secret. | | `project secret / has multiple values, but no key was specified` | The ref omits `key`, but the secret has more than one data key, so there's nothing to pick automatically. | Set `key` on the `projectSecretRef` to the specific data key to use. | | `project secret / key is empty` | The ref's `key` doesn't exist in the secret's data, or its value is empty. | Add that key to the secret, or point `key` at one that exists. | A ref that omits `key` only resolves automatically when the secret has exactly one data key. ## Argo CD ApplicationTemplate sync failures[​](#argo-cd-applicationtemplate-sync-failures "Direct link to Argo CD ApplicationTemplate sync failures") An `ArgoCDApplicationTemplate` is a reusable Argo CD `ApplicationSpec`, referenced from `deploy.argoCD.applications` in a tenant cluster's `vcluster.yaml`. See [Deploy applications](https://vcluster.com/docs/platform/integrations/argocd/deploy-applications.md). The platform renders each reference into an `ArgoCDApplication` object and reconciles that onto the connected Argo CD instance, a third mechanism with its own `Synced` condition and reasons, independent from both template sync and App sync. A tenant cluster's own `Ready` condition doesn't reflect the health of its Argo CD-deployed applications. A VirtualClusterInstance aggregates any of its `ArgoCDApplication` objects reporting an error-severity `Synced: False` into its own `ArgoCDIntegrationSynced` condition, reason `ArgoCDApplicationFailed`, but that condition isn't part of the `Ready` summary. It also only reflects error-severity failures. An info-severity issue like `ClusterRegistrationPending` never flips it, so a healthy `ArgoCDIntegrationSynced` doesn't rule out that case. Check `ArgoCDIntegrationSynced` first for error-severity failures, then check each application directly regardless of what it reports. Modify the following with your specific values to generate a copyable command: PROJECT\_NAMESPACEloft-p-my-project VCLUSTER\_INSTANCE\_NAMEmy-vcluster ``` kubectl -n loft-p-my-project get virtualclusterinstance my-vcluster \ -o jsonpath='{range .status.conditions[?(@.type=="ArgoCDIntegrationSynced")]}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}' ``` The `ArgoCDApplication` resource name can be platform-generated and differ from the application's `displayName` (see [Application naming](https://vcluster.com/docs/platform/integrations/argocd/deploy-applications.md#application-naming)), so list this tenant cluster's applications before inspecting one by name: Modify the following with your specific values to generate a copyable command: PROJECT\_NAMESPACEloft-p-my-project VCLUSTER\_INSTANCE\_NAMEmy-vcluster ``` kubectl -n loft-p-my-project get argocdapplication -l loft.sh/vcluster-instance-name=my-vcluster ``` Then inspect the one that's failing: Modify the following with your specific values to generate a copyable command: PROJECT\_NAMESPACEloft-p-my-project ARGOCD\_APPLICATION\_NAMEmy-app ``` kubectl -n loft-p-my-project get argocdapplication my-app \ -o jsonpath='{range .status.conditions[?(@.type=="Synced")]}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}' ``` Or get the full object: Modify the following with your specific values to generate a copyable command: PROJECT\_NAMESPACEloft-p-my-project ARGOCD\_APPLICATION\_NAMEmy-app ``` kubectl -n loft-p-my-project get argocdapplication my-app -o yaml ``` | Signal | Cause | Resolution | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Synced` condition `False`, reason `TemplateNotFound` | The `ArgoCDApplication`'s `templateRef.name` doesn't resolve to an existing `ArgoCDApplicationTemplate`. | Fix the template name in `deploy.argoCD.applications`, or create the missing `ArgoCDApplicationTemplate`. | | `Synced` condition `False`, reason `InvalidConnector` | The application's destination couldn't be resolved, or the named connector doesn't exist or failed to load. | Check the condition message, then fix the connector reference or its configuration. See [Connect to Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md) or [Connect to Akuity](https://vcluster.com/docs/platform/integrations/argocd/connect-akuity.md). | | `Synced` condition `False`, reason `CreateArgoClientFailed` | The platform couldn't create an API client from the connector's configuration, usually bad credentials or an unreachable endpoint. | Check the condition message, then fix the connector. See [Connect to Argo CD](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md) or [Connect to Akuity](https://vcluster.com/docs/platform/integrations/argocd/connect-akuity.md). | | `Synced` condition `False`, reason `ClusterRegistrationPending`, severity Info | The target cluster's registration with Argo CD hasn't finished yet. | Wait for the connected cluster to finish registering. This resolves on its own. | | `Synced` condition `False`, reason `InvalidSyncAnnotation` | An annotation on the application that controls Argo CD sync behavior is malformed. | Check the condition message for which annotation, then fix it in the template or application spec. | | `Synced` condition `False`, reason `BuildApplicationFailed`, `GetClusterFailed`, `GetApplicationFailed`, `RefreshApplicationFailed`, `UpsertApplicationFailed`, `SyncApplicationFailed`, or `DeleteApplicationFailed` | The platform failed to build, read, or write the application against the Argo CD API itself. | Check the condition message for the underlying API error. | To confirm a fix worked, re-run the command above and check that `Synced` reads `True`. On a managed integration, `ArgoCDIntegrationSynced` on the tenant cluster reads `True` once no application reports an error-severity failure, not once every application is fully healthy, so still verify this specific application's own `Synced` condition rather than relying on the aggregate alone. It doesn't disappear. Absence there is only expected if the Argo CD integration itself is no longer managed. --- # Resolve vCluster TSNet connection failures The [TSNet (Tailscale Network)](https://tailscale.com/kb/1244/tsnet) connection failure prevents the vCluster Platform from properly marking tenant clusters as ready, even when they are running successfully. This occurs because TSNet cannot establish a stable connection with the platform, which blocks communication between the vCluster on the connected cluster and the platform. As a result, the vCluster appears to be running in the control plane cluster but shows as it is not ready in the platform UI, which prevents proper management and monitoring. ## Error message[​](#error-message "Direct link to Error message") You might find the following error in the logs of your vCluster pods when TSNet fails to connect: ``` ERROR ts-net-controller tsnet/tsnet.go:148 Check if TSNet is online { "component": "vcluster", "failCounter": 5, "error": "vcluster tsnet server is not online: ... Tailscale could not connect to any relay server. ... The last login error was: register request: Post \"https://vcluster-test.io/coordinator/machine/register\": connection attempts aborted by context: context deadline exceeded" } ``` ## Issue[​](#issue "Direct link to Issue") When the TSNet connection fails, you might observe the following: * **vCluster status discrepancy**: The vCluster shows as `Running` in the remote control plane cluster but is not marked as ready in the vCluster Platform UI. * **Successful local connection**: You can successfully connect to the vCluster using `vcluster connect` command. * **Platform communication blocked**: The vCluster cannot communicate with the platform for status updates and management operations. To verify TSNet connection issues, check the vCluster logs: ``` kubectl -n logs | grep -i "tsnet\|ts-net" ``` Look for errors related to connection timeouts or relay server failures. ## Causes[​](#causes "Direct link to Causes") TSNet connection failures might occur due to the following: * **Restricted egress policies**: Control plane clusters with strict network policies (common in GKE, EKS, and other managed Kubernetes services) might block outbound connections to the coordination server. * **Ingress controller interference**: Ingress controllers like Istio might block or interfere with WebSocket upgrades required for TSNet communication. * **Direct connection attempts**: TSNet defaults to attempting direct peer-to-peer connections, which often fail in cloud environments with network restrictions. * **Firewall restrictions**: Corporate firewall rules or cloud security groups might block the required ports and protocols for TSNet communication. ## Solution[​](#solution "Direct link to Solution") To resolve TSNet connection failures, disable direct connection mode and configure TSNet to use the platform agent for communication. This approach works better in restricted network environments. 1. Configure TSNet to disable direct connections Update your vCluster configuration to explicitly disable direct peer-to-peer connections: vcluster.yaml ``` controlPlane: statefulSet: env: - name: "TS_DEBUG_DIAL_DIRECT" value: "false" ``` 2. Update your existing vCluster with the new configuration: ``` vcluster create -f vcluster.yaml --upgrade ``` Or if using Helm directly: ``` helm upgrade vcluster --repo https://charts.loft.sh -f vcluster.yaml -n ``` 3. Force a restart of the vCluster pods to ensure the new environment variable takes effect: ``` kubectl -n rollout restart statefulset ``` 4. Ensure the vCluster can reach the platform coordination server: ``` kubectl -n exec -it -- nslookup ``` ## Verification[​](#verification "Direct link to Verification") After completing the solution steps: 1. Ensure the vCluster pod is running in your namespace: ``` kubectl -n get pods | grep ``` 2. Check the pod logs for TSNet connection status: ``` kubectl -n logs | grep -i "tsnet" | tail -10 ``` 3. Open the vCluster Platform UI and confirm that the vCluster is marked as **Ready**. 4. Connect to the tenant cluster and check node access: ``` vcluster connect -n kubectl get nodes ``` ## Prevention[​](#prevention "Direct link to Prevention") To prevent future TSNet connection issues, you can apply the following preventive measures in your vCluster configuration: vcluster.yaml ``` controlPlane: statefulSet: env: - name: "TS_DEBUG_DIAL_DIRECT" value: "false" - name: "TS_DEBUG_NETCHECK" value: "true" ``` This configuration: * Disables direct peer-to-peer connections * Sets appropriate resource limits for stable operation ## Best practices[​](#best-practices "Direct link to Best practices") To ensure reliable TSNet connectivity in the platform: * **Always disable direct connections**: Set `TS_DEBUG_DIAL_DIRECT=false` in environments to avoid connection issues. * **Monitor network policies**: Ensure your control plane cluster's network policies allow outbound connections to the platform coordination server. * **Configure appropriate timeouts**: Set reasonable timeout values for network operations in restricted environments. * **Use resource limits**: Configure appropriate CPU and memory limits to prevent resource contention that could affect network connectivity. * **Test connectivity during deployment**: Verify platform connectivity as part of your vCluster deployment process to catch issues early. --- # Authentication and authorization This guide explains the key security concepts in vCluster Platform and demonstrates how to set up a complete authentication system with role-based permissions. ## Core security concepts[​](#core-security-concepts "Direct link to Core security concepts") The platform uses several objects to manage access and permissions: | Object | Description | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **User** | A person with platform access. Users can have Management Roles assigned directly and can belong to Groups and Teams. [API reference](https://www.vcluster.com/docs/platform/api/resources/user) | | **SSO Group** | A representation of a group from your [configured SSO provider](https://www.vcluster.com/docs/platform/configure/platform-configs/single-sign-on). These groups are imported during authentication and stored in User records. | | **Team** | A collection of platform users. Users join teams either through explicit assignment or via SSO Group membership. Teams can have Management Roles. [API reference](https://www.vcluster.com/docs/platform/api/resources/team) | | **Management Role** | Defines permissions granted to users or teams within specific scopes. [API reference](https://www.vcluster.com/docs/platform/api/resources/clusterroletemplate) | | **Project** | An organizational unit for resources such as tenant clusters. [Projects](https://www.vcluster.com/docs/platform/understand/what-are-projects) help enforce access control and resource quotas. | ## Setting up authentication with Microsoft Entra ID[​](#setting-up-authentication-with-microsoft-entra-id "Direct link to Setting up authentication with Microsoft Entra ID") This example shows how to implement a comprehensive security model using Microsoft Entra ID (formerly Azure AD). ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Microsoft Entra ID tenant configured as your identity provider * vCluster Platform installed and running * Administrative access to vCluster Platform ### Authentication setup[​](#authentication-setup "Direct link to Authentication setup") 1. Configure Microsoft Entra ID as your SSO provider using either: * [OIDC configuration](https://www.vcluster.com/docs/platform/configure/platform-configs/single-sign-on) * [SAML configuration](https://www.vcluster.com/docs/platform/configure/platform-configs/single-sign-on) 2. Configure your Entra ID application to include group memberships in authentication tokens 3. Configure vCluster Platform to accept specific groups for platform access ### Define roles for different personas[​](#define-roles-for-different-personas "Direct link to Define roles for different personas") Create Management Roles aligned with your organization's needs. This example uses four distinct roles: 1. **Platform Operations** - Full platform management permissions 2. **DevOps Engineers** - Create and manage tenant clusters within projects 3. **Developers** - Connect to and use tenant clusters 4. **Automation Service Account** - Programmatic access for GitOps workflows ### Team structure[​](#team-structure "Direct link to Team structure") 1. Create Teams that correspond to your organizational structure: ``` kubectl apply -f platform-ops-team.yaml kubectl apply -f app-team-devops.yaml kubectl apply -f app-team-developers.yaml kubectl apply -f gitops-automation-team.yaml ``` 2. Associate Teams with SSO Groups: ``` apiVersion: management.loft.sh/v1 kind: Team metadata: name: app-team-devops spec: displayName: "Application Team - DevOps" description: "DevOps engineers responsible for cluster provisioning" groups: - name: "app-team-devops@example.com" ``` 3. Assign appropriate Management Roles to each Team: ``` apiVersion: management.loft.sh/v1 kind: Team metadata: name: platform-ops-team spec: displayName: "Platform Operations" clusterRoles: - platform-admin ``` ### Project organization[​](#project-organization "Direct link to Project organization") Create Projects for each application or team: ``` apiVersion: management.loft.sh/v1 kind: Project metadata: name: app-team-project spec: displayName: "Application Team Project" description: "Resources for the application team" allowedClusters: - name: "prod-cluster" - name: "dev-cluster" quotas: virtualClusters: 5 ``` ### GitOps governance[​](#gitops-governance "Direct link to GitOps governance") For organizations requiring additional governance: 1. Manage Projects, Management Roles, and Teams using GitOps workflows 2. Store configurations in a version-controlled repository 3. Use your CI/CD system with the automation service account to apply changes ## Management role example[​](#management-role-example "Direct link to Management role example") ``` apiVersion: management.loft.sh/v1 kind: ClusterRoleTemplate metadata: name: loft-management-project-admin-custom labels: loft.sh/project-role: 'true' spec: displayName: Custom Project Admin description: >- Allows the user or team to manage the project. Gives full access to modifying the project and all objects inside it. management: true access: - verbs: - get users: - '*' clusterRoleTemplate: metadata: creationTimestamp: null rules: - verbs: - '*' apiGroups: - management.loft.sh resources: - virtualclusterinstances - virtualclusterinstances/log - virtualclusterinstances/kubeconfig - virtualclusterinstances/accesskey - virtualclusterinstances/restricted - projectsecrets - projects - projects/spec.* ``` ## Additional resources[​](#additional-resources "Direct link to Additional resources") * [Single sign-on configuration](https://www.vcluster.com/docs/platform/configure/platform-configs/single-sign-on) * [Platform permissions reference](https://www.vcluster.com/docs/platform/administer/users-permissions/permissions/platform) * [SSO group synchronization](https://www.vcluster.com/docs/platform/configure/platform-configs/single-sign-on#sso-group-sync) --- # What happens during control plane outages? vCluster Platform is the management plane for projects, access, templates, lifecycle operations, and connected cluster visibility. Tenant clusters have their own control planes. Those control planes might run on the same Kubernetes cluster as Platform or on separate connected clusters. Because these layers are separate, an outage affects different capabilities depending on what is unavailable. ## Platform outage[​](#platform-outage "Direct link to Platform outage") If vCluster Platform is unavailable but the connected clusters and tenant cluster control planes remain healthy, tenant clusters do not automatically stop. Existing workloads are not intentionally deleted because Platform is unavailable. | Capability | Status | Impact | | ------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------ | | Platform UI and API | Unavailable | Users and automation cannot reliably use Platform while it is down. | | Lifecycle operations | Unavailable | Creating, deleting, sleeping, waking, updating, or restoring tenant clusters fail or wait. | | Access, audit, and integrations | Limited | Platform-provided access features, audit, integrations, and automation are unavailable or delayed. | | Tenant cluster runtime | Available | Tenant clusters do not automatically stop if their own control planes and connected clusters remain healthy. | | External deployment tools | Available | Externally deployed tenant clusters continue to be managed by their original deployment tool. | Platform high availability protects the management plane by running multiple Platform replicas. It doesn't replace high availability for tenant cluster control planes or for the Kubernetes clusters that host tenant clusters. See [High Availability Installation](https://vcluster.com/docs/platform/maintenance/reliability-scaling/high-availability.md). ## Connected control plane cluster outage[​](#connected-control-plane-cluster-outage "Direct link to Connected control plane cluster outage") A connected control plane cluster is a Kubernetes cluster where Platform can deploy or manage tenant clusters. If a connected control plane cluster is unavailable, Platform can't complete operations that require access to that cluster. | Capability | Status | Impact | | ------------------------------------ | ----------- | ------------------------------------------------------------------------------------------------------ | | Platform UI and API | Available | Platform remains reachable and continues managing other healthy connected clusters. | | Connected cluster visibility | Limited | Platform marks the connected cluster or its network peer as offline until connectivity returns. | | Lifecycle operations on that cluster | Unavailable | Platform can't create, update, delete, sleep, wake, or inspect tenant clusters hosted there. | | Tenant cluster reconciliation | Unavailable | Platform-managed reconciliation pauses or fails for tenant clusters on the unavailable cluster. | | Tenant workloads | Limited | Availability depends on whether their control planes and worker nodes are still running and reachable. | If tenant cluster control planes are hosted on the unavailable control plane cluster, tenant users can't reliably use those tenant cluster APIs until the control plane cluster recovers. For the tenant cluster view of this behavior, see [What happens during control plane outages?](https://vcluster.com/docs/vcluster/understand/control-plane-outages.md). ## Tenant cluster control plane outage[​](#tenant-cluster-control-plane-outage "Direct link to Tenant cluster control plane outage") If an individual tenant cluster control plane is unavailable but Platform remains healthy, Platform can still be reachable, but operations against that tenant cluster are limited. Platform may report the tenant cluster's phase as pending or failed, and its status conditions can further indicate that it isn't ready or is offline, depending on the deployment model and connectivity path. | Capability | Status | Impact | | ---------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------ | | Platform UI and API | Available | Platform can remain reachable and continue managing other healthy tenant clusters. | | Operations for this tenant cluster | Limited | Operations that require the unavailable tenant cluster API can fail or remain pending. | | Tenant cluster status | Limited | Platform may report the tenant cluster's phase as pending or failed, with status conditions indicating it isn't ready or is offline. | | Existing tenant workloads | Limited | Platform does not intentionally delete them for tenant clusters connected through an agent on a connected cluster. | | Other tenant clusters | Available | Healthy tenant clusters can continue to be managed independently. | For externally deployed tenant clusters connected directly through a network peer, such as vCluster Standalone, using their own backing store, Platform deletes the instance if it stays disconnected for more than 7 days. Tenant clusters connected through an agent are not affected by this cleanup. Recovery resumes after the tenant cluster control plane becomes reachable again. For production environments, plan availability at each layer: Platform, the connected control plane clusters, and the tenant cluster control planes. --- # Platform-Managed vs. Externally Deployed Tenant Clusters Tenant clusters in vCluster Platform are either **platform-managed** or **externally deployed**, depending on how they are created and who manages their lifecycle. ![externally deployed](/docs/assets/images/external-clusters-563877aa49dcd3a070931444ad072829.png) ## Platform-managed tenant clusters[​](#platform-managed-tenant-clusters "Direct link to Platform-managed tenant clusters") Platform deploys and fully manages platform-managed tenant clusters. It handles provisioning, upgrades, configuration changes, and deletion. All platform features are available. Deployment methods: * Platform UI * `vcluster platform create vcluster ` CLI command ## Externally deployed tenant clusters[​](#externally-deployed-tenant-clusters "Direct link to Externally deployed tenant clusters") Externally deployed tenant clusters are created outside Platform using tools like Helm, Argo CD, Terraform, or ClusterAPI. You can connect them to Platform to gain visibility and access to certain features without changing how their lifecycle is managed. Perform lifecycle changes (upgrades, configuration updates, deletion) through the original deployment tool, not the Platform UI. Platform does not reconcile externally deployed clusters. ### Available Platform features when connected[​](#available-platform-features-when-connected "Direct link to Available Platform features when connected") When you add an externally deployed tenant cluster to Platform and connect it with the vCluster Platform agent, the following features become available: * Auto sleep (requires agent and `vcluster.yaml` configuration) - Pro features of vCluster (license key distribution) - Administrative overview of the tenant cluster fleet - UI inspection of cluster versions, enabled features, and resources - Issue detection (outdated Kubernetes versions, vCluster errors) - Health status monitoring and enhanced debugging - User access management through Platform (useful with SSO) - Platform integrations (Argo CD, Vault secrets, and others) note Features in the `platform:` [section of vcluster.yaml](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/platform.md) are only available if the agent is installed. ## Choose a deployment model[​](#choose-a-deployment-model "Direct link to Choose a deployment model") You can choose per-project or per-tenant-cluster, mixing both models in the same environment. **Use platform-managed when:** * You want the simplest operational model with full feature access * Tenant clusters are created on demand with no need for external provisioning tooling **Use externally deployed when:** * You already manage tenant cluster lifecycle with Argo CD, ClusterAPI, or Terraform and want to keep that workflow * You need GitOps-driven provisioning with Platform visibility and access control ## Connect an externally deployed tenant cluster[​](#connect-an-externally-deployed-tenant-cluster "Direct link to Connect an externally deployed tenant cluster") To add an existing tenant cluster to Platform without transferring lifecycle management: 1. Ensure the tenant cluster is running vCluster v0.20 or later. 2. Run: ``` vcluster platform add vcluster [name] ``` This is non-invasive and reversible. Your existing deployment workflow remains unchanged. ## Next steps[​](#next-steps "Direct link to Next steps") * [Create a tenant cluster](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-no-template.md) in the platform * [Add an externally deployed tenant cluster](https://vcluster.com/docs/platform/use-platform/virtual-clusters/add-virtual-clusters.md) to the platform --- # How licensing works vCluster Platform holds the license for a deployment. The license determines which features are enabled and sets limits on how many of certain resources you can create. In the normal connected model, clusters hold no license of their own. They read a list of enabled features from the platform. Most licensing questions therefore come down to how the platform obtained its license and whether clusters can reach the platform to read features from it. A cluster can instead be given its own offline license key, which the sections below cover separately. This page explains how that works, what each component caches, and how licensing degrades when it becomes unavailable. For the setup procedures themselves, see the air-gapped guides for an [offline license key](https://vcluster.com/docs/platform/install/air-gapped.md) or an [offline license server](https://vcluster.com/docs/platform/install/air-gapped/with-offline-license-server.md). ## How the platform obtains its license[​](#how-the-platform-obtains-its-license "Direct link to How the platform obtains its license") The platform selects one of three licensing modes at startup, based on environment variables set on the platform deployment. | Mode | Enabled by | Source of the license | | ---------------------- | ------------------------------------- | ----------------------------------------------------------------------------- | | Online | Default, no configuration needed | `https://admin.loft.sh/license/v2` | | Offline license key | `LICENSE_KEY` environment variable | The key itself, a signed token verified locally against a built-in public key | | Offline license server | `LICENSE_SERVER` environment variable | An in-cluster license server that serves the same API as the hosted one | Offline license key mode never contacts a license server. The platform decodes and verifies the key in-process, so use this mode when no egress is possible at all. A fourth variable, `LICENSE_TOKEN`, doesn't change the mode. It supplies a static token that the platform sends when it requests a license, so the instance doesn't need interactive activation. The platform logs `LICENSE_TOKEN env var is set, platform activation is not required` at startup when it's present. To find out which mode a running platform is in, see [Inspect and debug licensing](#inspect-and-debug-licensing). ### Instance identity[​](#instance-identity "Direct link to Instance identity") In online and offline license server modes, the platform identifies itself to the license server with: * The UID of the `kube-system` namespace in the control plane cluster. * A short-lived token signed with the private key from the `loft-cert` Secret, sent along with that Secret's certificate. This pairing ties a license to one platform instance. If the `loft-cert` Secret is lost, the platform generates a new identity and the existing license no longer matches it. Preserve that Secret across reinstalls and migrations. See [Restore license](https://vcluster.com/docs/platform/maintenance/backup-restore/preserve-license.md) for the backup and restore procedure. The `loft-license-cache` Secret isn't part of the platform's identity. It holds a cached copy of the license response and the platform rebuilds it automatically, so it doesn't need to be migrated. ## How clusters receive features[​](#how-clusters-receive-features "Direct link to How clusters receive features") In the normal connected model, a cluster reads its enabled features from the platform. The control plane detects which licensing mechanism is available at startup and selects it. | Type | Detected when | Features come from | | -------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ | | Online | A platform host and access key are both available, and FIPS mode is off | The platform's management API `Features` endpoint | | Offline | A key is set but no platform host | A signed offline license key, verified locally | | Standalone | `controlPlane.standalone.enabled` is set, and no platform host and access key are available (see below) | All Enterprise features stay enabled. Built-in allowances and `Feature` resources determine license warnings | | In-cluster API | `Feature` resources exist in the control plane cluster and the cluster's control plane has RBAC to list them | `Feature` resources in the control plane cluster | | None | Nothing above is detected | No Enterprise features are enabled | In production builds, the control plane evaluates these types in the order shown and selects the first match. It logs the selected type at startup, for example `detected license type {"licenseType": "Online"}`. Online is the common case. By default, the control plane reads the platform connection from the `vcluster-platform-api-key` Secret in the cluster's host namespace. To point it at a different Secret, set `platform.apiKey.secretName` and `platform.apiKey.namespace` in `vcluster.yaml`. Within whichever Secret applies, the control plane resolves the access key in this order. 1. The `LICENSE` environment variable. 2. The `LOFT_PLATFORM_ACCESS_KEY` environment variable. 3. The only key in the Secret, if the Secret holds exactly one key. 4. The `license` key in the Secret, then the `accessKey` key. It resolves the host from `LOFT_PLATFORM_HOST` first, then from the Secret's `host` key. Use an address the pod can reach The host recorded in the Secret must be reachable from inside the control plane cluster. Logging in through a port-forward records a `localhost` address that the cluster's control plane pod can't reach, so feature retrieval fails. See [Resolve Pro feature license errors](https://vcluster.com/docs/vcluster/troubleshoot/pro-feature-license-error.md). Two cases trip people up when the detected type isn't what they expected. FIPS mode is the first. With FIPS enabled, a host and access key don't select the online type, and detection falls through to the remaining rows. A cluster that looks correctly configured can land on standalone, in-cluster API, or none. Check the logged type before assuming the credentials are at fault. The second is that the standalone deployment mode and the standalone license type aren't synonymous. With FIPS off, a standalone deployment configured with a platform host and access key selects the online type first. It uses the non-expiring file cache described below. The standalone type also doesn't disable Enterprise features. It allows the standalone and Private Nodes features by default. For other Enterprise features, `Feature` resources in the standalone cluster determine whether `kubectl` requests receive an unlicensed-use warning. ## Required network access[​](#required-network-access "Direct link to Required network access") Only the platform contacts the license server. Neither connected cluster agents nor clusters do. | Source | Destination | Protocol and port | Purpose | | -------------------------- | -------------------------------------------- | -------------------------------- | -------------------------------------------------------------------- | | Platform pods | `admin.loft.sh` | HTTPS, TCP `443` | License retrieval every 6 hours and usage snapshots every 15 minutes | | Platform pods | Analytics endpoint from the license response | As specified by the endpoint URL | Analytics every 30 minutes when there has been activity | | Cluster control plane pods | Platform endpoint | HTTPS, TCP `443` | Feature retrieval and registration | | Connected cluster agents | None for licensing | Not applicable | Agents don't perform license validation | In offline license key mode, no egress to `admin.loft.sh` is required at all. For license retrieval in offline license server mode, the platform reaches the in-cluster server at the address given in `LICENSE_SERVER`. Analytics calls go to the exact endpoint URL named in the license response, which isn't necessarily `admin.loft.sh`. They only happen when there has been activity since the last call. A response with no analytics endpoint stops them entirely. The platform Service listens on port `443` and forwards to port `10443` on the platform pods. Kubernetes NetworkPolicy matches the pod port rather than the Service port, so an egress rule targeting platform pods directly has to allow `10443`. The cluster path is the same connection used for the rest of platform communication, so it needs no licensing-specific firewall rules. For the full connectivity model, see [Platform communication and connectivity](https://vcluster.com/docs/platform/understand/platform_communication.md). ## Cache and refresh behavior[​](#cache-and-refresh-behavior "Direct link to Cache and refresh behavior") The online platform and cluster paths cache their last successful results. These caches keep a temporary licensing outage from immediately taking those components down. | Component | Refresh trigger | Cache location | Cache lifetime | | ----------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | Platform in online or license server mode | Every 6 hours | `loft-license-cache` Secret in the platform namespace | Reloaded at startup, with a 7-day grace period after retrieval starts failing | | Platform in offline license key mode | Re-evaluated locally every minute | None needed, the key is in the environment | Until the key expires, plus a 30-day grace period | | Cluster, online | At startup, and whenever the credentials Secret changes | `-features-list` Secret in the host namespace, encrypted | 7 days | | Standalone deployment, online type | At startup, and whenever the credentials Secret changes | `platform/feature-cache.json` in the data directory | Doesn't expire | | Standalone type | Polled with backoff starting at 30 seconds, capped at 5 minutes | None | Not applicable | | Cluster, in-cluster API | Polled with backoff starting at 30 seconds, capped at 5 minutes | None | Not applicable | | Cluster, offline | Re-checked locally every hour | None needed, the key is in the environment | Until the key expires | The online cluster path isn't polled. The control plane retrieves features at startup and then only when the credentials Secret changes. A license change made in the platform reaches a running cluster on the next control plane restart, not on a timer. The platform's one-minute cycle in offline license key mode doesn't contact anything. It re-reads current resource usage, so limits are enforced against live counts. The cluster feature cache is encrypted with a key derived from the vCluster Service UID. A tampered Secret therefore can't grant features the platform never returned. ## What happens when licensing is unavailable[​](#what-happens-when-licensing-is-unavailable "Direct link to What happens when licensing is unavailable") Behavior depends on which component lost access and for how long. | Component / scenario | What happens | | --------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The platform can't reach the license server | Keeps running on the cached license and retries every 6 hours. Exits if no retrieval succeeds within 7 days of the last success (a restart mid-outage gets up to 6 hours less, since the measurement starts from a trailing timestamp). A platform that has never retrieved a license has nothing to fall back on, so it exits immediately rather than starting unlicensed. | | The platform's offline license key is near or past expiry | Raises an expiration warning starting 90 days before expiry. After it expires, a 30-day grace period follows during which the platform keeps working, then the instance blocks all create requests once that period ends. | | The platform is over a resource limit | Blocks creating more of that resource until usage drops below the limit; deleting enough of the resource clears the block automatically. In offline license key mode, the platform evaluates limits itself and warns once usage passes 90%. In online mode, the license server decides which limits apply and which announcements to return. | | A cluster on the online type can't reach the platform | A control plane that's already running is unaffected, since the online path holds its feature set in memory and doesn't poll. At startup or a credentials-triggered reload, it tries the platform first and falls back to the cache. A cache older than 7 days, or a control plane that has never retrieved features at all, both fail initialization rather than starting unlicensed. | | A cluster on the standalone or in-cluster API type can't reach the platform | Unaffected, since these types read `Feature` resources from the control plane cluster rather than from the platform. What changes their feature set is the platform updating those resources, which a disconnected platform stops doing. On the standalone type, Enterprise features stay enabled either way, so only the unlicensed-use warning changes. | | A cluster's own offline license key expires | The control plane warns for the last 30 days, then exits once the key expires. This path has no grace period, unlike the platform's offline key mode. | | The licensed feature set changes | The platform restarts itself to apply the new set. | Expiration, limit, and feature warnings surface in the platform UI. See [Inbox](https://vcluster.com/docs/platform/administer/inbox.md). ## Inspect and debug licensing[​](#inspect-and-debug-licensing "Direct link to Inspect and debug licensing") ### In the platform[​](#in-the-platform "Direct link to In the platform") View the current license in the platform UI under **Admin > Platform Config > License and Billing**. Retrieve the same data through the API. This reports the license the platform currently holds, without contacting the license server. Read the platform's current license ``` kubectl get licenses.management.loft.sh license -o yaml ``` A license the platform holds shows a non-empty `status.license.entity` (your organization's name) and a `status.license.plans` list. An empty or missing `status.license` means the platform has never successfully retrieved one. To make the platform retrieve a fresh license first, add the `extended=true` query parameter. It's a URL parameter rather than a field, so `kubectl get` can't set it and the request has to go through `--raw`. Force a license refresh, then read the result ``` kubectl get --raw '/apis/management.loft.sh/v1/licenses/license?extended=true' ``` The response is the same `License` object as above, so check it against the same `entity` and `plans` fields. If the platform can't reach the license server, this request fails with an error instead of returning the cached license. Drop `extended=true` and re-run the first command to read the cached license without forcing a refresh. Confirm which license server the platform is using. The platform logs this once at startup, so this also tells you which mode it's in. Check the configured license server ``` kubectl logs -n vcluster-platform -l app=loft | grep "Using license server" ``` This logs once per process, on every mode, with the server it's using as the `endpoint` field, for example `admin.loft.sh` in online mode or the in-cluster address in offline license server mode. No match usually means the platform pod restarted since and rotated the line out of its log buffer, not that the platform failed to start. Check the cached license and the instance identity Secrets. Inspect licensing Secrets ``` kubectl get secret loft-license-cache -n vcluster-platform -o jsonpath='{.data.updatedAt}' | base64 -d kubectl get secret loft-cert -n vcluster-platform ``` Use `updatedAt` to confirm the cache exists, not to work out how much grace period is left. The platform writes the timestamp it held *before* the current retrieval, so the stored value trails the real one by a refresh cycle. After the very first write it reads `0001-01-01T00:00:00Z`, because there was no previous retrieval to record. A missing `loft-cert` Secret means the platform generated a new identity, as described in [Instance identity](#instance-identity) above, and any existing license no longer matches it. Grace period is measured from the stored value On restart, the platform reloads `updatedAt` and measures the 7-day grace period from it. Because the value trails, a restart during a license server outage gets up to 6 hours less grace than the constant implies. A `0001-01-01T00:00:00Z` value is handled safely. The platform treats it as unset and falls back to the Secret's creation timestamp, so the grace period still runs from roughly the first successful retrieval. Tracked as [ENGPLAT-1069](https://linear.app/loft/issue/ENGPLAT-1069). Look for retrieval failures in the platform logs. Each of these lines carries a full stack trace in an `errorVerbose` field, so truncate them to keep the output readable. Check for license retrieval errors ``` kubectl logs -n vcluster-platform -l app=loft | grep -i "error retrieving license" | cut -c1-260 ``` An occasional failure here normally needs no action. The platform keeps serving from its cached license and retries on the next cycle. Outside the restart and timestamp edge cases above, only failures spanning more than 7 days make the platform exit. ### In a cluster[​](#in-a-cluster "Direct link to In a cluster") If you've just fixed credentials or the platform now grants a different feature set, restart the control plane before checking anything below. The online path only re-reads features at startup or when the credentials Secret changes, not on a timer, so a fix that's correct won't show up until then. Confirm which license type the control plane selected, and which features it enabled. Modify the following with your specific values to generate a copyable command: NAMESPACEmy-namespace NAMEmy-vcluster Check the cluster's license type and features ``` kubectl logs -n my-namespace my-vcluster-0 | grep -A5 "detected license type\|Enabled features" ``` A healthy online cluster logs the host it connected to, followed by the features it enabled. Startup log on a licensed control plane ``` loader/loader.go Using online license with host loft.vcluster-platform online/license.go Enabled features: online/license.go dra-sync ``` Confirm the credentials Secret and the feature cache both exist. Modify the following with your specific values to generate a copyable command: NAMESPACEmy-namespace NAMEmy-vcluster Verify the credentials Secret and feature cache ``` kubectl get secret vcluster-platform-api-key -n my-namespace kubectl get secret my-vcluster-features-list -n my-namespace ``` A `NotFound` error on the credentials Secret means the cluster was never connected to the Platform, or points at a different Secret name or namespace than configured. The feature cache is encrypted and can't be read directly, so its presence and age are what matter rather than its contents: a missing cache means the control plane has never successfully retrieved features, and the cache expires 7 days after it was last written. A control plane can crashloop with an error naming a feature the license doesn't allow. The cause is usually a missing credentials Secret rather than a plan limitation. See [Resolve Pro feature license errors](https://vcluster.com/docs/vcluster/troubleshoot/pro-feature-license-error.md). ## Related resources[​](#related-resources "Direct link to Related resources") * [Restore license](https://vcluster.com/docs/platform/maintenance/backup-restore/preserve-license.md) * [Air-gapped installation with an offline license key](https://vcluster.com/docs/platform/install/air-gapped.md) * [Air-gapped installation with an offline license server](https://vcluster.com/docs/platform/install/air-gapped/with-offline-license-server.md) * [Platform communication and connectivity](https://vcluster.com/docs/platform/understand/platform_communication.md) * [Inbox](https://vcluster.com/docs/platform/administer/inbox.md) * [Resolve Pro feature license errors](https://vcluster.com/docs/vcluster/troubleshoot/pro-feature-license-error.md) --- # Platform communication and connectivity vCluster Platform communicates with connected clusters through the vCluster Platform agent. The agent runs in each connected cluster, authenticates to Platform with cluster-specific credentials, and maintains an outbound connection back to the Platform endpoint. This egress-only model lets Platform reconcile resources and proxy requests without requiring inbound access to the connected cluster by default. For the full management plane, connected cluster, lifecycle, and network path overview, see [Platform architecture](https://vcluster.com/docs/platform/introduction/architecture.md) and [How Platform works](https://vcluster.com/docs/platform/introduction/how-platform-works.md). ## Communication model[​](#communication-model "Direct link to Communication model") When you connect a cluster, Platform creates a `Cluster` resource and issues connection credentials for the agent. The agent uses those credentials to register with Platform, then participates in Platform's Tailscale-based network so that Platform can reach cluster-side APIs and services through the agent. Platform uses this communication model for: * Deploying, upgrading, and reconciling Platform-managed tenant clusters * Applying cluster-side resources such as namespaces, RBAC, Helm releases, secrets, quotas, and sleep mode components * Reporting connected cluster and tenant cluster status back to Platform * Proxying Kubernetes API requests when users connect through Platform instead of a direct or regional endpoint * Supporting operational workflows such as logs, shell access, status checks, and agent upgrades If the agent disconnects, existing workloads and tenant cluster control planes keep running in the connected cluster. Platform can't reconcile changes or serve proxied access until it reconnects. See [What happens during control plane outages?](https://vcluster.com/docs/platform/understand/control-plane-outages.md) for the full breakdown of what stays available at each layer. Platform communication paths for connected clusters ## Connection state[​](#connection-state "Direct link to Connection state") Platform tracks each connected cluster's agent tunnel as a `NetworkPeer` object, and reflects reconcile progress on the `Cluster` resource's `status.phase`: empty while the agent hasn't connected yet, `Initialized` once a reconcile pass completes without a hard error, or `Failed` if one hit a hard error. `Initialized` alone doesn't mean the connection is healthy. A pass that finds the agent pod unhealthy still completes without error, so `status.phase` becomes `Initialized` with a non-empty `status.reason` describing the problem. `status.reason` and `status.message` carry a short code and the underlying error for whichever step failed, so check them alongside `status.phase` rather than treating `Initialized` alone as a clean bill of health. The platform UI's **Infrastructure > Control Plane Clusters** view also shows an `Online` state. That state reflects the `NetworkPeer` connection, so it's only meaningful for clusters connected through the agent path (`spec.networkPeer: true`). It doesn't apply to a cluster registered with a direct kubeconfig secret (`spec.config`) or the cluster Platform itself runs on (`spec.local: true`). For a breakdown of `status.reason` values and what to do about each, see [Troubleshoot connected cluster connection failures](https://vcluster.com/docs/platform/troubleshoot/connected-cluster-connectivity.md). ## Agent path and proxy path[​](#agent-path-and-proxy-path "Direct link to Agent path and proxy path") Two related paths are involved in Platform communication: | Path | Used for | Connectivity required | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Agent path | Platform lifecycle and reconciliation work, including tenant cluster creation, updates, upgrades, status reporting, and cluster-side resource sync. | The agent must make outbound HTTPS connections to the Platform endpoint. The connected cluster doesn't need inbound access from Platform. | | Platform proxy path | User and automation requests that go through Platform, such as kubeconfigs that use `/kubernetes/cluster/` or `/kubernetes/project//virtualcluster/`. | The user or automation must reach the Platform endpoint. Platform then forwards the request through the appropriate backend, often through the connected cluster agent. | Regional cluster endpoints and tenant cluster access points can shorten the request path for `kubectl` traffic. Those features are optional and don't replace the agent path Platform uses for lifecycle management. See [Regional Cluster Endpoints](https://vcluster.com/docs/platform/administer/clusters/advanced/regional-cluster-endpoints.md) and [ingress suffix configuration](https://vcluster.com/docs/platform/administer/clusters/advanced/ingress-suffix.md). ## Tailscale-based network path[​](#tailscale-based-network-path "Direct link to Tailscale-based network path") Platform uses Tailscale networking components internally for the agent connection. Platform acts as the coordination point for its network peers. Agents don't need to contact Tailscale's hosted coordination service. By default, the connected cluster agent uses this Platform-coordinated network path after it is installed. The agent attempts a direct WireGuard peer-to-peer connection when possible. If direct connectivity is blocked by NAT, firewall rules, or network policy, traffic falls back to DERP. DERP is a relay protocol for encrypted WireGuard packets. Platform includes a central DERP relay by default. You can add connected clusters as self-hosted DERP relays to reduce latency or improve redundancy. See [DERP relay configuration](https://vcluster.com/docs/platform/administer/clusters/advanced/derp-relay.md). note Tenant cluster control plane pods establish their own TSNet connections to Platform for license validation and feature activation, independent of the agent. Each pod registers with the Platform coordinator and probes `/derp/probe` to confirm DERP reachability before reporting ready. This applies to any tenant cluster using the Pro image, which is the chart default. If this connection fails, see [Resolve TSNet connection failures](https://vcluster.com/docs/platform/troubleshoot/tsnet-connectivity.md). ## Required outbound access[​](#required-outbound-access "Direct link to Required outbound access") For the default egress-only agent model, allow the Platform agent to make the following outbound connections from each connected cluster's agent namespace: | Destination | Protocol and port | Purpose | | ---------------------------------------------------------- | -------------------------------------------- | ----------------------------------------------------------------------- | | Platform public URL or load balancer | HTTPS, usually TCP `443` | Agent registration, coordination, API calls, and DERP fallback traffic. | | Platform pods, when Platform and the agent share a cluster | HTTPS TCP `10443` | In-cluster path that avoids the external Platform URL. | | DNS resolver | UDP/TCP `53`, or your environment's DNS path | Resolves the Platform endpoint and any configured DERP relay endpoints. | When Platform and the agent share a cluster, the Platform Kubernetes service routes to port `10443` on the Platform pods. If you use Kubernetes NetworkPolicy, allow egress from the agent namespace to Platform pods on port `10443`. See [Networking](https://vcluster.com/docs/platform/administer/clusters/advanced/networking.md). If Platform is exposed through an ingress controller, gateway, proxy, or service mesh, the route must support long-lived requests and protocol upgrades used by the agent network path. For Istio, see [Expose vCluster Platform with Istio](https://vcluster.com/docs/platform/how-to/exposing-vCluster-platform-with-Istio.md#enable-tailscale-connectivity). You don't need to allow inbound traffic from Platform to connected clusters for the default agent connection. Inbound access is only required when you expose optional regional cluster endpoints, tenant cluster access points, or self-hosted DERP relay endpoints. ## Air-gapped and restricted networks[​](#air-gapped-and-restricted-networks "Direct link to Air-gapped and restricted networks") Platform's Tailscale-based communication can run without internet access to Tailscale services because Platform provides the coordination endpoint and default DERP relay. In air-gapped or restricted environments, make sure: * Agents can resolve and reach the Platform URL configured for the installation. * Firewalls, network policies, ingress controllers, and service meshes allow HTTPS and WebSocket-style upgrade traffic to the Platform endpoint. * The Platform endpoint exposes the DERP and coordinator paths used by agents, including `/derp/probe`. * Any additional DERP relay endpoints are reachable by the peers that need to use them. If you intentionally enable public Tailscale DERP regions with `LOFT_USE_TAILSCALE_DERP=true`, agents also need egress to those public DERP endpoints. Leave that option disabled for fully air-gapped environments. ## Troubleshoot connectivity[​](#troubleshoot-connectivity "Direct link to Troubleshoot connectivity") If a connected cluster or tenant cluster cannot communicate with Platform: 1. Confirm the agent pod is running in the connected cluster. 2. Confirm the agent can resolve and reach the Platform URL over HTTPS. 3. Check whether your ingress, gateway, proxy, or service mesh supports the required upgrade traffic. 4. Check Platform and agent logs for DERP or TSNet errors. 5. If direct peer-to-peer connectivity is unreliable, rely on the Platform DERP relay or configure additional DERP relays. For detailed debugging steps, see [Troubleshoot connected cluster connection failures](https://vcluster.com/docs/platform/troubleshoot/connected-cluster-connectivity.md) for agent connection issues, or [Resolve TSNet connection failures](https://vcluster.com/docs/platform/troubleshoot/tsnet-connectivity.md) for tenant cluster TSNet issues. --- # What are numeric input ranges in app templates? ## What are template parameters?[​](#what-are-template-parameters "Direct link to What are template parameters?") App templates use parameters to make them customizable. When users deploy resources based on these templates, they provide values for these parameters: * Parameters are defined as yaml data structures * Each parameter controls a specific aspect of the deployed resource * Parameters can have constraints like minimum/maximum values * These constraints ensure users provide appropriate values ## Available parameter constraints[​](#available-parameter-constraints "Direct link to Available parameter constraints") Templates support various parameter types and constraints: ![Available parameter types](/docs/assets/images/parameter-types-22f948a469b639e9b4dddbe667575335.png) For numeric parameters (integers and floating-point numbers), you can set additional constraints: ![Advanced parameter constraints](/docs/assets/images/parameter-constraints-517b81d79f24a6fb16b6bfeed7fbb3e3.png) The minimum/maximum feature is particularly useful for controlling resource allocation. ## Set numeric ranges in the UI[​](#set-numeric-ranges-in-the-ui "Direct link to Set numeric ranges in the UI") To define a parameter with a valid numeric range: 1. **Navigate to the template creation page**. Access the template creation interface in the platform. 2. **Add a parameter with number type**. Add a new parameter to your template and select "number" as the parameter type. 3. **Define minimum and maximum values**. Enter the lower and upper bounds for your numeric parameter. These values create the acceptable range for user input. 4. **Add a descriptive explanation**. Write a clear description that explains the purpose of these limits and what the parameter controls. When users deploy with your template, a numeric input field that enforces these limits is displayed. The platform actively validates user input against your defined range. If a value is outside this range, the user cannot proceed with creating the resource. ## Define ranges in yaml[​](#define-ranges-in-yaml "Direct link to Define ranges in yaml") You can also define numeric ranges directly in the template yaml under `spec.parameters`: ``` parameters: - variable: valuecheckparameter label: testlabel description: checks value range type: number min: 3 # minimum allowed value max: 9 # maximum allowed value ``` This configuration: * Creates a parameter named `valuecheckparameter`. * Displays **testlabel** in the UI. * Allows only values between 3 and 9 (inclusive). * Prevents deployment if the value is outside this range. ## Use cases for numeric ranges[​](#use-cases-for-numeric-ranges "Direct link to Use cases for numeric ranges") Setting numeric ranges is valuable for: * **Resource allocation**: Ensuring tenant clusters receive sufficient but not excessive resources. * **Configuration safety**: Preventing users from entering values that could cause performance issues. * **Standardization**: Maintaining consistent deployments across your organization. * **User guidance**: Helping users understand appropriate values for different parameters. ## Next steps[​](#next-steps "Direct link to Next steps") After implementing numeric ranges in your templates, consider: * Adding validation for other parameter types. * Creating templates with sensible defaults. * Documenting the meaning of each parameter for your users. For more information on other parameter types, including selection from predefined options, see the [template parameters documentation](https://vcluster.com/docs/platform/administer/templates/parameters.md). --- # What are Apps | | | Enterprise | | | | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Apps](https://vcluster.com/docs/platform/understand/what-are-apps.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). ## What are Apps?[​](#what-are-apps "Direct link to What are Apps?") Apps in vCluster Platform are a way for admins to package applications and scripts in consumable packages. These Apps can then be deployed into clusters, spaces, or tenant clusters. An example application, and one that comes packaged in a default vCluster Platform deployment, is the 'Cert Manager' app. This app, as you may guess from the name, deploys the [cert-manager](https://github.com/cert-manager/cert-manager) controller into a cluster, space or tenant cluster. Apps can be an existing or custom (user provided) Helm chart, a Kubernetes manifest, or a bash script that installs resources into a given destination (cluster, space, or tenant cluster). Importantly, Apps can have *parameters*, that is, values that a user can provide to modify the deployment of an application. An overly simple and silly App example, could be an app that simply installs a ConfigMap into the given destination. This app could accept input parameters from a user to modify the contents of the ConfigMap. For example, the ConfigMap could contain a data field 'env', with a default value of 'dev'. An administrator could configure the app to accept a parameter called 'env' that has three valid values 'dev', 'test', 'stage'. When a user deploys this app, they will be prompted to select a value from the administrator provided values for this 'env' parameter. The administrator can then refer to this parameter in the ConfigMap manifest, properly setting the environment based on the user input. This is a simple, contrived, example of an app, but they can be much more elaborate and powerful. Apps can give administrators a very flexible way to constrain what resources can be deployed into clusters, while still affording users some flexibility by exposing parameters to customize the app deployment. An App is independently reconciled. When several applications need dependency ordering, health gates, or values from earlier applications, group them in a [Stack](https://vcluster.com/docs/platform/understand/what-are-stacks.md). A Stack task can create an AppInstance from an existing App. If an app instance doesn't deploy the way you expect, see [App sync failures](https://vcluster.com/docs/platform/troubleshoot/template-sync-failures.md#app-sync-failures). --- # What are Connected Clusters A connected cluster is a Kubernetes cluster linked to vCluster Platform. It serves as the Control Plane Cluster where tenant cluster control planes run. Platform deploys, manages, and monitors everything that runs inside it. ## How connection works[​](#how-connection-works "Direct link to How connection works") Connecting a cluster installs the vCluster Platform agent on it. The agent provides the communication layer between Platform and that cluster. It handles tenant cluster provisioning, access control enforcement, lifecycle operations, health monitoring, and proxy path forwarding when Platform routes requests to connected cluster or tenant cluster APIs. It uses an outbound connection to Platform, so no inbound firewall rules are required on the cluster. Once connected, the cluster appears in the Platform UI and can be assigned to projects. Members of those projects can then create tenant clusters and spaces within the cluster, scoped to their access level. ![Host clusters dashboard showing multiple connected clusters with status, Kubernetes version, agent version, and connected time](/docs/assets/images/connected-clusters-agent-status-95cd81a261270eaf513122bd42f7f26b.png) The Host Clusters view shows each connected cluster and the agent details Platform uses to manage it. ## Multi-cluster deployments[​](#multi-cluster-deployments "Direct link to Multi-cluster deployments") Platform can connect to as many clusters as your infrastructure requires. This lets you: * Distribute tenant clusters across regions or cloud providers for latency or data residency requirements * Isolate different environments (production, staging, development) onto separate clusters * Scale capacity by adding clusters without changing how tenants access their environments All connected clusters are managed from one Platform instance. Tenants interact only with their tenant clusters and never have direct access to the connected cluster. ## The cluster where Platform is installed[​](#the-cluster-where-platform-is-installed "Direct link to The cluster where Platform is installed") Platform itself runs on a Kubernetes cluster. That cluster is automatically connected and available as a Control Plane Cluster. For smaller deployments, all tenant clusters can run on the same cluster as Platform. For larger fleets or stricter isolation requirements, dedicated clusters are connected separately. ## Connectivity and resilience[​](#connectivity-and-resilience "Direct link to Connectivity and resilience") The agent maintains a persistent outbound connection to the Platform API server. If connectivity is lost, existing tenant clusters continue running without interruption. Platform resumes full management once the connection is restored. [Connect a cluster →](https://vcluster.com/docs/platform/administer/clusters/connect-cluster.md) [Platform architecture →](https://vcluster.com/docs/platform/introduction/architecture.md) [Troubleshoot connected cluster connection failures →](https://vcluster.com/docs/platform/troubleshoot/connected-cluster-connectivity.md) --- # What Are Namespaces Namespaces are regular Kubernetes namespaces managed by vCluster Platform that are owned by a user or team. In Kubernetes, namespaces provide a mechanism for isolating groups of resources within a single cluster. In vCluster Platform, each namespace belongs to a Control Plane Cluster and project. Namespaces can be [created from a template](https://vcluster.com/docs/platform/use-platform/namespaces/create/create-with-template.md) to enforce isolation best practices or [created manually](https://vcluster.com/docs/platform/use-platform/namespaces/create/create-no-template.md) by project admins. Admins can [manage access](https://vcluster.com/docs/platform/administer/users-permissions/permissions/namespace.md) and users can [connect via CLI](https://vcluster.com/docs/platform/use-platform/namespaces/retrieve-kube-config.md). info For applications that require multiple namespaces or need to create cluster scoped resources, [tenant clusters](https://vcluster.com/docs/vcluster/introduction/what-is-vcluster.md) are recommended instead. There are multiple key features you can use with namespaces: * [Auto Sleep & Auto Delete](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/sleep-mode.md) that puts namespaces to sleep when nobody is using them or delete namespaces that have been idle for a while. * [Isolation](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/isolation.md) that applies ResourceQuotas, LimitRanges, and NetworkPolicies to a Namespace. * [Prevent Deletion](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/prevent-deletion.md) that prevents a Namespace from deletion. * [Custom Links](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/custom-links.md) that adds custom links to namespaces that point to external resources such as GitHub pull requests, Jira issues, or documentation. --- # What are Projects Projects are the primary organizational unit in vCluster Platform. Each project is a policy boundary that groups tenant clusters, spaces, and the users or teams who access them. For AI cloud operators, projects map naturally to customer accounts: each customer gets a project, sees only their resources, and operates within the resource limits you define. For internal platforms, projects map to teams, cost centers, or environments. ![Tenant clusters dashboard with the project filter open, showing multiple projects available for filtering](/docs/assets/images/tenant-cluster-management-dashboard-with-project-filter-1639a2b6413dc27eb0ad0f1c27c34f59.png) Project filters help users move between the tenant clusters they are authorized to access. Platform represents each project as a namespace in the management plane. That namespace holds tenant cluster instances and project secrets, along with reconciled role bindings, access rules, quotas, integrations, and metadata. Project namespaces are internal to Platform and are distinct from namespaces that tenant users create inside a tenant cluster. ## Members[​](#members "Direct link to Members") Every project has members: the users and teams authorized to access it. Only members (and global admins) can see or interact with resources inside a project. Each member is assigned a role that defines what they can do within the project. [Manage project members →](https://vcluster.com/docs/platform/api/resources/project/members.md) ## Allowed clusters[​](#allowed-clusters "Direct link to Allowed clusters") Each project specifies which connected clusters its members can deploy tenant clusters into. This lets you direct different customers or teams to specific infrastructure — for example, routing GPU workloads to a cluster with GPU nodes, or keeping a customer's resources in a specific region. [Configure allowed clusters →](https://vcluster.com/docs/platform/administer/projects/allowed/clusters.md) ## Templates[​](#templates "Direct link to Templates") Projects define which templates members can use when creating tenant clusters. Admins can set a default template, restrict members to an approved set, or allow all templates. This ensures tenant clusters created within the project meet your configuration and security requirements. [Configure allowed templates →](https://vcluster.com/docs/platform/administer/projects/allowed/templates.md) ## Quotas[​](#quotas "Direct link to Quotas") Projects can define resource quotas that cap total compute and storage consumption across all tenant clusters and spaces in the project. Quotas prevent any single project from consuming disproportionate infrastructure resources. [Configure project quotas →](https://vcluster.com/docs/platform/administer/projects/quotas.md) [Enforce storage limits inside a tenant cluster →](https://vcluster.com/docs/platform/how-to/enforce-storage-quotas.md) ## Secrets[​](#secrets "Direct link to Secrets") Projects can store secrets that are shared across resources within the project. These are useful for distributing credentials, tokens, or configuration that multiple tenant clusters need access to. [Manage project secrets →](https://vcluster.com/docs/platform/administer/secrets/project/create.md) ## Integrations[​](#integrations "Direct link to Integrations") Projects support integrations with external tools including ArgoCD and Rancher, allowing you to connect tenant cluster lifecycle management with your existing GitOps or platform tooling. [View integrations →](https://vcluster.com/docs/platform/integrations/argocd.md) [Platform architecture →](https://vcluster.com/docs/platform/introduction/architecture.md) [View resources across all your projects →](https://vcluster.com/docs/platform/use-platform/all-projects.md) --- # What are Secrets ## What are Secrets?[​](#what-are-secrets "Direct link to What are Secrets?") Like [Kubernetes secrets](https://kubernetes.io/docs/concepts/configuration/secret/), vCluster Platform secrets are intended to hold confidential data in the form of key/value pairs. vCluster Platform extends Kubernetes secrets by allowing global or project level management of secret data, managing which users and teams can access secrets, and synchronizing secret data across multiple clusters and the spaces and tenant clusters on those clusters. After creating vCluster Platform secrets, native Kubernetes secrets can be created with labels that indicate to vCluster Platform that the secret data should be synchronized with vCluster Platform secrets. Once this secret synchronization is configured, the secret data can be mounted using the native secret as usual, but managed at the project or global level using vCluster Platform secrets. ## Project Secrets[​](#project-secrets "Direct link to Project Secrets") Project secrets are scoped to a Project, and implicitly only allow access to members of the project. Once a project secret is created, native Kubernetes secrets that synchronize to the project secret can be created in spaces and tenant clusters that belong to the project. This provides a convenient way to manage secret data for all members of the project. For more information on creating project secrets, see [creating project secrets](https://vcluster.com/docs/platform/administer/secrets/project/create.md) ## Global Secrets[​](#global-secrets "Direct link to Global Secrets") Global secrets or shared secrets can be synchronized across all spaces in vCluster Platform registered clusters. Additionally, global secrets can be use to synchronize project secrets. This allows organization wide management of secrets shared across multiple projects. Like project secrets, native Kubernetes secrets can be synchronized directly to global secrets, however this synchronization only works for secrets defined in spaces and not tenant clusters. For more information on creating project secrets, see [creating global secrets](https://vcluster.com/docs/platform/administer/secrets/global/create.md) --- # What are Stacks Stacks let vCluster Platform deploy several applications as one dependency-aware unit. Use them when an environment needs more than a collection of independently installed applications. A Stack can wait for one application to become healthy before it creates another. It can also capture values from one task and pass them to later tasks. Applications without dependencies can deploy concurrently. ## Understand stack resources[​](#understand-stack-resources "Direct link to Understand stack resources") A Stack uses two Platform resources: * A `StackTemplate` is a reusable, parameterized task graph. Platform administrators create these templates and control who can use them. * A `StackInstance` applies an inline graph or a referenced StackTemplate to one tenant cluster or control plane cluster. Each task creates one child resource: * An App task creates an AppInstance from an inline definition or an App reference. * An Argo CD Application task creates an ArgoCDApplication from an inline definition or an ArgoCDApplicationTemplate reference. A Stack can mix both task types. The destination, owner, and access settings come from the StackInstance and propagate to its children. ## Understand task orchestration[​](#understand-task-orchestration "Direct link to Understand task orchestration") The diagram below shows one example task graph. A Stack's applications, task count, and dependencies depend entirely on what your environment needs. In this example, GPU Operator and Ingress start without dependencies. Model Server waits until GPU Operator is healthy, and Monitoring waits on both Model Server and Ingress while reading Model Server's output. Tasks form a directed acyclic graph. A task starts after every task in its `dependsOn` list becomes healthy. Tasks on separate branches can run at the same time. Each task reports its own phase, reason, and child resource name. The StackInstance combines those results into an aggregate phase. This gives users one place to monitor the environment without hiding the status of each application. Tasks can capture scalar values from Secrets or namespaced Kubernetes resources. Later tasks consume those values through the `.Outputs` template context. Template authors can publish selected outputs to Stack users without publishing every captured value. ## Choose an application delivery resource[​](#choose-an-application-delivery-resource "Direct link to Choose an application delivery resource") | Resource | Use it when | | ----------------- | ----------------------------------------------------------------------------------------------------------- | | App | You need one independently managed Helm chart, manifest set, or script. | | ArgoCDApplication | Git is the source of truth and Argo CD continuously reconciles one application. | | Stack | Several Apps or Argo CD Applications need dependencies, health gates, shared parameters, or output handoff. | Argo CD sync waves order resources inside one Argo CD Application. A Stack adds dependency edges between separate applications and can mix App and Argo CD delivery. ## Understand lifecycle and licensing[​](#understand-lifecycle-and-licensing "Direct link to Understand lifecycle and licensing") A referenced StackTemplate is resolved during each reconciliation. StackTemplates don't have Stack revisions. Template changes can therefore affect existing StackInstances. When a template removes a task, the StackInstance prune policy controls its existing child. `Retain` is the default and keeps the child. `Prune` deletes it in reverse dependency order. Stacks don't have a separate license feature. App tasks use the [Apps](https://vcluster.com/docs/platform/understand/what-are-apps.md) feature. Argo CD Application tasks also require the [Argo CD Integration](https://vcluster.com/docs/platform/integrations/argocd.md) feature. Both are available on every plan, including the free plan. Certified Stacks use the same resources and controller. vCluster Platform bundles certified StackTemplates and their supporting resources for tested integrations such as NVIDIA Run:ai. See [Certified Stacks integrations](https://vcluster.com/docs/platform/integrations/certified-stacks/.md) for the full list. ## Next steps[​](#next-steps "Direct link to Next steps") * [Create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) to define tasks, dependencies, and outputs. * [Use a Stack](https://vcluster.com/docs/platform/use-platform/apps/use-stacks.md) to install a StackTemplate. * [Certified Stacks integrations](https://vcluster.com/docs/platform/integrations/certified-stacks/.md) to browse the bundled, tested StackTemplates. * [Manage Certified Stacks](https://vcluster.com/docs/platform/administer/templates/certified-stacks.md) to deploy the bundled NVIDIA Run:ai integration. * [Stack permissions](https://vcluster.com/docs/platform/administer/users-permissions/permissions/stacks.md) to configure RBAC for Stack resources. * [StackTemplate](https://vcluster.com/docs/platform/api/resources/stacktemplate.md) and [StackInstance](https://vcluster.com/docs/platform/api/resources/stackinstance.md) API references for the full field schema. * [Troubleshoot Stacks](https://vcluster.com/docs/platform/troubleshoot/stacks.md) to interpret Stack and task status. --- # What are Templates Templates define a baseline configuration applied to every tenant cluster or space created from them. They are how Platform administrators enforce consistency across a fleet without requiring each user to configure resources manually. Templates solve a governance problem at scale. Instead of trusting users to apply the right security settings, resource limits, and networking policies, you encode those requirements into a template and make it the only option available in a project. ## What templates can include[​](#what-templates-can-include "Direct link to What templates can include") * A base `vcluster.yaml` configuration defining the tenant cluster's control plane, worker node type, backing store, and sync settings * Resource quotas, limit ranges, and network policies for namespace-level isolation * Kubernetes objects deployed into the tenant cluster at creation time, such as tooling, pre-populated test data, or CRDs * [Apps](https://vcluster.com/docs/platform/understand/what-are-apps.md) — packaged applications deployed into the tenant cluster when it is created * [Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md) — dependency-aware groups of Apps and Argo CD Applications * Parameters that let users customize within defined bounds at creation time ## How templates work with projects[​](#how-templates-work-with-projects "Direct link to How templates work with projects") Templates are created at the global level by Platform administrators. To make a template available to project members, it must be explicitly allowed within that project. Admins can set a default template for a project, restrict members to a specific set, or allow all templates. This two-level model (global definition, per-project access) means you can maintain a library of approved configurations and expose only the relevant ones to each team or customer. [Create a template →](https://vcluster.com/docs/platform/administer/templates/create-templates.md) [Create a Stack template →](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) [Allow templates in a project →](https://vcluster.com/docs/platform/administer/templates/create-templates.md#management-access) --- # What are Users and Teams Users and teams are the core identities in vCluster Platform. A user represents an individual or service identity that can authenticate to the platform. A team represents a group of users that should receive the same access, ownership, and platform-level configuration. They are used throughout the platform, not just during login. Users and teams can receive management roles, become project members, own resources, and consume quota. They can also receive image pull secrets and appear as Kubernetes subjects in reconciled role-based access control (RBAC) objects that Platform automatically creates and keeps up to date. ## Users[​](#users "Direct link to Users") A user is the platform record for one authenticated identity that can be human or automated. Platform tracks the identifiers needed to recognize that identity, including a stable platform name, an SSO-matching subject, and the groups associated with it. Beyond identity, a user record can also grant management roles directly and hand out registry credentials for CLI-based image pulls. It can also be disabled to block login without deleting the user or its resources. Platform also computes each user's current effective team membership. Effective means the fully resolved result right now, not the raw configuration. This lets admins see which teams a user actually belongs to at this moment, not just which teams were configured to include them. See the [User resource reference](https://vcluster.com/docs/platform/api/resources/user.md) for the full list of fields. [Create users →](https://vcluster.com/docs/platform/administer/users-permissions/managing-users/create.md) ## Teams[​](#teams "Direct link to Teams") A team is the platform record for a shared identity boundary. Teams do not log in on their own. Instead, they become part of an authenticated user's effective identity when that user is a member. Membership comes from two places. Some users are explicitly assigned to the team, and others belong because their groups match a token or SSO group name configured on the team. This lets you combine manual team assignment with identity-provider-driven membership. For example, you might add a break-glass admin user (an emergency-access account for when normal login isn't available) directly to a team. The rest of the team's membership can then come from an SSO group, such as OIDC, SAML, GitHub, GitLab, Google, Microsoft, LDAP, or Dex. Like users, teams can be granted management roles and image pull secrets directly. Those apply to whichever users are currently members of the team. See the [Team resource reference](https://vcluster.com/docs/platform/api/resources/team.md) for the full list of fields. [Create teams →](https://vcluster.com/docs/platform/administer/users-permissions/managing-teams/create.md) For a user login or user access key, Platform resolves team membership from explicit assignment and matching SSO groups. ## SSO and account provisioning[​](#sso-and-account-provisioning "Direct link to SSO and account provisioning") With SSO, Platform can create or update users from identity provider claims. The important design choice is usually where to manage group membership: * Manage membership in the identity provider when your organization already has authoritative groups. * Manage membership in Platform when teams are platform-specific or short-lived. * Use both when you need exceptions, such as manually adding a service user to a team whose human members come from SSO. SSO configuration can also do more than manage membership. It can apply cluster account templates (predefined access configurations for connected clusters) to users when they first sign in. It can also keep those templates synchronized when tokens refresh. This is useful when users or teams should automatically receive access to particular connected clusters after authentication. [SSO configuration →](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md) ## How effective identity works[​](#how-effective-identity-works "Direct link to How effective identity works") After a user authenticates, Platform builds a Kubernetes-compatible identity for that request. The resulting identity includes: * the user's subject as the Kubernetes user name; * the user's direct groups; * `system:authenticated`; * platform groups such as `loft:user:`; * one `loft:team:` group for every effective team membership. This is why teams are useful beyond the UI. Platform permissions, project membership, admission policies, and tenant cluster access can all reason about users and teams as Kubernetes-style subjects and groups. [Authentication and authorization →](https://vcluster.com/docs/platform/understand/auth-explanation.md) ## Permissions and roles[​](#permissions-and-roles "Direct link to Permissions and roles") Platform uses Kubernetes RBAC concepts to authorize users and teams. A management role defines allowed actions, and Platform applies that role to one or more users or teams through reconciled role bindings. Grant roles to teams when many people should share the same access. Grant roles directly to users only for exceptions, such as platform administrators, automation users, or temporary troubleshooting access. Direct user roles and team-inherited roles are additive, so a user can receive permissions from several places at once. [Users and permissions overview →](https://vcluster.com/docs/platform/administer/users-permissions/overview.md) ## Project membership[​](#project-membership "Direct link to Project membership") Projects are where users and teams usually become operational. A project's member list can include specific users, specific teams, or all users at once. Each member receives a project role that defines what they can do inside that project. When a team is a project member, all current team members receive the team's project access. Platform's project members API resolves both the direct team entry and the users who belong to that team. This lets project views show the effective member set. Project ownership is related but separate. A project can have `spec.owner.user` or `spec.owner.team`, and the owner is included in the resolved project members list. Owners and members are often the same organizational unit, but they answer different questions. Ownership says who the project belongs to. Membership says who can access it and with which role. Project membership grants access with roles; project ownership identifies the user or team the project belongs to. [Manage project members →](https://vcluster.com/docs/platform/api/resources/project/members.md) [What are Projects →](https://vcluster.com/docs/platform/understand/what-are-projects.md) ## Resource ownership[​](#resource-ownership "Direct link to Resource ownership") Many platform resources can be owned by either a user or a team. Platform records that ownership through owner fields on the resource and through ownership labels applied to resources such as namespaces, space instances, tenant cluster instances, and access keys. Ownership affects how resources are displayed, who can manage them, how project quota is counted, and how sharing workflows behave. Use team ownership for shared environments and long-lived resources that should survive changes in individual staffing. Use user ownership for personal development environments, individual access keys, or short-lived work that should clearly belong to one person. [Create global secrets →](https://vcluster.com/docs/platform/administer/secrets/global/create.md) [SSH keys →](https://vcluster.com/docs/platform/use-platform/machines/ssh-keys.md) [Platform labels and annotations →](https://vcluster.com/docs/platform/reference/platform-annotations.md#user-team-management) ## Quotas[​](#quotas "Direct link to Quotas") Project quotas can apply to the whole project or to each user or team inside the project. A project-wide quota caps the total resources that the project can consume across all members. A per-user/team quota caps the resources that a single user or team can consume. Ownership determines which quota a resource's usage counts against. A team-owned resource counts against that team's quota, and a user-owned resource counts against that user's quota, even when the user also belongs to a team. Team and individual usage are tracked separately, so a team-owned resource never counts against a member's personal quota, and a member's personally owned resource never counts against the team's quota. [Configure project quotas →](https://vcluster.com/docs/platform/administer/projects/quotas.md) ## Automation and service accounts[​](#automation-and-service-accounts "Direct link to Automation and service accounts") As the Users section noted, a user or team doesn't need a human behind it. For GitOps, CI/CD, or external automation, create a dedicated user and issue an access key for that user. The access key authenticates as that user, and any access key scope can only narrow what the user is already allowed to do. Access keys can also authenticate as a team. A team access key acts as the team identity, so it receives the team's roles and project memberships rather than a particular user's direct roles. Use this when automation should belong to a shared operational identity instead of an individual user. [Access keys →](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) [GitOps end-to-end guide →](https://vcluster.com/docs/platform/how-to/gitops-end-to-end.md) ## Access troubleshooting[​](#access-troubleshooting "Direct link to Access troubleshooting") When troubleshooting access, check the actor first. Is the request using a normal user login, a user access key, or a team access key? Then review the user's teams, direct roles, team roles, project memberships, resource ownership, and any access key scope that might be narrowing the request. When changing someone's access, check which actor is authenticated, then review roles, project membership, ownership, and any access key scope. ## Access modeling[​](#access-modeling "Direct link to Access modeling") For most installations, start with teams as the primary unit of access. Create teams that match how work is owned, such as platform administrators, application teams, customer accounts, environments, or automation roles. Add those teams to projects, grant project roles to teams, and use direct user permissions only where the exception is intentional. Use user-owned resources for personal workspaces and use team-owned resources for shared services. This keeps access, ownership, and quota behavior predictable as people join, leave, or move between teams. --- # View resources across projects By default, every view in vCluster Platform is project-scoped. When you are working inside a project, the tenant clusters, namespaces, machines, and secrets you see belong only to that project. This is by design. Projects are isolation boundaries, and members of one project cannot see another project's resources. For users managing a single project, this is exactly the right behavior. For users managing multiple projects, switching context to check each one separately creates friction. Those projects might represent teams, environments, or customer accounts. The All Projects view removes that friction by surfacing resources from all active projects you can access in a single table. All Projects doesn't bypass project permissions. It shows only the projects and resources your account can access. If you support multiple customers, each mapped to a project, All Projects gives you a single place to check status and find the resources that need attention. ![All Tenant Clusters view with the All Projects mode active, showing the globe icon and All Projects label in the project switcher, along with the tenant cluster table and project and status filter controls](/docs/assets/images/all-projects-selected-fe3d24102a012c2a85d59503495b4ed2.png) All Tenant Clusters in All Projects mode. The project filter and status filter let you narrow the table without leaving the cross-project context. ## Open cross-project views[​](#open-cross-project-views "Direct link to Open cross-project views") In the left navigation, click the globe icon (**All Projects**) in the project switcher. The sidebar changes from project-specific navigation to cross-project navigation. The project switcher is a split control. The globe icon on the left represents All Projects. The folder icon and project name on the right open a dropdown to select a specific project. Clicking the globe icon from any project view switches to All Projects and opens the equivalent section. For example, clicking from a project's Tenant Clusters page opens All Tenant Clusters. Clicking from a section that has no All Projects equivalent (such as Secrets) opens All Tenant Clusters by default. Clicking the right side opens a project list. Selecting a project from the list switches to that project and opens the section that corresponds to wherever you are in All Projects. For example, selecting a project while viewing All Namespaces opens that project's Namespaces page. ## Cross-project views[​](#cross-project-views "Direct link to Cross-project views") All Projects provides four views. Each table includes a project column and filters so you can scope the list without leaving the cross-project context. Visible table columns are configurable by each user, so the columns you see might differ from another user's view. | View | Use it to | | ------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | [All Tenant Clusters](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-no-template.md) | Review tenant clusters, their status, project, template, owner, version, and control plane placement. | | [All Namespaces](https://vcluster.com/docs/platform/use-platform/namespaces/create/create-no-template.md) | Review namespaces, their status, project, template, owner, and control plane cluster. | | [All Machines](https://vcluster.com/docs/platform/use-platform/machines/overview.md) | Review machines, their project, assigned tenant cluster, infrastructure provider, network environment, and operating system image. | | [All Network Environments](https://vcluster.com/docs/platform/use-platform/machines/overview.md) | Review network environments, their project, provider, owner, and default status. | Network Environments is a sub-view of Machines, accessible from the Machines navigation item. To make changes to a resource, such as upgrading, deleting, or changing its configuration, click it to open the project-scoped detail page. The sidebar switches to that project automatically. Clicking a project link in a row opens that project's equivalent page. For example, clicking a project link in the All Tenant Clusters table opens that project's tenant cluster list. ## Filter by project[​](#filter-by-project "Direct link to Filter by project") All Projects tables include a project column and project filter. Use the project filter to narrow a table to one or more projects while staying in the cross-project view. ## Unavailable in cross-project views[​](#unavailable-in-cross-project-views "Direct link to Unavailable in cross-project views") The following are only available in project-scoped views: * **Secrets** — project secrets are managed from each project's Secrets page. * **Project settings** — membership, allowed clusters, allowed templates, and quota settings require opening a specific project. To access these, select a project from the project switcher. ## Permissions[​](#permissions "Direct link to Permissions") All Projects follows the same access rules as project-scoped pages: * Users only see active projects where they have access. * Projects pending deletion are not included, even if you could access them before deletion started. * Users only see resources their permissions allow them to list or use. * A user with access to one project can still use All Projects, but the tables and project filter only include that project. * Direct links to project-scoped routes still require access to that project. For more information about project membership and roles, see [Users and permissions](https://vcluster.com/docs/platform/administer/users-permissions/overview.md) and [What are Projects](https://vcluster.com/docs/platform/understand/what-are-projects.md). --- # Updating Apps App templates can be *versioned*. Versioning an App Template allows you to make changes over time to an App while retaining the current version/iteration. Apps that are deployed from a template with updated versions will show up with a warning indicating that a newer App version is available. ### Update an installed app version[​](#update-an-installed-app-version "Direct link to Update an installed app version") You can change the version of an installed App in a space, cluster, or tenant cluster to a newer (or older!) version in the vCluster Platform UI. * (Physical) Cluster * Tenant Cluster 1. Go to Infrastructure > Control Plane Clusters. 2. Find the cluster containing the App you would like to upgrade and click on that cluster. 3. In the Cluster Management view, click the Apps button. 4. Find the App you would like to upgrade (or change the version of) in the list of installed Apps. Apps that are outdated (meaning deployed with a version less than the latest version of the App) will show up with Outdated in the status column. 5. Either click the Outdated button in the status column of your target App (if the App is outdated), or click the Edit option for the App you want to modify. 6. From the App Version selector, select the desired target App Version. 7. Click the Upgrade button. 1) Select the project containing the tenant cluster using the project selector (top left), then click Tenant Clusters. 2) Find the tenant cluster containing the App you would like to upgrade and click that tenant cluster. 3) In the Tenant Cluster Management view, click the Apps button. 4) Find the App you would like to upgrade (or change the version of) in the list of installed Apps. Apps that are outdated (meaning deployed with a version less than the latest version of the App) will show up with Outdated in the status column. 5) Either click the Outdated button in the status column of your target App (if the App is outdated), or click the Edit option for the App you want to modify. 6) From the App Version selector, select the desired target App Version. 7) Click the Upgrade button. --- # Use In Templates Apps are a way for admins to package applications and scripts in consumable packages. These applications can then be deployed into namespaces or tenant clusters through their templates. Tenant clusters or namespaces combined with Apps are a great way to create repeatable, pre-packaged development, testing, or even production environments. Once an App has been created in the vCluster Platform instance, it can be installed into any tenant cluster or namespace -- either at time of creation, or into existing resources. This gives administrators the capability to create fully populated development, test, or even production, environments in a single easy to manage package. Templates can take this combination of tenant clusters/namespaces and apps to the next level, by creating simple, reusable, version-able templates that define a reproducible environment.When an App is included in a template, that app will be deployed in the given resource upon creation. 1. Go to Management > Templates and, if shown, click the Tenant Clusters tab. 2. Edit an existing template, or create a new tenant cluster template. 3. Click the Apps configuration tab.

**Note: **the apps defined in this section of the template configuration are apps to be deployed *in* the tenant cluster! 4. Select an App from the Please select an App... selector to include it in the template. You can add as many Apps as you like by selecting another. To remove one, click the trash icon next to the App name. 5. For each App you added to the template, you can configure the default namespace, and any App parameters supplied by the App. For Helm Apps you can also configure the Helm release-name. 6. Click the Space configuration tab. 7. Expand the Space Apps configuration section.

**Note:** the apps defined in this section of the template configuration are apps to be deployed in the *space* the tenant cluster will be deployed in (not inside the tenant cluster itself). 8. Select an App from the Please select an App... selector to include it in the template. You can add as many Apps as you like by selecting another. To remove one, click the trash icon next to the App name. 9. For each App you added to the template, you can any App parameters supplied by the App. For Helm Apps you can also configure the Helm release-name. 10. Finish configuring anything else you'd like on your template, then click the Save Changes button. ## Include Stacks in a tenant cluster template[​](#include-stacks-in-a-tenant-cluster-template "Direct link to Include Stacks in a tenant cluster template") vCluster 0.37 supports Stacks through `deploy.stacks` in the tenant cluster's `vcluster.yaml`. Use a Stack when applications need dependency ordering, health gates, or output handoff. Add the Stack in the template's **Stacks** configuration section or YAML. See [Use a Stack](https://vcluster.com/docs/platform/use-platform/apps/use-stacks.md#choose-how-to-install-a-stack) for the configuration shape. Template authors can also [create a StackTemplate](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) for users to select. --- # Use On-Demand Apps are a way for admins to package applications and scripts in consumable packages. These applications can then be deployed into clusters, namespaces, or tenant clusters managed by vCluster Platform. This can be a useful tool for adding pre-defined applications to newly connected clusters, or to newly created namespaces or tenant clusters that are not already using a template that includes Apps. Use an App for one independently managed application. Use a Stack when several applications need ordering, health gates, or values from earlier tasks. See [Use a Stack](https://vcluster.com/docs/platform/use-platform/apps/use-stacks.md). ## Install apps on a tenant cluster[​](#install-apps-on-a-tenant-cluster "Direct link to Install apps on a tenant cluster") Tenant clusters combined with Apps are a great way to create repeatable, pre-packaged development, testing, or even production environments. Once an App has been created in the vCluster Platform instance, it can be installed into any tenant cluster -- either at time of creation, or into existing tenant clusters. This gives administrators the capability to create fully populated development, test, or even production, environments in a single easy to manage package. * During Tenant Cluster Creation * On Existing Tenant Cluster 1. From the project drop-down menu (top left corner), select the project you'd like to create the tenant cluster in. 2. Click Tenant Clusters. 3. Click the New Tenant Cluster button. Do not select a template. 4. **\[Optional]** Select the cluster in which to create the virtual cluster. 5. Click the Configure to continue. 6. Click the Apps configuration tab.

**Note:** The apps defined in this section of the configuration are apps to be deployed *in* the tenant cluster. 7. Select an application from the Please select an App... selector. You can add as many as you like by selecting another. To remove one, click the trash icon next to the application name. 8. For each application that you added to the template, you can configure the default namespace, and any parameters supplied by the application. For Helm applications, you can also configure the Helm release-name. 9. Finish configuring anything else you'd like on your tenant cluster, then click the Create Tenant Cluster button. 1) 1) Select the project from the project selector at the top left. 2) Click Tenant Clusters. 3) Find the tenant cluster you would like to deploy an App to and hover over it. 4) Click Edit. 5) Click the Apps configuration tab.

**Note:** The apps defined in this section of the configuration are apps to be deployed *in* the virtual cluster. 6) Select an application from the Please select an App... selector. You can add as many as you like by selecting another application. To remove one, click the trash icon next to the application name. 7) For each application that you added to the template, you can configure the default namespace, and any parameters supplied by the application. For Helm applications, you can also configure the Helm release-name. 8) Finish configuring anything else you'd like on your tenant cluster, then click Save Changes. ## Install apps on a cluster[​](#install-apps-on-a-cluster "Direct link to Install apps on a cluster") 1. Go to Infrastructure > Control Plane Clusters. 2. Find the cluster you would like to deploy an App to and click it. 3. In the Cluster Management view, click the Apps button. 4. Select the App you would like to install from the App thumbnails, or, if there are no recommend Apps, click the Install App and select the desired App from the Select an App selector. 5. In the Install App screen enter the namespace to install the App to in the Namespace field. 6. Enter any other required parameters for your App -- this may vary depending on the type of App and whether or not the App creator has specified parameters. 7. Click the Install button. --- # Use Parameters with Apps ## Create an app with parameters[​](#create-an-app-with-parameters "Direct link to Create an app with parameters") Apps with parameters can be created through the vCluster Platform UI. 1. Go to Management > Apps and, if shown, click the Helm Apps tab. 2. Click Create App Template. 3. In the configuration sheet that opens, give your new app a name by replacing the 'my-app' placeholder name, or by updating the manifest YAML 'metadata.name' field. 4. Click the Parameters to access the Parameters configuration section. 5. Enter valid YAML representing the Parameters you would like to use with your App. 6. Make any additional modifications to your App. 7. Click the Create App button. --- # Use a Stack A Stack deploys a group of applications to one tenant cluster or control plane cluster. The StackTemplate controls dependencies, health gates, parameters, and values shared between tasks. Use an App for one independently managed application. Use a Stack when several applications need ordering, health gates, or values from earlier tasks. ## Understand licensing[​](#understand-licensing "Direct link to Understand licensing") Stacks don't have a separate license feature. App tasks use the Apps feature, which is available on every plan. Argo CD Application tasks also require the Argo CD Integration feature, which isn't available on the free plan. If a task isn't licensed, the StackInstance is blocked with the `FeatureNotAllowed` reason. ## Choose how to install a Stack[​](#choose-how-to-install-a-stack "Direct link to Choose how to install a Stack") You can install a Stack three ways: * From the catalog, to add a Stack to a tenant cluster or control plane cluster that already exists. * During tenant cluster creation, when the Stack should exist from the start. * Through `vcluster.yaml`, when you manage the tenant cluster's configuration declaratively, such as through GitOps.
* From the catalog * During tenant cluster creation * vcluster.yaml The current Platform UI shows **Stacks & Apps**. Older Platform versions can show **Apps** instead. 1. Select the project that owns the destination tenant cluster. 2. Go to Stacks & Apps, or open the destination and select its applications view. 3. Select a StackTemplate. Review its description, parameters, and dependency graph. 4. Select the destination tenant cluster or control plane cluster. 5. Enter the required parameters, and click Install. 6. Open the StackInstance and monitor the aggregate status and each task. You need access to the StackTemplate and permission to create StackInstances in the project. Published output values require separate permission to get the `stackinstances/outputs` subresource. See [Stack permissions](https://vcluster.com/docs/platform/administer/users-permissions/permissions/stacks.md). vCluster 0.37 adds a **Stacks** configuration section to tenant cluster creation. The UI hides this section for older vCluster versions. 1. Go to Tenant Clusters, and click New Tenant Cluster. 2. Select a template or configure the tenant cluster. 3. Open the Stacks section, and select a StackTemplate. 4. Set its name and parameters. Add other Stacks when required. 5. Finish the tenant cluster configuration, and click Create Tenant Cluster. The StackInstance is created for the new tenant cluster, and its lifecycle follows the tenant cluster's. Use [`deploy.stacks`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/deploy.md#platform-stacks) to include a Stack in a tenant cluster configuration: vcluster.yaml ``` deploy: stacks: - name: observability # Resolve this cluster-scoped StackTemplate during reconciliation. templateRef: name: observability parameters: retention: 7d # Keep applications if a future template removes their tasks. prunePolicy: Retain ``` Each entry needs a stable `name` and exactly one of `templateRef` or an inline `template`. A referenced StackTemplate is resolved live during reconciliation. You can also set `defaults.taskTimeout` and `prunePolicy`. The default task timeout is 10 minutes. The default prune policy is `Retain`. The destination is always the tenant cluster whose configuration contains the entry. Removing an entry deletes its managed StackInstance, and deleting the tenant cluster deletes all StackInstances it manages. The generated StackInstance name combines the tenant cluster and Stack names and adds a hash suffix when needed to stay within the Kubernetes name limit. The `platform.vcluster.com/stack-config-name` label records the original Stack name. ## Monitor a stack[​](#monitor-a-stack "Direct link to Monitor a stack") The StackInstance aggregate phase summarizes all tasks: | Phase | Meaning | | ------------- | --------------------------------------------------- | | `Pending` | A prerequisite needs manual action. | | `Progressing` | At least one task is rolling out. | | `Healthy` | Every task is healthy. | | `Degraded` | At least one task failed. | | `Deleting` | The Stack and its owned children are being removed. | Tasks can be `Pending`, `Waiting`, `Progressing`, `Blocked`, `Healthy`, or `Failed`. A waiting task hasn't started because one or more dependencies aren't healthy. Inspect status through the management API: ``` vcluster platform connect management kubectl get stackinstance -n \ -o jsonpath='{.status.phase}{"\n"}{range .status.tasks[*]}{.name}{"\t"}{.phase}{"\t"}{.message}{"\n"}{end}' ``` ## Update or remove a Stack[​](#update-or-remove-a-stack "Direct link to Update or remove a Stack") Edit the StackInstance to change its parameters. The controller renders the template again and reconciles its tasks. When a referenced StackTemplate removes a task, `prunePolicy` controls the existing child: * `Retain`, the default, keeps the child and reports it in `status.orphanedApplications`. * `Prune` deletes removed children in reverse dependency order. Deleting a StackInstance removes the applications that it still owns. Application charts can leave persistent volumes, namespaces, hooks, or custom resource definitions behind. Review the application's uninstall behavior before deletion. ## Next steps[​](#next-steps "Direct link to Next steps") * [Create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) to author task graphs and outputs. * [Manage Certified Stacks](https://vcluster.com/docs/platform/administer/templates/certified-stacks.md) to deploy NVIDIA Run:ai. * [Stack permissions](https://vcluster.com/docs/platform/administer/users-permissions/permissions/stacks.md) to configure RBAC for Stack resources. * [StackInstance reference](https://vcluster.com/docs/platform/api/resources/stackinstance.md) for the full field schema. * [Troubleshoot Stacks](https://vcluster.com/docs/platform/troubleshoot/stacks.md) to diagnose blocked or failed tasks. --- # Change your password If you log in to vCluster Platform with a username and password, you can change your password from your Profile page. 1. Click your profile picture and name at the bottom of the sidebar. 2. Click Profile in the menu that appears. 3. In the Basics section, click Change Password. If you haven't set a password before, this button reads Set Password instead, and you won't be asked for your current password. 4. Enter your current password in the Current Password field. 5. Enter your new password in the New Password field, then enter it again in the Repeat New Password field. 6. Click Save. ## Effect on your other sessions and access keys[​](#effect-on-your-other-sessions-and-access-keys "Direct link to Effect on your other sessions and access keys") Changing your password invalidates all of your existing [access keys](https://vcluster.com/docs/platform/administer/authentication/access-keys.md), including: * Browser sessions on other devices or browsers * CLI sessions started with `vcluster platform login` * Any access keys you generated for API or automation use The browser tab where you change your password stays logged in. The platform automatically re-authenticates you with the new password after the change succeeds, so you won't be logged out there. Every other session or access key stops working immediately. To restore access, log in again with your new password, or generate new access keys for any automation that relied on the old ones. info An administrator resetting your password on your behalf has the same effect once the new password is set. See [Reset a user's password](https://vcluster.com/docs/platform/administer/users-permissions/managing-users/reset-password.md). --- # Machine Config Templates A `MachineConfigTemplate` is a reusable, cluster-scoped resource that holds a Go template for either a cloud-init (user data) document or a network-data document. A node provider renders the template when it provisions a Machine, so you can define provisioning configuration once and reference it by name from any node type or Machine instead of inlining it on each one. The platform ships with a few predefined templates that you can use as a starting point. A template defines a `cloudInitTemplate`, a `networkDataTemplate`, or both. A node provider renders the matching field when a property references the template; if that field is unset, provisioning fails. ## Create a MachineConfigTemplate[​](#create-a-machineconfigtemplate "Direct link to Create a MachineConfigTemplate") A cloud-init template renders into a `cloud-config` document. The platform appends the vCluster join command and the resolved SSH keys to the rendered output before applying it to the Machine. ``` apiVersion: management.loft.sh/v1 kind: MachineConfigTemplate metadata: name: ubuntu-cloud-init spec: displayName: "Ubuntu cloud-init" cloudInitTemplate: | #cloud-config hostname: {{ .Values.NodeClaim.Name }} users: - name: ubuntu sudo: ALL=(ALL) NOPASSWD:ALL ssh_authorized_keys: {{- range .Values.SSHKeys }} - {{ . }} {{- end }} runcmd: - echo "Provisioning {{ .Values.NodeClaim.Name }} in project {{ .Values.Project }}" ``` A network-data template renders into a cloud-init `network-config` document. The provider supplies the allocated IP and gateway for the Machine as template variables. ``` apiVersion: management.loft.sh/v1 kind: MachineConfigTemplate metadata: name: static-network spec: displayName: "Static network" networkDataTemplate: | version: 2 ethernets: enp1s0: # AllocatedIP, PrefixLength, and Gateway are supplied during IP allocation. addresses: - {{ .Values.AllocatedIP }}/{{ .Values.PrefixLength }} gateway4: {{ .Values.Gateway }} nameservers: addresses: [8.8.8.8, 8.8.4.4] ``` ### Fields[​](#fields "Direct link to Fields") | Field | Description | | -------------------------- | ---------------------------------------------------------------- | | `spec.displayName` | Human-readable name shown in the UI. | | `spec.cloudInitTemplate` | Go template that renders a `cloud-config` document. | | `spec.networkDataTemplate` | Go template that renders a cloud-init `network-config` document. | | `spec.owner` | Optional user or team that owns this resource. | | `spec.access` | Optional access control list. | ## Template syntax[​](#template-syntax "Direct link to Template syntax") Templates use Go [`text/template`](https://pkg.go.dev/text/template) syntax with the [sprig](https://masterminds.github.io/sprig/) function library. The root object is `.Values`. The available fields depend on the template kind and the node provider that renders it. ### Cloud-init templates[​](#cloud-init-templates "Direct link to Cloud-init templates") | Variable | Description | | -------------------- | ------------------------------------------------------------------------------------- | | `.Values.NodeClaim` | The NodeClaim (Machine) being provisioned. | | `.Values.Project` | The project the Machine belongs to. | | `.Values.Properties` | Merged properties from the NodeProvider, NodeType, NodeEnvironment, and NodeClaim. | | `.Values.SSHKeys` | List of SSH public keys (strings) resolved from the `vcluster.com/ssh-keys` property. | The Metal3 provider additionally exposes `.Values.BareMetalHost` (the selected `BareMetalHost`), `.Values.AllocatedIP`, `.Values.PrefixLength`, and `.Values.Gateway`. ### Network-data templates[​](#network-data-templates "Direct link to Network-data templates") | Variable | Description | | ---------------------- | ----------------------------------------------------------------------------------------- | | `.Values.NodeClaim` | The NodeClaim (Machine) being provisioned. | | `.Values.Project` | The project the Machine belongs to. | | `.Values.Properties` | Merged properties from the NodeProvider, NodeType, NodeEnvironment, and NodeClaim. | | `.Values.AllocatedIP` | IP allocated for the Machine from the configured network. | | `.Values.PrefixLength` | Network prefix length for the allocated IP. Exposed by the KubeVirt and Metal3 providers. | | `.Values.Gateway` | Default gateway for the allocated network. | The Metal3 provider additionally exposes `.Values.BareMetalHost` (the selected `BareMetalHost`). ## Reference a template[​](#reference-a-template "Direct link to Reference a template") Reference a MachineConfigTemplate by name from the properties of a node type (applies to every Machine of that type) or directly on a Machine: | Property | Renders | | ------------------------------------------- | ------------------------------------------------ | | `vcluster.com/user-data-template-config` | The referenced template's `cloudInitTemplate`. | | `vcluster.com/network-data-template-config` | The referenced template's `networkDataTemplate`. | ``` properties: vcluster.com/user-data-template-config: ubuntu-cloud-init vcluster.com/network-data-template-config: static-network ``` The referenced template must define the matching field. A template referenced by `vcluster.com/user-data-template-config` must set `cloudInitTemplate`, and one referenced by `vcluster.com/network-data-template-config` must set `networkDataTemplate`. ## Access control[​](#access-control "Direct link to Access control") MachineConfigTemplate resources are cluster-scoped. Use the `spec.owner` and `spec.access` fields to control who can use or modify them: ``` apiVersion: management.loft.sh/v1 kind: MachineConfigTemplate metadata: name: ubuntu-cloud-init spec: displayName: "Ubuntu cloud-init" cloudInitTemplate: | #cloud-config hostname: {{ .Values.NodeClaim.Name }} owner: user: admin access: - users: - "*" verbs: - get ``` --- # OS Images Modify the following with your specific values to replace on the whole page and generate copyable commands: IMAGE\_URLhttps\://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img IMAGE\_CHECKSUMe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 CHECKSUM\_TYPEsha256 KUBEVIRT\_ROOT\_DISK\_SIZE20Gi An `OSImage` resource contains the information that a node provider needs to install an operating system on a Machine, stored as a reusable reference. Instead of specifying image URLs and checksums directly on every node type or Machine, you define them once in an OSImage and reference it by name. ## Create an OSImage[​](#create-an-osimage "Direct link to Create an OSImage") ``` apiVersion: management.loft.sh/v1 kind: OSImage metadata: name: ubuntu-noble spec: displayName: "Ubuntu Noble (24.04 LTS)" description: "Ubuntu Noble (24.04) server cloud image for bare metal provisioning" properties: metal3.vcluster.com/image-url: "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img" metal3.vcluster.com/image-checksum: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" metal3.vcluster.com/image-checksum-type: "sha256" ``` ### Fields[​](#fields "Direct link to Fields") | Field | Description | | ------------------ | ----------------------------------------------------------- | | `spec.displayName` | Human-readable name shown in the UI. | | `spec.description` | Optional description of the image. | | `spec.properties` | Key-value pairs with provider-specific image configuration. | | `spec.owner` | Optional user or team that owns this resource. | | `spec.access` | Optional access control list. | ## Provider-specific properties[​](#provider-specific-properties "Direct link to Provider-specific properties") OS images are a generic concept. The properties they carry depend on which node provider consumes them. ### Metal3[​](#metal3 "Direct link to Metal3") For the [Metal3 node provider](https://vcluster.com/docs/platform/administer/node-providers/metal3.md), the following properties are required: | Property | Description | | ----------------------------------------- | -------------------------------------------------------------------------------------- | | `metal3.vcluster.com/image-url` | URL of the OS image. | | `metal3.vcluster.com/image-checksum` | Checksum for image verification. | | `metal3.vcluster.com/image-checksum-type` | Checksum algorithm: `md5`, `sha256`, or `sha512`. Optional — auto-detected if omitted. | ### KubeVirt[​](#kubevirt "Direct link to KubeVirt") For the [KubeVirt node provider](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md), an OS image defines the source and size of a VM's root disk. The provider adds a Containerized Data Importer (CDI) `DataVolume` template to the generated VM. | Property | Description | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `machines.vcluster.com/os-image-type` | Classifies the image for provider filtering in the platform UI. Set this property to `kubevirt`. | | `kubevirt.vcluster.com/image-url` | HTTP, HTTPS, or `docker://` image source. Configure either this property or `kubevirt.vcluster.com/image-datasource`. | | `kubevirt.vcluster.com/image-checksum` | Optional checksum for an HTTP or HTTPS source. CDI receives the value unchanged. Include the algorithm prefix, for example `sha256:`. The provider ignores this property for `docker://` and CDI `DataSource` sources. | | `kubevirt.vcluster.com/image-datasource` | CDI `DataSource` in `` or `/` format. A name without a namespace resolves in the VM namespace. | | `kubevirt.vcluster.com/root-disk-size` | Storage request for the root disk, expressed as a Kubernetes quantity such as `20Gi`. Required for every KubeVirt image source. | Configure one source for each image. If both `image-url` and `image-datasource` are present, the provider uses the `DataSource`. It discards the URL and checksum. #### Container registry image source[​](#container-registry-image-source "Direct link to Container registry image source") Use a `docker://` URL to import an image from a container registry. A checksum doesn't apply to this source type. Modify the following with your specific values to generate a copyable command: OS\_IMAGE\_NAMEubuntu-container-disk IMAGE\_URLdocker://quay.io/containerdisks/ubuntu:22.04 kubevirt-registry-os-image.yaml ``` apiVersion: management.loft.sh/v1 kind: OSImage metadata: name: ubuntu-container-disk spec: displayName: "Ubuntu container disk" properties: # Show this image for KubeVirt providers in the platform UI. machines.vcluster.com/os-image-type: kubevirt # Import the root disk from this container registry image. kubevirt.vcluster.com/image-url: "docker://quay.io/containerdisks/ubuntu:22.04" # Request this amount of storage for the root disk. kubevirt.vcluster.com/root-disk-size: "20Gi" ``` The provider creates a registry-backed `DataVolume` template and requests the configured root disk size. #### Import from a web address[​](#import-from-a-web-address "Direct link to Import from a web address") Use an HTTP or HTTPS source to let CDI import a disk image. The checksum is optional and applies only to this source type. Modify the following with your specific values to generate a copyable command: OS\_IMAGE\_NAMEubuntu-noble-http IMAGE\_URLhttps\://example.com/images/ubuntu-noble.qcow2 IMAGE\_CHECKSUMsha256:abc123 kubevirt-http-os-image.yaml ``` apiVersion: management.loft.sh/v1 kind: OSImage metadata: name: ubuntu-noble-http spec: displayName: "Ubuntu Noble from HTTP" properties: # Show this image for KubeVirt providers in the platform UI. machines.vcluster.com/os-image-type: kubevirt # Import the root disk from this web address. kubevirt.vcluster.com/image-url: "https://example.com/images/ubuntu-noble.qcow2" # Validate the downloaded image with its published checksum. kubevirt.vcluster.com/image-checksum: "sha256:abc123" # Request this amount of storage for the root disk. kubevirt.vcluster.com/root-disk-size: "20Gi" ``` The provider creates an HTTP-backed `DataVolume` template and passes the checksum to CDI. Replace the example checksum with the checksum published for your image. If you omit the checksum property, CDI imports the image without checksum validation. #### Use a CDI data source[​](#use-a-cdi-data-source "Direct link to Use a CDI data source") Use a CDI `DataSource` when the source image already exists in the KubeVirt environment. Specify `/` to reference a `DataSource` outside the VM namespace. Modify the following with your specific values to generate a copyable command: OS\_IMAGE\_NAMEubuntu-noble-datasource DATA\_SOURCEos-images/ubuntu-noble kubevirt-datasource-os-image.yaml ``` apiVersion: management.loft.sh/v1 kind: OSImage metadata: name: ubuntu-noble-datasource spec: displayName: "Ubuntu Noble DataSource" properties: # Show this image for KubeVirt providers in the platform UI. machines.vcluster.com/os-image-type: kubevirt # Reference an existing CDI DataSource by namespace and name. kubevirt.vcluster.com/image-datasource: "os-images/ubuntu-noble" # Request this amount of storage for the root disk. kubevirt.vcluster.com/root-disk-size: "20Gi" ``` The provider creates a `DataVolume` template whose `sourceRef` points to the specified CDI `DataSource`. For advanced disk and `DataVolume` customization, see the [KubeVirt image configuration properties](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md#image-configuration). ## Use an OSImage[​](#use-an-osimage "Direct link to Use an OSImage") Reference an OSImage by name in node type properties or Machine properties: ``` properties: vcluster.com/os-image: ubuntu-noble ``` The platform resolves the OSImage and applies its properties during provisioning. On node providers that don't [support the property](#provider-specific-properties), it has no effect. ## Resolve image problems[​](#resolve-image-problems "Direct link to Resolve image problems") Inspect the referenced image when a KubeVirt Machine doesn't provision: Modify the following with your specific values to generate a copyable command: OS\_IMAGE\_NAMEubuntu-noble-http ``` kubectl get osimage ubuntu-noble-http -o yaml ``` | Problem | Resolution | | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The provider reports `failed to get OSImage` | The value of `vcluster.com/os-image` doesn't match an existing cluster-scoped `OSImage`. List the images with `kubectl get osimages`, then correct the reference. | | The image is missing from KubeVirt image selection in the platform UI | Check `machines.vcluster.com/os-image-type`. Set it to `kubevirt`. A missing or different value classifies the image as another provider type or as custom. | | The image defines both `image-url` and `image-datasource` | Remove the source that you don't intend to use. The provider gives `image-datasource` precedence and ignores `image-url` and `image-checksum`. | | Provisioning reports that `kubevirt.vcluster.com/root-disk-size` is required | Add the property to the OS image or another effective property source. Use a valid Kubernetes storage quantity such as `20Gi`. | | Provisioning reports an unsupported image URL scheme | Use a URL beginning with `http://`, `https://`, or `docker://`. | ## Access control[​](#access-control "Direct link to Access control") OSImage resources are cluster-scoped. Use the `spec.owner` and `spec.access` fields to control who can use or modify them: ``` apiVersion: management.loft.sh/v1 kind: OSImage metadata: name: ubuntu-noble spec: displayName: "Ubuntu Noble (24.04 LTS)" properties: metal3.vcluster.com/image-url: "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img" metal3.vcluster.com/image-checksum: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" metal3.vcluster.com/image-checksum-type: "sha256" owner: user: admin access: - users: - "*" verbs: - get ``` --- # Overview | | | Enterprise | | | | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Machine Management](https://vcluster.com/docs/platform/use-platform/machines/overview.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). A Machine is a provisioned server — physical or virtual — managed by vCluster Platform through a node provider. Machines abstract the provisioning details so that platform administrators can offer self-service access to compute resources. Users request Machines either indirectly through tenant cluster auto-nodes, or directly when they need standalone servers outside the tenant cluster lifecycle. Each Machine is backed by a [node provider](https://vcluster.com/docs/platform/administer/node-providers/overview.md) that handles the actual provisioning — creating VMs, provisioning bare metal servers, or running Terraform scripts. For KubeVirt-backed infrastructure, administrators can also manage KubeVirt VMs directly from the [Virtual Machines](https://vcluster.com/docs/platform/administer/virtual-machines/overview.md) page. ## How machines are created[​](#how-machines-are-created "Direct link to How machines are created") Machines are created in two ways: * **Auto nodes**: When a tenant cluster is configured with [private nodes and auto nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/auto-nodes.md), the platform automatically creates scalable pools of Machines. [Karpenter](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/auto-nodes.md) decides when nodes are needed, and the platform provisions them through the configured node provider. The Machine joins the tenant cluster as a worker node. * **Manual**: Machines can also be created independently, without being connected to a tenant cluster. This is useful for provisioning servers that are managed outside of the tenant cluster lifecycle. In both cases, the node provider determines how the Machine is provisioned — what OS image is installed, what user data is applied, and how the server is configured. ## Configuration[​](#configuration "Direct link to Configuration") Machines inherit configuration from the node type they are provisioned from. This includes resource specifications, provider-specific properties, and provisioning settings like OS images and SSH keys. The following resources can be referenced by name in node type or Machine properties to configure provisioning: * [OS images](https://vcluster.com/docs/platform/use-platform/machines/os-images.md): Reusable image definitions containing the URL, checksum, and format for the operating system installed on the Machine. * [SSH keys](https://vcluster.com/docs/platform/use-platform/machines/ssh-keys.md): Public keys included in the Machine's user data during provisioning. --- # SSH Keys SSH keys are stored as `SSHKey` resources and referenced by name in node type or Machine properties. During provisioning, the public keys are included in the user data (e.g. cloud-init) so they are configured on the Machine. SSH keys may be used with any provider that supports user data during provisioning, including [Metal3](https://vcluster.com/docs/platform/administer/node-providers/metal3.md), [KubeVirt](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md), and [Terraform](https://vcluster.com/docs/platform/administer/node-providers/terraform.md). ## Create an SSHKey[​](#create-an-sshkey "Direct link to Create an SSHKey") SSH keys can be managed through the platform UI under your **Profile > SSH Keys**, or by creating the resource directly: ``` apiVersion: management.loft.sh/v1 kind: SSHKey metadata: name: my-ssh-key spec: displayName: "My SSH Key" description: "Development access key" publicKey: "ssh-ed25519 AAAA... user@host" ``` ### Fields[​](#fields "Direct link to Fields") | Field | Description | | ------------------ | ---------------------------------------------- | | `spec.displayName` | Human-readable name shown in the UI. | | `spec.description` | Optional description. | | `spec.publicKey` | The SSH public key content. | | `spec.owner` | Optional user or team that owns this resource. | | `spec.access` | Optional access control list. | ## Use SSH keys[​](#use-ssh-keys "Direct link to Use SSH keys") Reference SSH keys by name in node type properties or Machine properties. Multiple keys can be specified as a comma-separated list: ``` properties: vcluster.com/ssh-keys: my-ssh-key,team-shared-key ``` All referenced keys are added to the provisioned server's `authorized_keys`. ## Access control[​](#access-control "Direct link to Access control") SSHKey resources are cluster-scoped. Use the `spec.owner` and `spec.access` fields to control who can use or modify them: ``` apiVersion: management.loft.sh/v1 kind: SSHKey metadata: name: team-key spec: displayName: "Team Shared Key" publicKey: "ssh-ed25519 AAAA... team@org" owner: team: platform-team access: - teams: - platform-team verbs: - get - update - delete ``` --- # Create Namespace Manually Each namespace created in vCluster Platform belongs to a [project](https://vcluster.com/docs/platform/understand/what-are-projects.md). Projects control whether users can create namespaces with or without [templates](https://vcluster.com/docs/platform/understand/what-are-templates.md). ## Who can create namespaces without a template[​](#who-can-create-namespaces-without-a-template "Direct link to Who can create namespaces without a template") By default, projects require templates for namespace creation. This means: * **Project users** can only create namespaces from [allowed templates](https://vcluster.com/docs/platform/administer/projects/allowed/templates.md) configured by the project admin. * **Project admins and platform admins** can always create namespaces without a template, regardless of project settings. If project admins disable the `requireTemplate` setting, project users can also create namespaces without templates. Keep this setting enabled in production environments to maintain security controls. ## Create without template[​](#create-without-template "Direct link to Create without template") Security consideration Creating namespaces without a template bypasses the security controls that templates provide. Users with this capability can configure any namespace settings, including [namespace objects](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/isolation.md#configure-namespace-objects) and [namespace apps](https://vcluster.com/docs/platform/use-platform/apps/use-in-templates.md), which deploy resources directly to the control plane cluster. For production environments: * Keep templates required for all projects (the default `requireTemplate` setting). * Grant project admin roles only to users who need to create namespaces without templates. * Control who can create projects, since project creators become project admins. See [project membership](https://vcluster.com/docs/platform/administer/users-permissions/permissions/project.md) for details on project roles. 1. Go to the Projects view using the menu on the left 2. Click on Namespaces and the Create Namespace button 3. In the popup make sure the namespace template field is empty and click the Select button 4. Optional: Select a cluster where to create the namespace in. If no cluster is specified, the platform will automatically select one for you. 5. Specify a name of the namespace at the top and optionally configure the namespace you want to create. 6. Click on the Create Namespace button 7. Retrieve a kube-context for this namespace using vCluster CLI: ``` vcluster platform connect namespace [namespace-name] --project my-project ``` See [Connect](https://vcluster.com/docs/platform/use-platform/namespaces/retrieve-kube-config.md) for the kubeconfig without the CLI. --- # Create Namespace from Templates | | | Enterprise | | | | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Namespace Management](https://vcluster.com/docs/platform/use-platform/namespaces/create/create-with-template.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Once you have created a namespace template you can then refer to that template when creating a new namespace. All template configurations will be automatically applied to the newly created namespace. Namespaces referring to templates can be created in the UI or using the vCluster CLI. * UI * CLI 1. Go to the Projects view using the menu on the left 2. Click on Namespaces and the Create Namespace button 3. In the popup select the namespace template and click the Select button 4. Optional: Select a cluster where to create the namespace in. If no cluster is specified, the platform will automatically select one for you. 5. Specify a name of the namespace at the top and optionally configure the template parameters and namespace access. 6. Click on the Create Namespace button 7. Retrieve a kube-context for this namespace using the vCluster CLI: ``` vcluster platform connect namespace [namespace-name] --project my-project ``` info The Kubernetes namespace in the underlying cluster this namespace points to usually has a different name than the namespace inside the project to avoid conflicts, you can adjust the space name pattern within a project. To create a namespace using vCluster CLI, run: ``` vCluster platform create namespace [namespace-name] --project my-project ``` Template If you omit specifying a template, vCluster Platform will automatically select one for you (if there is a default template in the project specified) or prompt you to select a template. Kube-Context Running `loft create namespace` will automatically add a kube-context to your kube-config file, so you can immediately run `kubectl` commands right after creating a namespace. ## Template version strings[​](#template-version-strings "Direct link to Template version strings") Regardless of how you create a namespace from a template, either by the UI or the CLI, vCluster Platform allows you to provide a template version string. If you do not provide this string, the latest version will be automatically selected for you. If you *do* provide the string though, you can do something very neat -- you can set any of the MAJOR, MINOR, PATCH version components to an `X` wildcard character. This allows the namespace to be automatically updated to a more recent template version that matches your provided template string. For example, given a template with a version of 1.0.0, and a namespace created from this template with a version string "1.X.X". Adding a new template version "1.1.0", will cause this namespace to be automatically updated. Whereas adding a new version "2.0.0", will *not* cause the namespace to be upgraded. This is a very handy way to keep namespaces up to date with the latest templates without having to manually upgrade them. Note that you can even set the version to "X.X.X" to *always* have your namespace updated to the latest template version. --- # Custom links You can add custom links to namespaces that point to external resources such as GitHub pull requests, Jira issues, or documentation. This allows team members to access all relevant resources directly from the namespace view. Tenant cluster custom links For more information on configuring custom links for tenant clusters, see the [Custom Links](https://vcluster.com/docs/vcluster/key-features/custom-links.md) documentation. * UI * CLI 1. Navigate to the Namespaces view. 2. Edit an existing Namespace or create a new one. 3. Click the edit icon next to the links section to open the links editor. Click Add to add a new link. 4. Enter a URL and optionally a label for your link. 5. Click Save. 6. Click Create Namespace, or Save Changes if you're editing an existing namespace, to apply all changes. ### Single named link[​](#single-named-link "Direct link to Single named link") Add a custom link with a descriptive label: ``` vcluster platform create namespace namespace-1 --link 'my-issue=https://github.com/kubernetes/kubernetes/pull/1234' ``` ### Single unnamed link[​](#single-unnamed-link "Direct link to Single unnamed link") Add a direct URL link: ``` vcluster platform create namespace namespace-1 --link 'https://loft.sh' ``` ### Multiple links[​](#multiple-links "Direct link to Multiple links") Add multiple custom links in a single command: ``` vcluster platform create namespace namespace-1 --link 'my-issue=https://github.com/kubernetes/kubernetes/pull/1234,documentation=https://example.com/docs' ```
The custom links are displayed in the vCluster Platform UI and allow team members to access related resources for the namespace. --- # Isolation Namespaces can be configured to provide moderate levels of isolation. This "isolation" typically includes applying ResourceQuotas, LimitRanges, and NetworkPolicies to a Namespace. These policies can be configured using the Space Template `Objects` option, to provide your desired quotas, limits and network policies. To make your life easier, vCluster Platform is deployed with a default Space Template called `Isolated Space Template`. This space template can give you a head start on configuring your own namespace isolation policies, or you may find the template sufficient for your needs. ## Default isolated space template[​](#default-isolated-space-template "Direct link to Default isolated space template") The `Isolated Space Template` applies three resources: * **ResourceQuota**: Limits namespace to 10 CPU cores (20 max), 20Gi memory (40Gi max), 100Gi storage, 20 pods * **LimitRange**: Sets container defaults (20m CPU, 64Mi memory) and limits (2 CPU, 4Gi memory) * **NetworkPolicy**: Restricts egress to DNS and platform services, blocks private IP ranges ## Configure namespace objects[​](#configure-namespace-objects "Direct link to Configure namespace objects") The `Objects` field in a space template defines Kubernetes resources to create in each namespace. ### Example: Custom isolation configuration[​](#example-custom-isolation-configuration "Direct link to Example: Custom isolation configuration") ``` apiVersion: management.loft.sh/v1 kind: SpaceTemplate metadata: name: dev-team-isolated spec: displayName: Development Team Isolated Space description: Isolated space template for development teams with moderate resource limits access: - users: - '*' verbs: - get template: metadata: {} instanceTemplate: metadata: {} objects: | # ResourceQuota for development environments apiVersion: v1 kind: ResourceQuota metadata: name: dev-resource-quota spec: hard: requests.cpu: "4" # Total CPU requests limited to 4 cores requests.memory: "8Gi" # Total memory requests limited to 8GB limits.cpu: "8" # Total CPU limits cannot exceed 8 cores limits.memory: "16Gi" # Total memory limits cannot exceed 16GB persistentvolumeclaims: "10" # Maximum 10 PVCs services.loadbalancers: "2" # Maximum 2 LoadBalancer services count/deployments.apps: "20" # Maximum 20 deployments count/pods: "50" # Maximum 50 pods --- # LimitRange for container defaults apiVersion: v1 kind: LimitRange metadata: name: dev-limit-range spec: limits: - type: Container default: cpu: "500m" # Default CPU limit if not specified memory: "1Gi" # Default memory limit if not specified defaultRequest: cpu: "100m" # Default CPU request memory: "128Mi" # Default memory request max: cpu: "2" # Single container max 2 cores memory: "4Gi" # Single container max 4GB min: cpu: "10m" # Minimum CPU per container memory: "32Mi" # Minimum memory per container --- # NetworkPolicy with more permissive rules for development apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: dev-network-policy spec: podSelector: {} # Apply to all pods in namespace policyTypes: - Egress - Ingress ingress: - from: - podSelector: {} # Allow all traffic within namespace - namespaceSelector: matchLabels: loft.sh/allow-traffic: 'true' # Allow from platform services egress: - to: - podSelector: {} # Allow to all pods in namespace - ports: # Allow DNS - port: 53 protocol: UDP - port: 53 protocol: TCP - to: # Allow external HTTPS/HTTP for development - namespaceSelector: {} - ipBlock: cidr: 0.0.0.0/0 except: # Block local networks except for specific needs - 169.254.0.0/16 ports: - protocol: TCP port: 443 - protocol: TCP port: 80 ``` The `objects` field accepts a yaml string with Kubernetes resources separated by `---`. CNIs and Network Policies Not all CNIs will support all network policies. Make sure you understand what capabilities your CNI supports when investigating namespace isolation. To enforce storage limits inside a tenant cluster, including per-PVC size caps and storage class restrictions, see [Enforce storage quotas per tenant cluster](https://vcluster.com/docs/platform/how-to/enforce-storage-quotas.md). --- # Prevent Deletion One can prevent namespace deletion by following one of the two methods: * Within the UI * Within the YAML manifest You can prevent namespace deletion, by setting the annotation "loft.sh/non-deletable" to "true" in the YAML manifest of the space instance. ``` apiVersion: management.loft.sh/v1 kind: SpaceInstance metadata: name: my-namespace annotations: loft.sh/non-deletable: "true" spec: templateRef: name: isolated-namespace clusterRef: cluster: loft-cluster ``` 1. Go to Projects. 2. Select the project and then the namespace you want to prevent deletion for. 3. Either click the Create Namespace button or click the Edit option for the namespace you want to modify. 4. Navigate to Advanced section on the left. 5. Select the Prevent Deletion expander and enable Prevent Deletion. 6. Click the Save button to save the changes, or the Create button when creating a new namespace. note If you want to delete this namespace in the future, you need to remove the label either in the YAML manifest or through the toggle in the UI. Once you have done this, you can delete the namespace as usual. --- # Manage namespace sleep and auto-delete | | | Enterprise | | | | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Auto Sleep for Namespaces](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/sleep-mode.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). vCluster Platform can put an idle namespace to sleep by scaling down its workloads. It retains workload-controller definitions and restores their replicas when the namespace wakes. The platform can also delete namespaces after an inactivity period. Standalone Pods Sleep deletes Pods that no workload controller manages. Kubernetes can't restore these Pods during wakeup. ## Put a namespace to sleep[​](#put-a-namespace-to-sleep "Direct link to Put a namespace to sleep") 1. Select the project that contains the namespace. 2. Click **Namespaces** in the sidebar. 3. Open the namespace's actions, then click **Sleep**. 4. Optionally select **Prevent automatic wake up** and enter a duration in hours. Enter `0` to prevent automatic wakeup indefinitely. 5. Click **Ok**. To perform the same operation with the CLI, run: ``` vcluster platform sleep namespace --project ``` Add `--prevent-wakeup ` to block activity-based wakeup for a period. Use `--prevent-wakeup 0` to require a manual wakeup. ## Wake a namespace[​](#wake-a-namespace "Direct link to Wake a namespace") Open the sleeping namespace's actions, click **Wake Up**, then confirm the wakeup. You can also run: ``` vcluster platform wakeup namespace --project ``` Unless wakeup is prevented, a Kubernetes API request routed through the platform can wake the namespace automatically. Supported Ingress, Istio, and Gateway API traffic can also trigger wakeup. ## Configure auto sleep[​](#configure-auto-sleep "Direct link to Configure auto sleep") 1. Select the project that contains the namespace. 2. Click **Namespaces** in the sidebar. 3. Click **Edit** for the namespace. 4. Expand **Sleep Mode**. 5. Set **Sleep After Inactivity**, **Sleep Schedule**, **Wake up Schedule**, and **Schedule timezone** as needed. 6. Click **Save Changes**. For a namespace created from a template, edit the template instead of the instance. ## Configure auto-delete[​](#configure-auto-delete "Direct link to Configure auto-delete") Auto-delete permanently deletes a namespace after an inactivity period. It is separate from sleep and can run whether the namespace is awake or sleeping. Edit the namespace, expand **Sleep Mode**, and set **Delete After Inactivity**. Configure the value in the template for template-created namespaces. ## Activity detection[​](#activity-detection "Direct link to Activity detection") Kubernetes API requests routed through vCluster Platform count as namespace activity. This includes `kubectl` requests when the kubeconfig routes through the platform proxy. The platform can also detect supported Ingress, Istio, and Gateway API traffic. These integrations and the last-activity update interval work the same way for namespaces and tenant clusters. See [activity detection and wakeup](https://vcluster.com/docs/vcluster/next/configure/vcluster-yaml/sleep#activity-detection-and-wakeup) for the shared behavior and agent configuration. ## Exclude workloads[​](#exclude-workloads "Direct link to Exclude workloads") Add `sleepmode.loft.sh/exclude: "true"` to a Deployment, StatefulSet, ReplicaSet, or Pod to leave its existing Pods running while the namespace sleeps. See [Exclude workloads from auto sleep](https://vcluster.com/docs/vcluster/next/configure/vcluster-yaml/sleep#exclude-workloads-from-auto-sleep) for the quota behavior and a full example. ## Filter activity and long-running connections[​](#filter-activity-and-long-running-connections "Direct link to Filter activity and long-running connections") Add request-filter annotations, including `sleepmode.loft.sh/ignore-active-connections`, to the namespace. See [filter platform activity](https://vcluster.com/docs/vcluster/next/configure/vcluster-yaml/sleep#filter-platform-activity) and [handle long-running connections](https://vcluster.com/docs/vcluster/next/configure/vcluster-yaml/sleep#handle-long-running-connections) for the shared configuration. The [platform annotation reference](https://vcluster.com/docs/platform/reference/platform-annotations.md#sleepmode-annotations) lists every supported filter and value format. --- # Get kubeconfig File You can either manually construct a kubeconfig to access your namespace or use the vCluster CLI to automatically create a kubeconfig. ## Retrieve using the CLI[​](#retrieve-using-the-cli "Direct link to Retrieve using the CLI") You can update your current kube-config by running the following command: ``` vcluster platform connect namespace my-namespace --project my-project ``` If you want to print the kubeconfig, you can append `--print` to the command: ``` vcluster platform connect namespace my-namespace --project my-project --print ``` info Depending on if you have regional cluster endpoints enabled, the kubeconfig will look different. See the [Platform CLI hub](https://vcluster.com/docs/platform/reference/platform-cli.md) for UI-to-CLI equivalents across common Platform tasks, including namespace commands. ## kubeconfig without CLI[​](#kubeconfig-without-cli "Direct link to kubeconfig without CLI") You can create the kubeconfig also manually by creating an access key for your user first and then using the following template, with the following placeholders: * `$PROJECT`: the project the namespace is in * `$SPACE`: the space name to connect to * `$NAMESPACE`: the target namespace (check the `spec.clusterRef.namespace` field in the SpaceInstance object) * `$PLATFORM_URL`: the platform host you connect to * `$ACCESS_KEY`: the access key to use Then replace these placeholders in the following template and save it as `my-kubeconfig.yaml`: ``` apiVersion: v1 kind: Config clusters: - cluster: server: https://$PLATFORM_URL/kubernetes/project/$PROJECT/space/$SPACE # Optional: if the server uses an insecure certificate # insecure-skip-tls-verify: true name: loft contexts: - context: cluster: loft namespace: $NAMESPACE user: loft name: loft current-context: loft users: - name: loft user: token: $ACCESS_KEY ``` Access your namespace with: ``` KUBECONFIG=my-kubeconfig.yaml kubectl get pods ``` --- # Sync Secrets | | | Enterprise | | | | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Secrets Sync](https://vcluster.com/docs/platform/use-platform/secrets/secret-sync.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Creating project secrets with data or synchronized from a global secret is great and all, but we still need a Kubernetes native secret to make use of the secret data from a Pod. vCluster Platform makes this possible by reading a special label from Kubernetes native secrets and updating it with the project secret data. ## Create an Opaque Secret[​](#create-an-opaque-secret "Direct link to Create an Opaque Secret") Create a Kubernetes secret in the namespace it's needed, and add the `loft.sh/project-secret-name` with the name of the project secret as its value: ``` apiVersion: v1 kind: Secret metadata: name: my-secret namespace: my-space labels: loft.sh/project-secret-name: type: Opaque ``` Replace `` with the name of the project secret. The secret will be synchronized with the project secret momentarily. Secrets may be synchronized in any namespace or tenant cluster that belong to the project. ## Syncing Secrets in Intervals[​](#syncing-secrets-in-intervals "Direct link to Syncing Secrets in Intervals") Secrets inside tenant clusters will be synchronized whenever the project secret is created or modified. However, since it would be inefficient to watch all secrets in all tenant clusters, vCluster Platform synchronizes the tenant cluster secrets on an interval. This means there's a small delay before a tenant cluster secret is synchronized once it's created. ## Create other Secret Types[​](#create-other-secret-types "Direct link to Create other Secret Types") There are other types of secrets that can be created because any type of native Kubernetes secret is allowed. In this case example, the secret `type` will remain unchanged by vCluster Platform, and the data will be populated as-is from the project secret. The project secret data must conform to the format required by the Kubernetes secret type. All mandatory keys must be present with a valid value and cannot be omitted, even when using a placeholder. The type `kubernetes.io/dockerconfigjson` requires the `.dockerconfigjson` key, an empty JSON object `{}` is the minimum valid value. ``` apiVersion: v1 kind: Secret metadata: name: my-registry-secret namespace: my-space labels: loft.sh/project-secret-name: data: .dockerconfigjson: e30K # base64("{}") type: kubernetes.io/dockerconfigjson ``` Replace `` with the name of the project secret. --- # Troubleshooting If you run into any issues while deploying vCluster Platform or troubleshooting vCluster instances, this page contains some information that may help. If you're still stuck, join the [Slack](https://slack.loft.sh/) #vCluster channel where you can ask questions. ## vCluster Platform[​](#vcluster-platform "Direct link to vCluster Platform") ### Flag `--reset`[​](#flag---reset "Direct link to flag---reset") If you need a clean vCluster Platform installation, you can use the `--reset` flag to tell vCluster CLI to purge any previous vCluster Platform installs: ``` vcluster platform start --reset # --insecure ...add any other `vcluster platform start` flags if needed ``` ### Debug `ImageErr` + `ImagePullBackOff`[​](#debug-imageerr--imagepullbackoff "Direct link to debug-imageerr--imagepullbackoff") If your vCluster Platform pods are not starting because the vCluster Platform images cannot be pulled, you may need to use a private image registry rather than pulling vCluster Platform images from over the internet. ### Use port-forwarding[​](#use-port-forwarding "Direct link to Use port-forwarding") You can always try to connect to vCluster Platform directly and circumvent any potential networking issues introduced at the load balancer and ingress controller level. To start port-forwarding, either use `vcluster platform start` or run: ``` kubectl port-forward deploy/loft -n vcluster-platform 8080:10443 ``` After that you can access vCluster Platform at `https://localhost:8080`. You can even login with the vCluster CLI to this URL with: ``` vcluster platform login localhost:8080 --insecure ``` If you can access vCluster Platform through port-forwarding only, this is usually an indicator that the problem might be related to a misconfigured LoadBalancer or Ingress Controller. ### Quota and admission denials[​](#quota-and-admission-denials "Direct link to Quota and admission denials") If creating or waking a tenant cluster, namespace, Machine, pod, service, or PVC fails with a quota-related `Forbidden` error, use the [quota and admission troubleshooting runbook](https://vcluster.com/docs/platform/troubleshoot/quota-admission.md) to identify the project, owner, quota expression, and generated cluster quota that denied the request. ### Connected cluster connection failures[​](#connected-cluster-connection-failures "Direct link to Connected cluster connection failures") If a connected cluster never finishes connecting, goes offline or flaps after connecting, or gets stuck mid-upgrade, use the [connected cluster connectivity runbook](https://vcluster.com/docs/platform/troubleshoot/connected-cluster-connectivity.md) to match the cluster's status reason to a cause and resolution. ## vCluster[​](#vcluster "Direct link to vCluster") ### Debug shell[​](#debug-shell "Direct link to Debug shell") The Debug Shell opens a terminal in an ephemeral container attached to a vCluster control plane pod. When a debug shell already exists for that pod, the existing ephemeral container is reused. The debug shell container runs for up to 60 minutes and then exits. If embedded etcd is detected, environment variables and endpoints for `etcdctl` are automatically configured. info By default, Global Admins, Management Admins, and Project Admins can access Debug Shell. Administrators can grant additional access by allowing the `virtualclusterinstances/debug-shell` and `virtualclusterinstances/debug-shell-pods` subresources. This role should be assigned to users who already have access to the tenant cluster. For example, the following project role grants access: ``` apiVersion: management.loft.sh/v1 kind: ClusterRoleTemplate metadata: {} spec: access: - users: - '*' verbs: - get clusterRoleTemplate: metadata: {} rules: - apiGroups: - management.loft.sh/v1 resources: - virtualclusterinstances/debug-shell - virtualclusterinstances/debug-shell-pods verbs: - get - create description: |- Allows the user or team to access the vCluster Debug Shell without granting create/delete permissions on the vClusterInstance itself. displayName: vCluster Debug Shell management: true status: {} ``` --- # Add existing tenant clusters Any running tenant cluster can be added to the Platform. When adding a tenant cluster to the Platform, you can choose between two integration levels based on your needs. Related documentation For conceptual understanding of externally vs. Platform-deployed tenant clusters, see [Externally deployed tenant clusters](https://vcluster.com/docs/platform/understand/externally-vs-platform-deployed.md). See the [Platform CLI hub](https://vcluster.com/docs/platform/reference/platform-cli.md) for UI-to-CLI equivalents across common Platform tasks, including tenant cluster commands. ## Integration levels[​](#integration-levels "Direct link to Integration levels") When adding an existing vCluster to the Platform, you can choose between two integration levels based on your needs: ### Management modes[​](#management-modes "Direct link to Management modes") * **Visibility and access only** - Platform provides monitoring and access features while you continue managing the vCluster with your existing tools * **Full Platform management** - Platform takes over complete lifecycle management, enabling all Platform features like templates, auto sleep, and auto-delete Technical details The Platform uses a VirtualClusterInstance resource to track tenant clusters. The management level is controlled automatically based on how you add the vCluster. For more details, see the [VirtualClusterInstance API reference](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance/.md). ## Add for visibility and access[​](#add-for-visibility-and-access "Direct link to Add for visibility and access") Use this approach when you want to keep managing the vCluster with your existing tools (Helm, Argo CD, or similar) while gaining visibility and access through the Platform, including SSO integration. 1. **Connect to the Platform**: to use any `vcluster platform` command, you need to be connected to the platform. See [authentication documentation](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) for more details. When you started the platform, you would have automatically connected to the platform when you ran `vcluster platform start`. If you aren't sure if you are connected and already have a running platform, then you can login. Before running this command, be sure that your kubecontext is set to the cluster running the platform. Login to the platform. ``` vcluster platform login ``` 2. **Add Tenant Cluster to the Platform**: before running this command, be sure that your kubecontext is set to the control plane cluster running the vCluster that you want to add. Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster PROJECTdefault IMPORT\_NAMEMy vCluster Add vCluster to the platform in a project. ``` VCLUSTER_NAME=my-vcluster PROJECT=default IMPORT_NAME="My vCluster" vcluster platform add vcluster $VCLUSTER_NAME \ --project $PROJECT \ --import-name "$IMPORT_NAME" ``` This creates a VirtualClusterInstance with `external: true`, meaning the Platform will not manage the lifecycle. 3. **Add Control Plane Cluster to the Platform (Optional)**: for additional visibility of the control plane cluster in the Platform UI, you can also add the control plane cluster itself. Before running this command, be sure that your kubecontext is set to the control plane cluster. Modify the following with your specific values to generate a copyable command: CLUSTERmy-host-cluster DISPLAY\_NAMEMy Control Plane Cluster Connect control plane cluster to platform. ``` CLUSTER=my-host-cluster DISPLAY_NAME="My Control Plane Cluster" vcluster platform add cluster $CLUSTER --display-name "$DISPLAY_NAME" ``` This step is optional and provides additional management capabilities for the control plane cluster infrastructure. With visibility and access mode: * ✅ View vCluster status in Platform UI * ✅ Access through Platform with SSO integration * ✅ Basic monitoring and metrics * ✅ Continue managing with original deployment tool * ❌ No Platform lifecycle management * ❌ No templates, platform auto sleep, or auto-delete ## Enable full platform management[​](#enable-full-platform-management "Direct link to Enable full platform management") Use this approach when you want the Platform to take over complete management of your externally deployed vCluster, unlocking all Platform features. ### Why enable platform management?[​](#why-enable-platform-management "Direct link to Why enable platform management?") Organizations often have existing deployment pipelines using tools like Helm, Argo CD, or Terraform. This option allows you to: * Maintain existing CI/CD workflows for initial deployment * Transition to Platform management for ongoing operations * Enable Platform features like auto sleep, SSO, and centralized management * Integrate existing vCluster deployments into Platform-managed projects with quotas and policies ### Post-deployment platform registration[​](#post-deployment-platform-registration "Direct link to Post-deployment platform registration") After deploying your vCluster using your existing tools, add it to the Platform and enable full management. First, login with the platform [access key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md): ``` vcluster platform login [domain] --access-key=[ACCESS_KEY] ``` Then, add the tenant cluster to the platform by running: Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEvc-name NAMESPACEvc-namespace PROJECTdefault ``` # Assuming you have already deployed a vCluster using Helm, Argo CD, or another tool # Enable full Platform management vcluster platform add vcluster vc-name \ --namespace vc-namespace \ --project default --external=false ``` You can also use Helm values by updating the `vcluster.yaml` to include the access key of the platform as follows and then upgrade or restart the tenant cluster. ``` platform: apiKey: secretName: vcluster-platform-api-key # Defaults to vcluster-platform-api-key if undefined namespace: vcluster-namespace # Namespace to search for the secret. If undefined, it searches the vCluster namespace. If different, vCluster needs access to the target namespace. ``` The platform access key can only be provided as a Kubernetes Secret. You can find more information of how to create such Secret at the [`apiKey` configuration section of vCluster.](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/platform.md). ### Helm wrapper for VirtualClusterInstance[​](#helm-wrapper-for-virtualclusterinstance "Direct link to Helm wrapper for VirtualClusterInstance") Create a Helm chart that deploys the VirtualClusterInstance directly, allowing the Platform to manage the vCluster from creation. This approach is ideal when you want Helm to deploy the custom resource but let the Platform handle the actual vCluster. templates/virtualclusterinstance.yaml ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterInstance metadata: name: {{ .Values.name }} namespace: p-{{ .Values.project | default "default" }} # Project namespace spec: clusterRef: cluster: {{ .Values.cluster | default "loft-cluster" }} external: false # Platform manages lifecycle owner: user: {{ .Values.owner }} template: helmRelease: chart: version: {{ .Values.vclusterVersion }} values: | {{ .Values.vclusterValues | nindent 8 }} ``` Namespace requirements The Platform must create and manage the namespace when `external: false`. Pre-existing namespaces will cause the error: "namespace exists and is not managed" ### GitOps with VirtualClusterInstance[​](#gitops-with-virtualclusterinstance "Direct link to GitOps with VirtualClusterInstance") For [GitOps deployments](https://vcluster.com/docs/platform/install/gitops.md), create the [VirtualClusterInstance](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance/.md) directly in your Git repository. Tools like [Argo CD](https://vcluster.com/docs/platform/integrations/argocd.md) or Flux apply the manifest, and the Platform manages the tenant cluster: gitops/vcluster-instance.yaml ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterInstance metadata: name: gitops-vcluster namespace: p-default # Platform project namespace spec: clusterRef: cluster: loft-cluster external: false # Platform manages from creation owner: user: admin template: helmRelease: chart: version: "0.27.0" values: | controlPlane: backingStore: etcd: deploy: enabled: true ``` GitOps best practices * Store VirtualClusterInstance manifests in your Git repository * Use Kustomize or Helm for environment-specific configurations * Let Argo CD or Flux handle the VirtualClusterInstance lifecycle while Platform manages the vCluster * See [Platform GitOps installation guide](https://vcluster.com/docs/platform/install/gitops.md) for complete setup ## Implementation example[​](#implementation-example "Direct link to Implementation example") ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Your current kube-context must have administrative privileges, which you can verify with `kubectl auth can-i create clusterrole -A` info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm` installed: Helm v3.10.0+ is required for deploying the platform. Refer to the [Helm Installation Guide](https://helm.sh/docs/intro/install/) if you need to install it. * `kubectl` installed: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. Additionally, you'll need: * vCluster CLI installed and configured: See [vCluster CLI Installation](https://www.vcluster.com/docs/getting-started/setup) ### Implementation steps[​](#implementation-steps "Direct link to Implementation steps") 1. **Deploy vCluster with Helm** Deploy your vCluster using Helm or your preferred deployment tool. For detailed Platform installation instructions, see the [Platform Helm installation guide](https://vcluster.com/docs/platform/install/helm.md). info For complete vCluster configuration options, see the [vCluster configuration reference](https://vcluster.com/docs/vcluster/configure/vcluster-yaml). 2. **Add the vCluster to Platform** Modify the following with your specific values to generate a copyable command: NAMEvc-name NAMESPACEvc-namespace PROJECTdefault IMPORT\_NAMEimported-vc ``` vcluster platform add vcluster vc-name \ --namespace vc-namespace \ --project default \ --import-name "imported-vc" --external=false ``` This creates a Platform managed VirtualClusterInstance. 3. **Verify Platform features are active** Modify the following with your specific values to generate a copyable command: PROJECTdefault IMPORT\_NAMEimported-vc ``` # Check VirtualClusterInstance status kubectl get virtualclusterinstances.storage.loft.sh \ -n p-default imported-vc -o yaml # Verify in Platform UI vcluster platform list vcluster ``` ## Verification[​](#verification "Direct link to Verification") Verify that the tenant cluster is correctly managed by the Platform. You can check this using the CLI: Modify the following with your specific values to generate a copyable command: NAMEvc-name PROJECTdefault ``` # List all VirtualClusterInstances kubectl get virtualclusterinstances.storage.loft.sh -A # Verify external field is false (empty means false) kubectl get virtualclusterinstances.storage.loft.sh vc-name \ -n p-default \ -o jsonpath='{.spec.external}' # Check Platform-managed tenant clusters vcluster list --driver=platform # Access Platform UI vcluster ui ``` ## Platform features available with full management[​](#platform-features-available-with-full-management "Direct link to Platform features available with full management") Once `external: false` is set, the following Platform features become available: * **[Full lifecycle management](https://vcluster.com/docs/platform/use-platform/virtual-clusters/manage-access.md)**: Platform handles upgrades and configuration changes * **[Templates](https://vcluster.com/docs/platform/administer/templates/create-templates.md)**: Use and enforce vCluster templates for consistency * **[Auto-delete](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md#configure-auto-delete)**: Automatic cleanup after inactivity periods * **[Auto Sleep](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md)**: Automatic sleep/wake, to include the vCluster control plane, based on activity to save resources * **[Project assignment](https://vcluster.com/docs/platform/administer/projects/create.md)**: Organize tenant clusters into projects with quotas * **[SSO integration](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md)**: User access through Platform SSO providers * **[Audit logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md)**: Centralized audit trails for compliance * **[Resource quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md)**: Enforce resource limits using project settings * **[UI management](https://vcluster.com/docs/platform/use-platform/virtual-clusters/add-virtual-clusters.md)**: Full control through Platform UI * **[Backup/restore](https://vcluster.com/docs/platform/maintenance/backup-restore/backup-restore-platform.md)**: Platform-managed backup policies ## Limitations and considerations[​](#limitations-and-considerations "Direct link to Limitations and considerations") ### Architectural considerations[​](#architectural-considerations "Direct link to Architectural considerations") **Single source of truth principle:** when `external: false`, the Platform becomes the sole manager of the tenant cluster. This means: * Avoid making changes through the original deployment tool (Helm, Argo CD) * Configuration updates should be done through Platform UI or by modifying the VirtualClusterInstance * Competing reconciliation loops can cause conflicts and unpredictable behavior ### Technical requirements[​](#technical-requirements "Direct link to Technical requirements") * **Namespace management**: The Platform must create and manage the namespace when `external: false`. Pre-existing namespaces will cause deployment failures with error: "namespace exists and is not managed" * **Reconciliation conflicts**: If external tools continue to manage the tenant cluster after setting `external: false`, both systems may fight to enforce their desired state * **Version compatibility**: Requires vCluster v0.20.0 or later for full Platform integration ### Alternative approaches[​](#alternative-approaches "Direct link to Alternative approaches") For specific scenarios where full Platform management isn't suitable: 1. **Hybrid management**: Keep `external: true` to maintain external lifecycle management while still gaining Platform features like SSO and monitoring 2. **GitOps compatibility**: Use VirtualClusterInstance manifests in your GitOps repository, letting Argo CD deploy them while Platform manages the resulting tenant clusters 3. **Gradual migration**: Start with external management, evaluate Platform features, then transition to full management when ready ## Next steps[​](#next-steps "Direct link to Next steps") * Learn about [managing tenant cluster access](https://vcluster.com/docs/platform/use-platform/virtual-clusters/manage-access.md) * Configure [auto sleep](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md) for cost optimization * Set up [templates](https://vcluster.com/docs/platform/administer/templates/create-templates.md) for consistent deployments * Explore [GitOps integration](https://vcluster.com/docs/platform/install/gitops.md) for declarative management --- # Manually configure a tenant cluster Each tenant cluster belongs to a [project](https://vcluster.com/docs/platform/understand/what-are-projects.md). Projects control whether users can create tenant clusters with or without [templates](https://vcluster.com/docs/platform/understand/what-are-templates.md). ## Who can create tenant clusters without a template[​](#who-can-create-tenant-clusters-without-a-template "Direct link to Who can create tenant clusters without a template") By default, projects require templates for tenant cluster creation. This means: * **Project users** can only create tenant clusters from [allowed templates](https://vcluster.com/docs/platform/administer/projects/allowed/templates.md) configured by the project admin. * **Project admins and platform admins** can always create tenant clusters without a template, regardless of project settings. If project admins disable the `requireTemplate` setting, project users can also create tenant clusters without templates. Keep this setting enabled in production environments to maintain security controls. ## Create without template[​](#create-without-template "Direct link to Create without template") Security consideration Creating tenant clusters without a template bypasses the security controls that templates provide. Users with this capability can configure any vCluster settings, including sync configurations that could grant elevated access to control plane cluster resources. For production environments: * Keep templates required for all projects (the default `requireTemplate` setting) * Grant project admin roles only to users who need to create tenant clusters without templates * Control who can create projects, since project creators become project admins. See [project membership](https://vcluster.com/docs/platform/administer/users-permissions/permissions/project.md) for details on project roles * Use [hardened templates](https://vcluster.com/docs/platform/administer/templates/create-templates.md#security-hardening) to control which resources can be synced to and from the control plane cluster 1. From the project drop-down menu (top left corner), select the project you'd like to create the tenant cluster in. 2. Click on Tenant Clusters. 3. Click the Create Tenant Cluster button. 4. Follow the steps in the UI to create the virtual cluster. 5. Retrieve a kube-context for a virtual cluster using the CLI: ``` vcluster connect [vcluster-name] --project [project-name] --driver platform ``` tip The platform uses Helm to manage virtual clusters. If your cluster is running in an air-gapped environment, you may host Helm charts in an OCI compatible private registry. To use a private registry for virtual clusters there are several configuration options: * Configure the platform for [offline Helm charts](https://vcluster.com/docs/platform/install/air-gapped.md#pull-and-push-the-vcluster-helm-chart) * Configure the Helm chart repository and authentication for each [virtual cluster](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance/.md). * Configure the Helm chart repository and authentication using [virtual cluster templates](https://vcluster.com/docs/platform/api/resources/virtualclustertemplate.md). --- # Create a tenant cluster from a template Each tenant cluster that is created in the platform belongs to a [project](https://vcluster.com/docs/platform/understand/what-are-projects.md). There are two primary ways which tenant clusters can be created: from a template or manually. Tenant clusters created from a [template](https://vcluster.com/docs/platform/understand/what-are-templates.md) inherit all template settings. Standard project users (non project admins) are only allowed to create tenant clusters from templates that the project admin has allowed. This ensures that tenant clusters in each project adhere to the standards set by the project admin. Project admins and platform admins can create tenant clusters manually, that is without a template. Once you have created a tenant cluster template you can then refer to that template when creating a new tenant cluster. All template configurations will be automatically applied to the newly created tenant cluster. Tenant clusters referring to templates can be created in the UI or using the vCluster CLI. info See the [Platform CLI hub](https://vcluster.com/docs/platform/reference/platform-cli.md) for UI-to-CLI equivalents across common Platform tasks, including tenant cluster commands. * UI * CLI 1. From the project drop-down menu (top left corner), select the project you'd like to create the tenant cluster in. 2. Click on Tenant Clusters. 3. Click the Create Tenant Cluster button. 4. The system redirects you to the configure virtual cluster page. Select your template: * If your project has "require template" enabled, choose from a list of available templates * If not, click Use Template to select your template 5. Follow the steps in the UI to create the virtual cluster. 6. Retrieve a kube-context for a virtual cluster using the CLI: ``` vcluster connect [vcluster-name] --project [project-name] --driver platform ``` When creating a tenant cluster from the vCluster CLI, provide the project name to deploy into and the template name to use: ``` vcluster create [vcluster-name] --project [project-name] --template [template-name] --driver platform ``` If you omit `--template`, vCluster Platform resolves one automatically. * Uses the project's default template if one is set. * Auto-selects the template if the project only allows one. * Prompts you to choose if the project allows multiple templates without defining a default. * Fails with an error if no templates are allowed in the project. Kube-Context Running `vcluster create` adds a kube-context to your kube-config file and switches your active context to it. This allows you to immediately run `kubectl` commands for the new cluster right after creating it. Disable the automatic context switch by adding `--connect=false` to the `create` command. ## Template version strings[​](#template-version-strings "Direct link to Template version strings") Regardless of how you create a tenant cluster from a template, either by the UI or the CLI, you will be prompted to provide a template version string. If you do not provide this string, the latest version will be automatically selected for you. If you *do* provide the string though, you can do something very neat -- you can set any of the MAJOR, MINOR, PATCH version components to an `X` wildcard character. This allows the tenant cluster to be automatically updated to a more recent template version that matches your provided template string. For example, given a template with a version of 1.0.0, and a tenant cluster created from this template with a version string "1.X.X". Adding a new template version "1.1.0", will cause this tenant cluster to be automatically updated. Whereas adding a new version "2.0.0", will *not* cause the tenant cluster to be upgraded. This is a very handy way to keep tenant clusters up to date with the latest templates without having to manually upgrade them. Note that you can even set the version to "X.X.X" to *always* have your tenant cluster updated to the latest template version. tip The platform uses Helm to manage virtual clusters. If your cluster is running in an air-gapped environment, you may host Helm charts in an OCI compatible private registry. To use a private registry for virtual clusters there are several configuration options: * Configure the platform for [offline Helm charts](https://vcluster.com/docs/platform/install/air-gapped.md#pull-and-push-the-vcluster-helm-chart) * Configure the Helm chart repository and authentication for each [virtual cluster](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance/.md). * Configure the Helm chart repository and authentication using [virtual cluster templates](https://vcluster.com/docs/platform/api/resources/virtualclustertemplate.md). --- # Custom Links vCluster Platform lets you associate custom links to tenant cluster instances, for example to point to a github pull request or jira issue. * Platform UI * CLI 1. Select your project from the project selector, then click Tenant Clusters. 2. Edit an existing tenant cluster or create a new one. 3. Click the edit icon next to the links section to open the links editor. Click Add to add a new link. 4. Enter a URL and optionally a label for your link. 5. Click Create Tenant Cluster, or Save Changes if you're editing an existing tenant cluster, to apply all changes. ``` vcluster create vcluster-1 --link 'my-issue=https://github.com/kubernetes/kubernetes/pull/1234' vcluster create vcluster-1 --link 'https://loft.sh' vcluster create vcluster-1 --link 'my-issue=https://github.com/kubernetes/kubernetes/pull/1234,documentation=https://example.com/docs' ``` --- # Kubectl shell Platform administrators and users alike often find themselves in a situation where they just need to execute a couple of kubectl commands against a cluster to troubleshoot or get a specific piece of information from it. The Kubectl Shell feature is a terminal that is connected to a pod containing kubectl, among other tools. The pod exists in the target vCluster within the namespace `vcluster-platform-shell`. A kubeconfig is mounted to the pod with a vCluster scoped AccessKey. This means the AccessKey cannot be used to access other resources in vCluster Platform. The Kubectl Shell Pod and AccessKey will automatically be cleaned up after 15 minutes of inactivity. To learn more about the security implications and options for security hardening, read the [Security Considerations section](#security-considerations). ## How to use[​](#how-to-use "Direct link to How to use") The Kubectl Shell feature can be accessed in the Project Overview page under the three-dot menu for any vCluster row. The Kubectl Shell feature can also be accessed on the vCluster Config page under the three-dot menu. ## Who can use it[​](#who-can-use-it "Direct link to Who can use it") Any user who has access to the VirtualClusterInstance, VirtualClusterInstances/Shell, and pod/exec permissions for the `vcluster-browser-shell` namespace within the vCluster can use the Kubectl Shell feature. Read the [Security Consideration section](#security-considerations) to understand the implications of granting these aforementioned permissions. The following YAML can be used to create a project role that grants the VirtualClusterInstance permissions needed for the Kubectl Shell feature: ``` kind: ClusterRoleTemplate apiVersion: management.loft.sh/v1 metadata: name: kubectlshell labels: loft.sh/project-role: 'true' spec: displayName: kubectlshell owner: user: admin management: true clusterRoleTemplate: metadata: {} rules: - verbs: - '*' apiGroups: - management.loft.sh resources: - virtualclusterinstances - virtualclusterinstances/shell ``` By default, users with VirtualClusterInstance access are granted the `cluster-admin` ClusterRole in the vCluster, which will grant the necessary pod/exec permission for the Kubectl Shell feature. Any ClusterRole that grants or Role that grants the following permission will suffice instead if you configure custom RBAC settings: ``` rules: - apiGroups: - "" resources: - pods/attach - pods/exec verbs: - create ``` If using a RoleBinding and Role, you would need to target the `vcluster-platform-shell` namespace. ## Security considerations[​](#security-considerations "Direct link to Security considerations") Required permissions for this feature should only be given to users who are otherwise vCluster admins. vCluster admins, not to be confused with vCluster Platform admins, are any users who have full access or effectively full access to perform sensitive actions within the targeted vCluster. The Kubectl Shell pods exist within the vCluster in the `vcluster-platform-shell` namespace. The kubeconfig is mounted from a secret that also exists in the `vcluster-platform-shell` namespace. Any user that has access to either exec into pods or get secrets in the `vcluster-platform-shell` have access to the kubeconfig and the AccessKey token embedded within the kubeconfig. The AccessKey is scoped to the vCluster, meaning it can only be used to access vCluster related resources and cannot be used to access any other resources in the vCluster Platform. Consequently, these permissions should only be granted to users who would be considered admins for the targeted vCluster. It is common for all users with access to a vCluster to effectively be admins for that vCluster. Pod exec permissions are a sensitive privilege and should only be given to vCluster admins regardless. Use normal Kubernetes RBAC to configure who has access to the `vcluster-platform-shell` namespace. --- # Prevent Deletion Accidental deletion of tenant clusters is something that can not be reverted. In order to prevent deletion of tenant clusters, a setting can be enabled on the tenant cluster instance. note If you want to delete this tenant cluster in the future, you need to remove the label either in the YAML manifest or through the toggle in the UI. Once you have done this, you can delete the tenant cluster as usual. * Within the UI * Within the YAML manifest You can prevent tenant cluster deletion, by setting the annotation `loft.sh/non-deletable: "true"` in the YAML manifest of the tenant cluster instance. ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterInstance metadata: name: my-vcluster annotations: loft.sh/non-deletable: "true" spec: templateRef: name: isolated-vcluster clusterRef: cluster: loft-cluster ``` 1. From the project drop-down menu (top left corner), select the project you'd like to create the tenant cluster in. 2. Click Tenant Clusters. 3. Click the Edit option for the tenant cluster you want to modify. 4. Navigate to Advanced section on the left. 5. Select the Prevent Deletion expander and enable Enabled. 6. Click the Save button to save the changes. --- # Manage sleep and auto-delete | | | Enterprise | | | | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Auto Sleep for Virtual Clusters](https://vcluster.com/docs/platform/use-platform/virtual-clusters/key-features/sleep-mode.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). vCluster Platform can scale down an idle tenant cluster's workloads and control plane. It can restore them after detected activity or at a scheduled time. The platform can also delete tenant clusters after an inactivity period. ## Put a tenant cluster to sleep[​](#put-a-tenant-cluster-to-sleep "Direct link to Put a tenant cluster to sleep") 1. Select the project that contains the tenant cluster. 2. Click **Tenant Clusters** in the sidebar. 3. Open the **More actions** menu for the tenant cluster. 4. Click **Sleep**. 5. Optionally select **Prevent automatic wake up** and enter a duration in hours. Enter `0` to prevent automatic wakeup indefinitely. 6. Click **Ok**. To perform the same operation with the CLI, run: ``` vcluster platform sleep vcluster --project ``` Add `--prevent-wakeup ` to block activity-based wakeup for a period. Use `--prevent-wakeup 0` to require a manual wakeup. ## Wake a tenant cluster[​](#wake-a-tenant-cluster "Direct link to Wake a tenant cluster") Open the sleeping tenant cluster's **More actions** menu, click **Wake Up**, then confirm **Wake Up**. You can also run: ``` vcluster platform wakeup vcluster --project ``` Unless wakeup is currently prevented, a Kubernetes API request routed through the platform can wake the tenant cluster automatically. Supported Ingress, Istio, and Gateway API traffic can also trigger wakeup. ## Configure auto sleep[​](#configure-auto-sleep "Direct link to Configure auto sleep") See [Configure auto sleep](https://vcluster.com/docs/vcluster/next/configure/vcluster-yaml/sleep) for the Platform UI and `vcluster.yaml` procedures, schedules, and activity detection. For a tenant cluster created from a template, configure sleep in the template. ## Configure auto-delete[​](#configure-auto-delete "Direct link to Configure auto-delete") Auto-delete permanently deletes a tenant cluster after an inactivity period. It is separate from sleep and can run whether the tenant cluster is awake or sleeping. See [Configure automatic deletion](https://vcluster.com/docs/vcluster/next/configure/vcluster-yaml/deletion) for the Platform UI and `vcluster.yaml` procedures. --- # Grant access to a tenant cluster Grant users or teams access to a tenant cluster using either the platform UI or the vCluster CLI.
* Using the UI * Using the CLI 1. Select the project from the project selector at the top left. 2. Click Tenant Clusters. 3. Click Edit on the virtual cluster that you want to edit. 4. Click Permissions. 5. Click Add Rule to select the user or team to add. If the user or team you want to grant access to is not listed, verify that they have access to the project. 6. Specify the Cluster Role you want to assign the user or team. 7. After specifying all options, click Save Changes. To grant a user access to a tenant cluster with the vCluster CLI, run the following command: ``` vcluster platform share vcluster VCLUSTER_NAME [flags] ``` See the [vCluster CLI documentation](https://vcluster.com/docs/vcluster/cli/vcluster_platform_share_vcluster) for more information. --- # Get a kubeconfig file for your tenant cluster To connect to a tenant cluster's Kubernetes API server, you can either manually create a kubeconfig file or use the vCluster CLI to generate and update it automatically. ## Retrieve using the vCluster CLI[​](#retrieve-using-the-vcluster-cli "Direct link to Retrieve using the vCluster CLI") The easiest way to retrieve a kubeconfig for your tenant cluster is by using the vCluster CLI. You can update your current kubeconfig on your local machine by running the following command: ``` vcluster platform connect vcluster my-vcluster ``` If you want to print the kubeconfig file, you can append `--print` to the command: ``` vcluster platform connect vcluster my-vcluster --print ``` info If you have multiple tenant clusters with the same name, a sub-menu allows you to specify the vCluster by project. info If you have regional [cluster endpoints](https://vcluster.com/docs/platform/administer/clusters/advanced/regional-cluster-endpoints.md) or [ingress endpoints for tenant clusters](https://vcluster.com/docs/vcluster/key-features/ingress-access.md) enabled, the resulting kubeconfig will look different. ## Create kubeconfig without the CLI[​](#create-kubeconfig-without-the-cli "Direct link to Create kubeconfig without the CLI") You can also create a kubeconfig manually by creating an [access key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) for your user first and then using the following template, with the following placeholders: * **$PROJECT**: the project the tenant cluster is in * **$VIRTUAL\_CLUSTER**: the tenant cluster name to connect to * **$PLATFORM\_URL**: the platform host you connect to * **$ACCESS\_KEY**: the access key to use Then replace the placeholders in the following template and save the configuration as `my-kubeconfig.yaml`: ``` apiVersion: v1 kind: Config clusters: - cluster: server: https://$PLATFORM_URL/kubernetes/project/$PROJECT/virtualcluster/$VIRTUAL_CLUSTER # Optional: if the server uses an insecure certificate # insecure-skip-tls-verify: true name: loft contexts: - context: cluster: loft user: loft name: loft current-context: loft users: - name: loft user: token: $ACCESS_KEY ``` You can access your tenant cluster by using the freshly created kubeconfig file: ``` KUBECONFIG=my-kubeconfig.yaml kubectl get namespaces ``` --- # Start Here vCluster provisions isolated tenant clusters, delivering managed Kubernetes, Slurm, inference, Ray, and other cluster types. The tenant clusters can go on your existing infrastructure, or they can be built from scratch on bare metal and VMs. Each tenant gets a full Kubernetes API experience while the virtualized control plane stays completely invisible to them. Planning a customer-facing or production platform? These quick starts prove the deployment mechanics. They don't select a production security architecture. [Choose a worker node model](https://vcluster.com/docs/vcluster/production-guide/choose-worker-node-model.md) first. [group](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md) ### [Shared Nodes](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md) [You have a Kubernetes cluster and want isolated environments for internal, trusted teams. Tenant workloads run on the existing node pool. No extra infrastructure needed. Not for external or resold tenants. Works with OSS CLI or Platform.](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md) [lock](https://vcluster.com/docs/vcluster/quick-start/private-nodes.md) ### [Private Nodes](https://vcluster.com/docs/vcluster/quick-start/private-nodes.md) [You need dedicated infrastructure per tenant. Each tenant cluster runs on its own nodes. Best for hard tenancy: GPU clouds, regulated platforms, and sovereign deployments. Requires vCluster Platform (free mode OK).](https://vcluster.com/docs/vcluster/quick-start/private-nodes.md) [laptop](https://vcluster.com/docs/vcluster/quick-start/docker.md) ### [Docker (vind)](https://vcluster.com/docs/vcluster/quick-start/docker.md) [You want an ephemeral cluster for local development or CI. No Kubernetes required — just Docker.](https://vcluster.com/docs/vcluster/quick-start/docker.md) [storage](https://vcluster.com/docs/vcluster/quick-start/standalone.md) ### [Standalone](https://vcluster.com/docs/vcluster/quick-start/standalone.md) [You're building a managed Kubernetes platform from bare metal or VMs. Bootstrap a Control Plane Cluster and install Platform in one pass.](https://vcluster.com/docs/vcluster/quick-start/standalone.md) ## Match your situation to a guide[​](#match-your-situation-to-a-guide "Direct link to Match your situation to a guide") * Build a production Kubernetes platform from bare metal or VMs → [Standalone](https://vcluster.com/docs/vcluster/quick-start/standalone.md) * Use an existing cluster for a GPU cloud, AI inference service, or regulated platform that needs dedicated infrastructure per tenant → [Private Nodes](https://vcluster.com/docs/vcluster/quick-start/private-nodes.md) * Use an existing Kubernetes cluster for internal platforms, CI, testing, or development with trusted teams → [Shared Nodes](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md) * Create a Kubernetes cluster on Docker for local development or CI → [Docker (vind)](https://vcluster.com/docs/vcluster/quick-start/docker.md) Still unsure? Read [What is vCluster?](https://vcluster.com/docs/vcluster/introduction/what-is-vcluster.md) to understand the deployment models and tradeoffs. --- # Control Plane Distribution support Kubernetes (K8s) is the only supported distribution. **K0s** support was removed in vCluster `v0.26` and **K3s** support was removed in `v0.33`. If your tenant clusters still use K3s, see the [K3s to K8s migration guide](https://vcluster.com/docs/vcluster/manage/upgrade/distro-migration.md) before upgrading. ## Config reference[​](#config-reference "Direct link to Config reference") ## `controlPlane` object [​](#controlPlane "Direct link to controlPlane")Configure vCluster's control plane components and deployment. ### `endpoint` string [​](#controlPlane-endpoint "Direct link to controlPlane-endpoint")Endpoint is the endpoint of the virtual cluster. This is used to connect to the virtual cluster. ### `distro` object [​](#controlPlane-distro "Direct link to controlPlane-distro")Distro holds virtual cluster related distro options. A distro cannot be changed after vCluster is deployed. #### `k8s` object [​](#controlPlane-distro-k8s "Direct link to controlPlane-distro-k8s")K8S holds K8s relevant configuration. ##### `enabled` boolean false [​](#controlPlane-distro-k8s-enabled "Direct link to controlPlane-distro-k8s-enabled")Enabled specifies if the K8s distro should be enabled. Only one distro can be enabled at the same time. ##### `version` string [​](#controlPlane-distro-k8s-version "Direct link to controlPlane-distro-k8s-version")Version is the Kubernetes version to use. ##### `apiServer` object [​](#controlPlane-distro-k8s-apiServer "Direct link to controlPlane-distro-k8s-apiServer")APIServer holds configuration specific to starting the api server. ##### `enabled` boolean true [​](#controlPlane-distro-k8s-apiServer-enabled "Direct link to controlPlane-distro-k8s-apiServer-enabled")Enabled signals this container should be enabled. ##### `command` string\[] \[] [​](#controlPlane-distro-k8s-apiServer-command "Direct link to controlPlane-distro-k8s-apiServer-command")Command is the command to start the distro binary. This will override the existing command. ##### `extraArgs` string\[] \[] [​](#controlPlane-distro-k8s-apiServer-extraArgs "Direct link to controlPlane-distro-k8s-apiServer-extraArgs")ExtraArgs are additional arguments to pass to the distro binary. ##### `controllerManager` object [​](#controlPlane-distro-k8s-controllerManager "Direct link to controlPlane-distro-k8s-controllerManager")ControllerManager holds configuration specific to starting the controller manager. ##### `enabled` boolean true [​](#controlPlane-distro-k8s-controllerManager-enabled "Direct link to controlPlane-distro-k8s-controllerManager-enabled")Enabled signals this container should be enabled. ##### `command` string\[] \[] [​](#controlPlane-distro-k8s-controllerManager-command "Direct link to controlPlane-distro-k8s-controllerManager-command")Command is the command to start the distro binary. This will override the existing command. ##### `extraArgs` string\[] \[] [​](#controlPlane-distro-k8s-controllerManager-extraArgs "Direct link to controlPlane-distro-k8s-controllerManager-extraArgs")ExtraArgs are additional arguments to pass to the distro binary. ##### `scheduler` object [​](#controlPlane-distro-k8s-scheduler "Direct link to controlPlane-distro-k8s-scheduler")Scheduler holds configuration specific to starting the scheduler. ##### `enabled` boolean false [​](#controlPlane-distro-k8s-scheduler-enabled "Direct link to controlPlane-distro-k8s-scheduler-enabled")Enabled signals this container should be enabled. ##### `command` string\[] \[] [​](#controlPlane-distro-k8s-scheduler-command "Direct link to controlPlane-distro-k8s-scheduler-command")Command is the command to start the distro binary. This will override the existing command. ##### `extraArgs` string\[] \[] [​](#controlPlane-distro-k8s-scheduler-extraArgs "Direct link to controlPlane-distro-k8s-scheduler-extraArgs")ExtraArgs are additional arguments to pass to the distro binary. ##### `image` object [​](#controlPlane-distro-k8s-image "Direct link to controlPlane-distro-k8s-image")Image is the distro image ##### `registry` string ghcr.io [​](#controlPlane-distro-k8s-image-registry "Direct link to controlPlane-distro-k8s-image-registry")Registry is the registry of the container image, e.g. my-registry.com or ghcr.io. This setting can be globally overridden via the controlPlane.advanced.defaultImageRegistry option. Empty means docker hub. ##### `repository` string loft-sh/kubernetes [​](#controlPlane-distro-k8s-image-repository "Direct link to controlPlane-distro-k8s-image-repository")Repository is the repository of the container image, e.g. my-repo/my-image ##### `tag` string v1.36.6 [​](#controlPlane-distro-k8s-image-tag "Direct link to controlPlane-distro-k8s-image-tag")Tag is the tag of the container image, and is the default version. ##### `imagePullPolicy` string [​](#controlPlane-distro-k8s-imagePullPolicy "Direct link to controlPlane-distro-k8s-imagePullPolicy")ImagePullPolicy is the pull policy for the distro image ##### `env` object\[] \[] [​](#controlPlane-distro-k8s-env "Direct link to controlPlane-distro-k8s-env")Env are extra environment variables to use for the main container and NOT the init container. ##### `resources` object map\[limits:map\[cpu:100m memory:256Mi] requests:map\[cpu:40m memory:64Mi]] [​](#controlPlane-distro-k8s-resources "Direct link to controlPlane-distro-k8s-resources")Resources for the distro init container ##### `securityContext` object {} [​](#controlPlane-distro-k8s-securityContext "Direct link to controlPlane-distro-k8s-securityContext")Security options can be used for the distro init container ### `standalone` object [​](#controlPlane-standalone "Direct link to controlPlane-standalone")Standalone holds configuration for standalone mode. Standalone mode is set automatically when no container is detected and also implies privateNodes.enabled. #### `enabled` boolean [​](#controlPlane-standalone-enabled "Direct link to controlPlane-standalone-enabled")Enabled defines if standalone mode should be enabled. #### `dataDir` string /var/lib/vcluster [​](#controlPlane-standalone-dataDir "Direct link to controlPlane-standalone-dataDir")DataDir defines the data directory for the standalone mode. #### `advertiseAddress` string [​](#controlPlane-standalone-advertiseAddress "Direct link to controlPlane-standalone-advertiseAddress")AdvertiseAddress pins the IP address the standalone control plane advertises to the kubernetes and konnectivity Services. Defaults to the VCLUSTER\_STANDALONE\_IP\_ADDRESS env var, then default-route interface detection. Set this on multi-homed hosts where the default route is not reachable from the pod network. #### `autoNodes` object [​](#controlPlane-standalone-autoNodes "Direct link to controlPlane-standalone-autoNodes")AutoNodes automatically deploys nodes for standalone mode. ##### `provider` string [​](#controlPlane-standalone-autoNodes-provider "Direct link to controlPlane-standalone-autoNodes-provider")Provider is the node provider of the nodes in this pool. ##### `quantity` integer [​](#controlPlane-standalone-autoNodes-quantity "Direct link to controlPlane-standalone-autoNodes-quantity")Quantity is the number of nodes to deploy for standalone mode. ##### `nodeTypeSelector` object\[] [​](#controlPlane-standalone-autoNodes-nodeTypeSelector "Direct link to controlPlane-standalone-autoNodes-nodeTypeSelector")NodeTypeSelector filters the types of nodes that can be provisioned by this pool. All requirements must be met for a node type to be eligible. ##### `property` required string [​](#controlPlane-standalone-autoNodes-nodeTypeSelector-property "Direct link to controlPlane-standalone-autoNodes-nodeTypeSelector-property")Property is the property on the node type to select. ##### `operator` string [​](#controlPlane-standalone-autoNodes-nodeTypeSelector-operator "Direct link to controlPlane-standalone-autoNodes-nodeTypeSelector-operator")Operator is the comparison operator, such as "In", "NotIn", "Exists". If empty, defaults to "In". ##### `values` string\[] [​](#controlPlane-standalone-autoNodes-nodeTypeSelector-values "Direct link to controlPlane-standalone-autoNodes-nodeTypeSelector-values")Values is the list of values to use for comparison. This is mutually exclusive with value. ##### `value` string [​](#controlPlane-standalone-autoNodes-nodeTypeSelector-value "Direct link to controlPlane-standalone-autoNodes-nodeTypeSelector-value")Value is the value to use for comparison. This is mutually exclusive with values. #### `joinNode` object [​](#controlPlane-standalone-joinNode "Direct link to controlPlane-standalone-joinNode")JoinNode holds configuration for the standalone control plane node. ##### `enabled` boolean true [​](#controlPlane-standalone-joinNode-enabled "Direct link to controlPlane-standalone-joinNode-enabled")Enabled defines if the standalone node should be joined into the cluster. If false, only the control plane binaries will be executed and no node will show up in the actual cluster. ##### `preInstallCommands` string\[] [​](#controlPlane-standalone-joinNode-preInstallCommands "Direct link to controlPlane-standalone-joinNode-preInstallCommands")PreInstallCommands are commands that will be executed before containerd, kubelet etc. is installed. ##### `preJoinCommands` string\[] [​](#controlPlane-standalone-joinNode-preJoinCommands "Direct link to controlPlane-standalone-joinNode-preJoinCommands")PreJoinCommands are commands that will be executed before kubeadm join is executed. ##### `postJoinCommands` string\[] [​](#controlPlane-standalone-joinNode-postJoinCommands "Direct link to controlPlane-standalone-joinNode-postJoinCommands")PostJoinCommands are commands that will be executed after kubeadm join is executed. ##### `containerd` object [​](#controlPlane-standalone-joinNode-containerd "Direct link to controlPlane-standalone-joinNode-containerd")Containerd holds configuration for the containerd join process. ##### `enabled` boolean true [​](#controlPlane-standalone-joinNode-containerd-enabled "Direct link to controlPlane-standalone-joinNode-containerd-enabled")Enabled defines if containerd should be installed and configured by vCluster. ##### `registry` object [​](#controlPlane-standalone-joinNode-containerd-registry "Direct link to controlPlane-standalone-joinNode-containerd-registry")Registry holds configuration for how containerd should be configured to use a registries. ##### `configPath` string [​](#controlPlane-standalone-joinNode-containerd-registry-configPath "Direct link to controlPlane-standalone-joinNode-containerd-registry-configPath")ConfigPath is the path to the containerd registry config. ##### `mirrors` {key: object} [​](#controlPlane-standalone-joinNode-containerd-registry-mirrors "Direct link to controlPlane-standalone-joinNode-containerd-registry-mirrors")Mirrors holds configuration for the containerd registry mirrors. E.g. myregistry.io:5000 or docker.io. See for more details. ##### `server` string [​](#controlPlane-standalone-joinNode-containerd-registry-mirrors-server "Direct link to controlPlane-standalone-joinNode-containerd-registry-mirrors-server")Server is the fallback server to use for the containerd registry mirror. E.g. . See for more details. ##### `caCert` string\[] [​](#controlPlane-standalone-joinNode-containerd-registry-mirrors-caCert "Direct link to controlPlane-standalone-joinNode-containerd-registry-mirrors-caCert")CACert are paths to CA certificates to use for the containerd registry mirror. ##### `skipVerify` boolean [​](#controlPlane-standalone-joinNode-containerd-registry-mirrors-skipVerify "Direct link to controlPlane-standalone-joinNode-containerd-registry-mirrors-skipVerify")SkipVerify is a boolean to skip the certificate verification for the containerd registry mirror and allows http connections. ##### `capabilities` string\[] [​](#controlPlane-standalone-joinNode-containerd-registry-mirrors-capabilities "Direct link to controlPlane-standalone-joinNode-containerd-registry-mirrors-capabilities")Capabilities is a list of capabilities to enable for the containerd registry mirror. If empty, will use pull and resolve capabilities. ##### `overridePath` boolean [​](#controlPlane-standalone-joinNode-containerd-registry-mirrors-overridePath "Direct link to controlPlane-standalone-joinNode-containerd-registry-mirrors-overridePath")OverridePath is a boolean to override the path for the containerd registry mirror. ##### `hosts` object\[] [​](#controlPlane-standalone-joinNode-containerd-registry-mirrors-hosts "Direct link to controlPlane-standalone-joinNode-containerd-registry-mirrors-hosts")Hosts holds configuration for the containerd registry mirror hosts. See for more details. ##### `server` string [​](#controlPlane-standalone-joinNode-containerd-registry-mirrors-hosts-server "Direct link to controlPlane-standalone-joinNode-containerd-registry-mirrors-hosts-server")Server is the server to use for the containerd registry mirror host. E.g. . ##### `caCert` string\[] [​](#controlPlane-standalone-joinNode-containerd-registry-mirrors-hosts-caCert "Direct link to controlPlane-standalone-joinNode-containerd-registry-mirrors-hosts-caCert")CACert are paths to CA certificates to use for the containerd registry mirror host. ##### `skipVerify` boolean [​](#controlPlane-standalone-joinNode-containerd-registry-mirrors-hosts-skipVerify "Direct link to controlPlane-standalone-joinNode-containerd-registry-mirrors-hosts-skipVerify")SkipVerify is a boolean to skip the certificate verification for the containerd registry mirror and allows http connections. ##### `capabilities` string\[] [​](#controlPlane-standalone-joinNode-containerd-registry-mirrors-hosts-capabilities "Direct link to controlPlane-standalone-joinNode-containerd-registry-mirrors-hosts-capabilities")Capabilities is a list of capabilities to enable for the containerd registry mirror. If empty, will use pull and resolve capabilities. ##### `overridePath` boolean [​](#controlPlane-standalone-joinNode-containerd-registry-mirrors-hosts-overridePath "Direct link to controlPlane-standalone-joinNode-containerd-registry-mirrors-hosts-overridePath")OverridePath is a boolean to override the path for the containerd registry mirror. ##### `auth` {key: object} [​](#controlPlane-standalone-joinNode-containerd-registry-auth "Direct link to controlPlane-standalone-joinNode-containerd-registry-auth")Auth holds configuration for the containerd registry auth. See for more details. ##### `username` string [​](#controlPlane-standalone-joinNode-containerd-registry-auth-username "Direct link to controlPlane-standalone-joinNode-containerd-registry-auth-username")Username is the username for the containerd registry. ##### `password` string [​](#controlPlane-standalone-joinNode-containerd-registry-auth-password "Direct link to controlPlane-standalone-joinNode-containerd-registry-auth-password")Password is the password for the containerd registry. ##### `identityToken` string [​](#controlPlane-standalone-joinNode-containerd-registry-auth-identityToken "Direct link to controlPlane-standalone-joinNode-containerd-registry-auth-identityToken")IdentityToken is the token for the containerd registry. ##### `auth` string [​](#controlPlane-standalone-joinNode-containerd-registry-auth-auth "Direct link to controlPlane-standalone-joinNode-containerd-registry-auth-auth")Auth is the auth config for the containerd registry. ##### `pauseImage` string [​](#controlPlane-standalone-joinNode-containerd-pauseImage "Direct link to controlPlane-standalone-joinNode-containerd-pauseImage")PauseImage is the image for the pause container. ##### `caCertPath` string [​](#controlPlane-standalone-joinNode-caCertPath "Direct link to controlPlane-standalone-joinNode-caCertPath")CACertPath is the path to the SSL certificate authority used to secure communications between node and control-plane. Defaults to "/etc/kubernetes/pki/ca.crt". ##### `skipPhases` string\[] [​](#controlPlane-standalone-joinNode-skipPhases "Direct link to controlPlane-standalone-joinNode-skipPhases")SkipPhases is a list of phases to skip during command execution. The list of phases can be obtained with the "kubeadm join --help" command. ##### `nodeRegistration` object [​](#controlPlane-standalone-joinNode-nodeRegistration "Direct link to controlPlane-standalone-joinNode-nodeRegistration")NodeRegistration holds configuration for the node registration similar to the kubeadm node registration. ##### `criSocket` string [​](#controlPlane-standalone-joinNode-nodeRegistration-criSocket "Direct link to controlPlane-standalone-joinNode-nodeRegistration-criSocket")CRI socket is the socket for the CRI. ##### `kubeletExtraArgs` object\[] [​](#controlPlane-standalone-joinNode-nodeRegistration-kubeletExtraArgs "Direct link to controlPlane-standalone-joinNode-nodeRegistration-kubeletExtraArgs")KubeletExtraArgs passes through extra arguments to the kubelet. The arguments here are passed to the kubelet command line via the environment file kubeadm writes at runtime for the kubelet to source. This overrides the generic base-level configuration in the kubelet-config ConfigMap Flags have higher priority when parsing. These values are local and specific to the node kubeadm is executing on. An argument name in this list is the flag name as it appears on the command line except without leading dash(es). Extra arguments will override existing default arguments. Duplicate extra arguments are allowed. ##### `name` string [​](#controlPlane-standalone-joinNode-nodeRegistration-kubeletExtraArgs-name "Direct link to controlPlane-standalone-joinNode-nodeRegistration-kubeletExtraArgs-name")Name is the name of the argument. ##### `value` string [​](#controlPlane-standalone-joinNode-nodeRegistration-kubeletExtraArgs-value "Direct link to controlPlane-standalone-joinNode-nodeRegistration-kubeletExtraArgs-value")Value is the value of the argument. ##### `taints` object\[] [​](#controlPlane-standalone-joinNode-nodeRegistration-taints "Direct link to controlPlane-standalone-joinNode-nodeRegistration-taints")Taints are additional taints to set for the kubelet. ##### `key` string [​](#controlPlane-standalone-joinNode-nodeRegistration-taints-key "Direct link to controlPlane-standalone-joinNode-nodeRegistration-taints-key")Required. The taint key to be applied to a node. ##### `value` string [​](#controlPlane-standalone-joinNode-nodeRegistration-taints-value "Direct link to controlPlane-standalone-joinNode-nodeRegistration-taints-value")The taint value corresponding to the taint key. ##### `effect` string [​](#controlPlane-standalone-joinNode-nodeRegistration-taints-effect "Direct link to controlPlane-standalone-joinNode-nodeRegistration-taints-effect")Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. ##### `ignorePreflightErrors` string\[] [​](#controlPlane-standalone-joinNode-nodeRegistration-ignorePreflightErrors "Direct link to controlPlane-standalone-joinNode-nodeRegistration-ignorePreflightErrors")IgnorePreflightErrors provides a slice of pre-flight errors to be ignored when the current node is registered, e.g. 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. ##### `imagePullPolicy` string [​](#controlPlane-standalone-joinNode-nodeRegistration-imagePullPolicy "Direct link to controlPlane-standalone-joinNode-nodeRegistration-imagePullPolicy")ImagePullPolicy specifies the policy for image pulling during kubeadm "init" and "join" operations. The value of this field must be one of "Always", "IfNotPresent" or "Never". If this field is unset kubeadm will default it to "IfNotPresent", or pull the required images if not present on the host. ### `backingStore` object [​](#controlPlane-backingStore "Direct link to controlPlane-backingStore")BackingStore defines which backing store to use for virtual cluster. If not defined will use embedded database as a default backing store. #### `etcd` object [​](#controlPlane-backingStore-etcd "Direct link to controlPlane-backingStore-etcd")Etcd defines that etcd should be used as the backend for the virtual cluster ##### `embedded` object [​](#controlPlane-backingStore-etcd-embedded "Direct link to controlPlane-backingStore-etcd-embedded")Embedded defines to use embedded etcd as a storage backend for the virtual cluster ##### `enabled` boolean false [​](#controlPlane-backingStore-etcd-embedded-enabled "Direct link to controlPlane-backingStore-etcd-embedded-enabled")Enabled defines if the embedded etcd should be used. ##### `migrateFromDeployedEtcd` boolean false [​](#controlPlane-backingStore-etcd-embedded-migrateFromDeployedEtcd "Direct link to controlPlane-backingStore-etcd-embedded-migrateFromDeployedEtcd")MigrateFromDeployedEtcd signals that vCluster should migrate from the deployed external etcd to embedded etcd. ##### `snapshotCount` integer [​](#controlPlane-backingStore-etcd-embedded-snapshotCount "Direct link to controlPlane-backingStore-etcd-embedded-snapshotCount")SnapshotCount defines the number of snapshots to keep for the embedded etcd. Defaults to 10000 if less than 1. ##### `extraArgs` string\[] \[] [​](#controlPlane-backingStore-etcd-embedded-extraArgs "Direct link to controlPlane-backingStore-etcd-embedded-extraArgs")ExtraArgs are additional arguments to pass to the embedded etcd. ##### `deploy` object [​](#controlPlane-backingStore-etcd-deploy "Direct link to controlPlane-backingStore-etcd-deploy")Deploy defines to use an external etcd that is deployed by the helm chart ##### `enabled` boolean false [​](#controlPlane-backingStore-etcd-deploy-enabled "Direct link to controlPlane-backingStore-etcd-deploy-enabled")Enabled defines that an external etcd should be deployed. ##### `statefulSet` object [​](#controlPlane-backingStore-etcd-deploy-statefulSet "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet")StatefulSet holds options for the external etcd statefulSet. ##### `enabled` boolean true [​](#controlPlane-backingStore-etcd-deploy-statefulSet-enabled "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-enabled")Enabled defines if the statefulSet should be deployed ##### `enableServiceLinks` boolean true [​](#controlPlane-backingStore-etcd-deploy-statefulSet-enableServiceLinks "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-enableServiceLinks")EnableServiceLinks for the StatefulSet pod ##### `image` object [​](#controlPlane-backingStore-etcd-deploy-statefulSet-image "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-image")Image is the image to use for the external etcd statefulSet ##### `registry` string registry.k8s.io [​](#controlPlane-backingStore-etcd-deploy-statefulSet-image-registry "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-image-registry")Registry is the registry of the container image, e.g. my-registry.com or ghcr.io. This setting can be globally overridden via the controlPlane.advanced.defaultImageRegistry option. Empty means docker hub. ##### `repository` string etcd [​](#controlPlane-backingStore-etcd-deploy-statefulSet-image-repository "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-image-repository")Repository is the repository of the container image, e.g. my-repo/my-image ##### `tag` string 3.6.16-0 [​](#controlPlane-backingStore-etcd-deploy-statefulSet-image-tag "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-image-tag")Tag is the tag of the container image, and is the default version. ##### `imagePullPolicy` string [​](#controlPlane-backingStore-etcd-deploy-statefulSet-imagePullPolicy "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-imagePullPolicy")ImagePullPolicy is the pull policy for the external etcd image ##### `env` object\[] \[] [​](#controlPlane-backingStore-etcd-deploy-statefulSet-env "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-env")Env are extra environment variables ##### `extraArgs` string\[] \[] [​](#controlPlane-backingStore-etcd-deploy-statefulSet-extraArgs "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-extraArgs")ExtraArgs are appended to the etcd command. ##### `resources` object [​](#controlPlane-backingStore-etcd-deploy-statefulSet-resources "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-resources")Resources the etcd can consume ##### `limits` object [​](#controlPlane-backingStore-etcd-deploy-statefulSet-resources-limits "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-resources-limits")Limits are resource limits for the container ##### `requests` object map\[cpu:20m memory:150Mi] [​](#controlPlane-backingStore-etcd-deploy-statefulSet-resources-requests "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-resources-requests")Requests are minimal resources that will be consumed by the container ##### `pods` object [​](#controlPlane-backingStore-etcd-deploy-statefulSet-pods "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-pods")Pods defines extra metadata for the etcd pods. ##### `annotations` object {} [​](#controlPlane-backingStore-etcd-deploy-statefulSet-pods-annotations "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-pods-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#controlPlane-backingStore-etcd-deploy-statefulSet-pods-labels "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-pods-labels")Labels are extra labels for this resource. ##### `highAvailability` object [​](#controlPlane-backingStore-etcd-deploy-statefulSet-highAvailability "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-highAvailability")HighAvailability are high availability options ##### `replicas` integer 1 [​](#controlPlane-backingStore-etcd-deploy-statefulSet-highAvailability-replicas "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-highAvailability-replicas")Replicas are the amount of pods to use. ##### `scheduling` object [​](#controlPlane-backingStore-etcd-deploy-statefulSet-scheduling "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-scheduling")Scheduling options for the etcd pods. ##### `nodeSelector` object {} [​](#controlPlane-backingStore-etcd-deploy-statefulSet-scheduling-nodeSelector "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-scheduling-nodeSelector")NodeSelector is the node selector to apply to the pod. ##### `affinity` object {} [​](#controlPlane-backingStore-etcd-deploy-statefulSet-scheduling-affinity "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-scheduling-affinity")Affinity is the affinity to apply to the pod. ##### `tolerations` object\[] \[] [​](#controlPlane-backingStore-etcd-deploy-statefulSet-scheduling-tolerations "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-scheduling-tolerations")Tolerations are the tolerations to apply to the pod. ##### `priorityClassName` string [​](#controlPlane-backingStore-etcd-deploy-statefulSet-scheduling-priorityClassName "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-scheduling-priorityClassName")PriorityClassName is the priority class name for the the pod. ##### `podManagementPolicy` string Parallel [​](#controlPlane-backingStore-etcd-deploy-statefulSet-scheduling-podManagementPolicy "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-scheduling-podManagementPolicy")PodManagementPolicy is the statefulSet pod management policy. ##### `topologySpreadConstraints` object\[] \[] [​](#controlPlane-backingStore-etcd-deploy-statefulSet-scheduling-topologySpreadConstraints "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-scheduling-topologySpreadConstraints")TopologySpreadConstraints are the topology spread constraints for the pod. ##### `security` object [​](#controlPlane-backingStore-etcd-deploy-statefulSet-security "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-security")Security options for the etcd pods. ##### `podSecurityContext` object {} [​](#controlPlane-backingStore-etcd-deploy-statefulSet-security-podSecurityContext "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-security-podSecurityContext")PodSecurityContext specifies security context options on the pod level. ##### `containerSecurityContext` object {} [​](#controlPlane-backingStore-etcd-deploy-statefulSet-security-containerSecurityContext "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-security-containerSecurityContext")ContainerSecurityContext specifies security context options on the container level. ##### `persistence` object [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence")Persistence options for the etcd pods. ##### `volumeClaim` object [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-volumeClaim "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-volumeClaim")VolumeClaim can be used to configure the persistent volume claim. ##### `enabled` boolean true [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-enabled "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-enabled")Enabled enables deploying a persistent volume claim. ##### `accessModes` string\[] \[ReadWriteOnce] [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-accessModes "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-accessModes")AccessModes are the persistent volume claim access modes. ##### `retentionPolicy` string Retain [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-retentionPolicy "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-retentionPolicy")RetentionPolicy is the persistent volume claim retention policy. ##### `size` string 5Gi [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-size "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-size")Size is the persistent volume claim storage size. ##### `storageClass` string [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-storageClass "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-storageClass")StorageClass is the persistent volume claim storage class. ##### `volumeClaimTemplates` object\[] \[] [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-volumeClaimTemplates "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-volumeClaimTemplates")VolumeClaimTemplates defines the volumeClaimTemplates for the statefulSet ##### `addVolumes` object\[] \[] [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumes "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumes")AddVolumes defines extra volumes for the pod ##### `addVolumeMounts` object\[] [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts")AddVolumeMounts defines extra volume mounts for the container ##### `name` string [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-name "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-name")This must match the Name of a Volume. ##### `readOnly` boolean [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-readOnly "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-readOnly")Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false. ##### `mountPath` string [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-mountPath "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-mountPath")Path within the container at which the volume should be mounted. Must not contain ':'. ##### `subPath` string [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-subPath "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-subPath")Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root). ##### `mountPropagation` string [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-mountPropagation "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-mountPropagation")mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10. ##### `subPathExpr` string [​](#controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-subPathExpr "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-subPathExpr")Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR\_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. ##### `annotations` object {} [​](#controlPlane-backingStore-etcd-deploy-statefulSet-annotations "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#controlPlane-backingStore-etcd-deploy-statefulSet-labels "Direct link to controlPlane-backingStore-etcd-deploy-statefulSet-labels")Labels are extra labels for this resource. ##### `service` object [​](#controlPlane-backingStore-etcd-deploy-service "Direct link to controlPlane-backingStore-etcd-deploy-service")Service holds options for the external etcd service. ##### `enabled` boolean true [​](#controlPlane-backingStore-etcd-deploy-service-enabled "Direct link to controlPlane-backingStore-etcd-deploy-service-enabled")Enabled defines if the etcd service should be deployed ##### `annotations` object {} [​](#controlPlane-backingStore-etcd-deploy-service-annotations "Direct link to controlPlane-backingStore-etcd-deploy-service-annotations")Annotations are extra annotations for the external etcd service ##### `headlessService` object [​](#controlPlane-backingStore-etcd-deploy-headlessService "Direct link to controlPlane-backingStore-etcd-deploy-headlessService")HeadlessService holds options for the external etcd headless service. ##### `annotations` object {} [​](#controlPlane-backingStore-etcd-deploy-headlessService-annotations "Direct link to controlPlane-backingStore-etcd-deploy-headlessService-annotations")Annotations are extra annotations for the external etcd headless service ##### `external` object [​](#controlPlane-backingStore-etcd-external "Direct link to controlPlane-backingStore-etcd-external")External defines to use a self-hosted external etcd that is not deployed by the helm chart ##### `enabled` boolean false [​](#controlPlane-backingStore-etcd-external-enabled "Direct link to controlPlane-backingStore-etcd-external-enabled")Enabled defines if the external etcd should be used. ##### `endpoint` string [​](#controlPlane-backingStore-etcd-external-endpoint "Direct link to controlPlane-backingStore-etcd-external-endpoint")Endpoint holds the endpoint of the external etcd server, e.g. my-example-service:2379 ##### `tls` object [​](#controlPlane-backingStore-etcd-external-tls "Direct link to controlPlane-backingStore-etcd-external-tls")TLS defines the tls configuration for the external etcd server ##### `caFile` string [​](#controlPlane-backingStore-etcd-external-tls-caFile "Direct link to controlPlane-backingStore-etcd-external-tls-caFile")CaFile is the path to the ca file ##### `certFile` string [​](#controlPlane-backingStore-etcd-external-tls-certFile "Direct link to controlPlane-backingStore-etcd-external-tls-certFile")CertFile is the path to the cert file ##### `keyFile` string [​](#controlPlane-backingStore-etcd-external-tls-keyFile "Direct link to controlPlane-backingStore-etcd-external-tls-keyFile")KeyFile is the path to the key file #### `database` object [​](#controlPlane-backingStore-database "Direct link to controlPlane-backingStore-database")Database defines that a database backend should be used as the backend for the virtual cluster. This uses a project called kine under the hood which is a shim for bridging Kubernetes and relational databases. ##### `embedded` object [​](#controlPlane-backingStore-database-embedded "Direct link to controlPlane-backingStore-database-embedded")Embedded defines that an embedded database (sqlite) should be used as the backend for the virtual cluster ##### `enabled` boolean false [​](#controlPlane-backingStore-database-embedded-enabled "Direct link to controlPlane-backingStore-database-embedded-enabled")Enabled defines if the database should be used. ##### `dataSource` string [​](#controlPlane-backingStore-database-embedded-dataSource "Direct link to controlPlane-backingStore-database-embedded-dataSource")DataSource is the kine dataSource to use for the database. This depends on the database format. This is optional for the external database. Examples:- mysql: mysql://username:password\@tcp(hostname:3306)/vcluster - postgres: postgres\://username:password\@hostname:5432/vcluster ##### `identityProvider` string [​](#controlPlane-backingStore-database-embedded-identityProvider "Direct link to controlPlane-backingStore-database-embedded-identityProvider")IdentityProvider is the kine identity provider to use when generating temporary authentication tokens for enhanced security. This is optional for the external database. Examples:- aws: RDS IAM Authentication ##### `keyFile` string [​](#controlPlane-backingStore-database-embedded-keyFile "Direct link to controlPlane-backingStore-database-embedded-keyFile")KeyFile is the key file to use for the database. This is optional. ##### `certFile` string [​](#controlPlane-backingStore-database-embedded-certFile "Direct link to controlPlane-backingStore-database-embedded-certFile")CertFile is the cert file to use for the database. This is optional. ##### `caFile` string [​](#controlPlane-backingStore-database-embedded-caFile "Direct link to controlPlane-backingStore-database-embedded-caFile")CaFile is the ca file to use for the database. This is optional. ##### `extraArgs` string\[] \[] [​](#controlPlane-backingStore-database-embedded-extraArgs "Direct link to controlPlane-backingStore-database-embedded-extraArgs")ExtraArgs are additional arguments to pass to Kine. ##### `external` object [​](#controlPlane-backingStore-database-external "Direct link to controlPlane-backingStore-database-external")External defines that an external database should be used as the backend for the virtual cluster ##### `enabled` boolean false [​](#controlPlane-backingStore-database-external-enabled "Direct link to controlPlane-backingStore-database-external-enabled")Enabled defines if the database should be used. ##### `dataSource` string [​](#controlPlane-backingStore-database-external-dataSource "Direct link to controlPlane-backingStore-database-external-dataSource")DataSource is the kine dataSource to use for the database. This depends on the database format. This is optional for the external database. Examples:- mysql: mysql://username:password\@tcp(hostname:3306)/vcluster - postgres: postgres\://username:password\@hostname:5432/vcluster ##### `identityProvider` string [​](#controlPlane-backingStore-database-external-identityProvider "Direct link to controlPlane-backingStore-database-external-identityProvider")IdentityProvider is the kine identity provider to use when generating temporary authentication tokens for enhanced security. This is optional for the external database. Examples:- aws: RDS IAM Authentication ##### `keyFile` string [​](#controlPlane-backingStore-database-external-keyFile "Direct link to controlPlane-backingStore-database-external-keyFile")KeyFile is the key file to use for the database. This is optional. ##### `certFile` string [​](#controlPlane-backingStore-database-external-certFile "Direct link to controlPlane-backingStore-database-external-certFile")CertFile is the cert file to use for the database. This is optional. ##### `caFile` string [​](#controlPlane-backingStore-database-external-caFile "Direct link to controlPlane-backingStore-database-external-caFile")CaFile is the ca file to use for the database. This is optional. ##### `extraArgs` string\[] \[] [​](#controlPlane-backingStore-database-external-extraArgs "Direct link to controlPlane-backingStore-database-external-extraArgs")ExtraArgs are additional arguments to pass to Kine. ##### `connector` string [​](#controlPlane-backingStore-database-external-connector "Direct link to controlPlane-backingStore-database-external-connector")Connector specifies a secret located in a connected vCluster Platform that contains database server connection information to be used by Platform to create a database and database user for the vCluster. and non-privileged user. A kine endpoint should be created using the database and user on Platform registration. This is optional. ### `coredns` object [​](#controlPlane-coredns "Direct link to controlPlane-coredns")CoreDNS defines everything related to the coredns that is deployed and used within the vCluster. #### `enabled` boolean true [​](#controlPlane-coredns-enabled "Direct link to controlPlane-coredns-enabled")Enabled defines if coredns is enabled #### `embedded` boolean false [​](#controlPlane-coredns-embedded "Direct link to controlPlane-coredns-embedded")Embedded defines if vCluster will start the embedded coredns service within the control-plane and not as a separate deployment. This is a PRO feature. #### `security` object [​](#controlPlane-coredns-security "Direct link to controlPlane-coredns-security")Security defines pod or container security context. ##### `podSecurityContext` object {} [​](#controlPlane-coredns-security-podSecurityContext "Direct link to controlPlane-coredns-security-podSecurityContext")PodSecurityContext specifies security context options on the pod level. ##### `containerSecurityContext` object {} [​](#controlPlane-coredns-security-containerSecurityContext "Direct link to controlPlane-coredns-security-containerSecurityContext")ContainerSecurityContext specifies security context options on the container level. #### `service` object [​](#controlPlane-coredns-service "Direct link to controlPlane-coredns-service")Service holds extra options for the coredns service deployed within the virtual cluster ##### `spec` object map\[type:ClusterIP] [​](#controlPlane-coredns-service-spec "Direct link to controlPlane-coredns-service-spec")Spec holds extra options for the coredns service ##### `annotations` object {} [​](#controlPlane-coredns-service-annotations "Direct link to controlPlane-coredns-service-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#controlPlane-coredns-service-labels "Direct link to controlPlane-coredns-service-labels")Labels are extra labels for this resource. #### `deployment` object [​](#controlPlane-coredns-deployment "Direct link to controlPlane-coredns-deployment")Deployment holds extra options for the coredns deployment deployed within the virtual cluster ##### `image` string [​](#controlPlane-coredns-deployment-image "Direct link to controlPlane-coredns-deployment-image")Image is the coredns image to use ##### `replicas` integer 1 [​](#controlPlane-coredns-deployment-replicas "Direct link to controlPlane-coredns-deployment-replicas")Replicas is the amount of coredns pods to run. ##### `nodeSelector` object {} [​](#controlPlane-coredns-deployment-nodeSelector "Direct link to controlPlane-coredns-deployment-nodeSelector")NodeSelector is the node selector to use for coredns. ##### `affinity` object {} [​](#controlPlane-coredns-deployment-affinity "Direct link to controlPlane-coredns-deployment-affinity")Affinity is the affinity to apply to the pod. ##### `tolerations` object\[] \[] [​](#controlPlane-coredns-deployment-tolerations "Direct link to controlPlane-coredns-deployment-tolerations")Tolerations are the tolerations to apply to the pod. ##### `resources` object [​](#controlPlane-coredns-deployment-resources "Direct link to controlPlane-coredns-deployment-resources")Resources are the desired resources for coredns. ##### `limits` object map\[cpu:1000m memory:170Mi] [​](#controlPlane-coredns-deployment-resources-limits "Direct link to controlPlane-coredns-deployment-resources-limits")Limits are resource limits for the container ##### `requests` object map\[cpu:20m memory:64Mi] [​](#controlPlane-coredns-deployment-resources-requests "Direct link to controlPlane-coredns-deployment-resources-requests")Requests are minimal resources that will be consumed by the container ##### `pods` object [​](#controlPlane-coredns-deployment-pods "Direct link to controlPlane-coredns-deployment-pods")Pods is additional metadata for the coredns pods. ##### `annotations` object {} [​](#controlPlane-coredns-deployment-pods-annotations "Direct link to controlPlane-coredns-deployment-pods-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#controlPlane-coredns-deployment-pods-labels "Direct link to controlPlane-coredns-deployment-pods-labels")Labels are extra labels for this resource. ##### `annotations` object {} [​](#controlPlane-coredns-deployment-annotations "Direct link to controlPlane-coredns-deployment-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#controlPlane-coredns-deployment-labels "Direct link to controlPlane-coredns-deployment-labels")Labels are extra labels for this resource. ##### `topologySpreadConstraints` object\[] \[map\[labelSelector:map\[matchLabels:map\[k8s-app:vcluster-kube-dns]] maxSkew:1 topologyKey:kubernetes.io/hostname whenUnsatisfiable:DoNotSchedule]] [​](#controlPlane-coredns-deployment-topologySpreadConstraints "Direct link to controlPlane-coredns-deployment-topologySpreadConstraints")TopologySpreadConstraints are the topology spread constraints for the CoreDNS pod. #### `overwriteConfig` string [​](#controlPlane-coredns-overwriteConfig "Direct link to controlPlane-coredns-overwriteConfig")OverwriteConfig can be used to overwrite the coredns config #### `overwriteManifests` string [​](#controlPlane-coredns-overwriteManifests "Direct link to controlPlane-coredns-overwriteManifests")OverwriteManifests can be used to overwrite the coredns manifests used to deploy coredns #### `priorityClassName` string [​](#controlPlane-coredns-priorityClassName "Direct link to controlPlane-coredns-priorityClassName")PriorityClassName specifies the priority class name for the CoreDNS pods. ### `proxy` object [​](#controlPlane-proxy "Direct link to controlPlane-proxy")Proxy defines options for the virtual cluster control plane proxy that is used to do authentication and intercept requests. #### `bindAddress` string 0.0.0.0 [​](#controlPlane-proxy-bindAddress "Direct link to controlPlane-proxy-bindAddress")BindAddress under which vCluster will expose the proxy. #### `port` integer 8443 [​](#controlPlane-proxy-port "Direct link to controlPlane-proxy-port")Port under which vCluster will expose the proxy. Changing port is currently not supported. #### `extraSANs` string\[] \[] [​](#controlPlane-proxy-extraSANs "Direct link to controlPlane-proxy-extraSANs")ExtraSANs are extra hostnames to sign the vCluster proxy certificate for. ### `hostPathMapper` object [​](#controlPlane-hostPathMapper "Direct link to controlPlane-hostPathMapper")HostPathMapper defines if vCluster should rewrite host paths. #### `enabled` boolean [​](#controlPlane-hostPathMapper-enabled "Direct link to controlPlane-hostPathMapper-enabled")Enabled specifies if the host path mapper will be used #### `central` boolean [​](#controlPlane-hostPathMapper-central "Direct link to controlPlane-hostPathMapper-central")Central specifies if the central host path mapper will be used ### `ingress` object [​](#controlPlane-ingress "Direct link to controlPlane-ingress")Ingress defines options for vCluster ingress deployed by Helm. #### `enabled` boolean false [​](#controlPlane-ingress-enabled "Direct link to controlPlane-ingress-enabled")Enabled defines if the control plane ingress should be enabled #### `host` string my-host.com [​](#controlPlane-ingress-host "Direct link to controlPlane-ingress-host")Host is the host where vCluster will be reachable #### `pathType` string ImplementationSpecific [​](#controlPlane-ingress-pathType "Direct link to controlPlane-ingress-pathType")PathType is the path type of the ingress #### `spec` object map\[tls:\[]] [​](#controlPlane-ingress-spec "Direct link to controlPlane-ingress-spec")Spec allows you to configure extra ingress options. #### `annotations` object map\[nginx.ingress.kubernetes.io/backend-protocol:HTTPS nginx.ingress.kubernetes.io/ssl-passthrough:true nginx.ingress.kubernetes.io/ssl-redirect:true] [​](#controlPlane-ingress-annotations "Direct link to controlPlane-ingress-annotations")Annotations are extra annotations for this resource. #### `labels` object {} [​](#controlPlane-ingress-labels "Direct link to controlPlane-ingress-labels")Labels are extra labels for this resource. ### `tlsRoute` object [​](#controlPlane-tlsRoute "Direct link to controlPlane-tlsRoute")TLSRoute defines options for vCluster TLS route deployed by Helm. #### `enabled` boolean false [​](#controlPlane-tlsRoute-enabled "Direct link to controlPlane-tlsRoute-enabled")Enabled defines if the control plane should be exposed via a gateway api tls route. Make sure to enable tls passthrough in the gateway via tls.mode to "Passthrough" #### `apiVersion` string gateway.networking.k8s.io/v1 [​](#controlPlane-tlsRoute-apiVersion "Direct link to controlPlane-tlsRoute-apiVersion")APIVersion is the version of the gateway api tls route. #### `host` string my-host.com [​](#controlPlane-tlsRoute-host "Direct link to controlPlane-tlsRoute-host")Host is the host where vCluster will be reachable #### `parentRefs` object\[] \[] [​](#controlPlane-tlsRoute-parentRefs "Direct link to controlPlane-tlsRoute-parentRefs")ParentRefs are the parent references for the TLS route #### `spec` object {} [​](#controlPlane-tlsRoute-spec "Direct link to controlPlane-tlsRoute-spec")Spec allows you to configure extra tls route options. #### `annotations` object {} [​](#controlPlane-tlsRoute-annotations "Direct link to controlPlane-tlsRoute-annotations")Annotations are extra annotations for this resource. #### `labels` object {} [​](#controlPlane-tlsRoute-labels "Direct link to controlPlane-tlsRoute-labels")Labels are extra labels for this resource. ### `service` object [​](#controlPlane-service "Direct link to controlPlane-service")Service defines options for vCluster service deployed by Helm. #### `enabled` boolean true [​](#controlPlane-service-enabled "Direct link to controlPlane-service-enabled")Enabled defines if the control plane service should be enabled #### `spec` object map\[type:ClusterIP] [​](#controlPlane-service-spec "Direct link to controlPlane-service-spec")Spec allows you to configure extra service options. #### `kubeletNodePort` integer 0 [​](#controlPlane-service-kubeletNodePort "Direct link to controlPlane-service-kubeletNodePort")KubeletNodePort is the node port where the fake kubelet is exposed. Defaults to 0. #### `httpsNodePort` integer 0 [​](#controlPlane-service-httpsNodePort "Direct link to controlPlane-service-httpsNodePort")HTTPSNodePort is the node port where https is exposed. Defaults to 0. #### `annotations` object {} [​](#controlPlane-service-annotations "Direct link to controlPlane-service-annotations")Annotations are extra annotations for this resource. #### `labels` object {} [​](#controlPlane-service-labels "Direct link to controlPlane-service-labels")Labels are extra labels for this resource. ### `statefulSet` object [​](#controlPlane-statefulSet "Direct link to controlPlane-statefulSet")StatefulSet defines options for vCluster statefulSet deployed by Helm. #### `highAvailability` object [​](#controlPlane-statefulSet-highAvailability "Direct link to controlPlane-statefulSet-highAvailability")HighAvailability holds options related to high availability. ##### `replicas` integer 1 [​](#controlPlane-statefulSet-highAvailability-replicas "Direct link to controlPlane-statefulSet-highAvailability-replicas")Replicas is the amount of replicas to use for the statefulSet. ##### `leaseDuration` integer 60 [​](#controlPlane-statefulSet-highAvailability-leaseDuration "Direct link to controlPlane-statefulSet-highAvailability-leaseDuration")LeaseDuration is the time to lease for the leader. ##### `renewDeadline` integer 40 [​](#controlPlane-statefulSet-highAvailability-renewDeadline "Direct link to controlPlane-statefulSet-highAvailability-renewDeadline")RenewDeadline is the deadline to renew a lease for the leader. ##### `retryPeriod` integer 15 [​](#controlPlane-statefulSet-highAvailability-retryPeriod "Direct link to controlPlane-statefulSet-highAvailability-retryPeriod")RetryPeriod is the time until a replica will retry to get a lease. #### `resources` object [​](#controlPlane-statefulSet-resources "Direct link to controlPlane-statefulSet-resources")Resources are the resource requests and limits for the statefulSet container. ##### `limits` object map\[ephemeral-storage:10Gi memory:4Gi] [​](#controlPlane-statefulSet-resources-limits "Direct link to controlPlane-statefulSet-resources-limits")Limits are resource limits for the container ##### `requests` object map\[cpu:200m ephemeral-storage:1Gi memory:256Mi] [​](#controlPlane-statefulSet-resources-requests "Direct link to controlPlane-statefulSet-resources-requests")Requests are minimal resources that will be consumed by the container #### `scheduling` object [​](#controlPlane-statefulSet-scheduling "Direct link to controlPlane-statefulSet-scheduling")Scheduling holds options related to scheduling. ##### `nodeSelector` object {} [​](#controlPlane-statefulSet-scheduling-nodeSelector "Direct link to controlPlane-statefulSet-scheduling-nodeSelector")NodeSelector is the node selector to apply to the pod. ##### `affinity` object {} [​](#controlPlane-statefulSet-scheduling-affinity "Direct link to controlPlane-statefulSet-scheduling-affinity")Affinity is the affinity to apply to the pod. ##### `tolerations` object\[] \[] [​](#controlPlane-statefulSet-scheduling-tolerations "Direct link to controlPlane-statefulSet-scheduling-tolerations")Tolerations are the tolerations to apply to the pod. ##### `priorityClassName` string [​](#controlPlane-statefulSet-scheduling-priorityClassName "Direct link to controlPlane-statefulSet-scheduling-priorityClassName")PriorityClassName is the priority class name for the the pod. ##### `podManagementPolicy` string Parallel [​](#controlPlane-statefulSet-scheduling-podManagementPolicy "Direct link to controlPlane-statefulSet-scheduling-podManagementPolicy")PodManagementPolicy is the statefulSet pod management policy. ##### `topologySpreadConstraints` object\[] \[] [​](#controlPlane-statefulSet-scheduling-topologySpreadConstraints "Direct link to controlPlane-statefulSet-scheduling-topologySpreadConstraints")TopologySpreadConstraints are the topology spread constraints for the pod. #### `security` object [​](#controlPlane-statefulSet-security "Direct link to controlPlane-statefulSet-security")Security defines pod or container security context. ##### `podSecurityContext` object {} [​](#controlPlane-statefulSet-security-podSecurityContext "Direct link to controlPlane-statefulSet-security-podSecurityContext")PodSecurityContext specifies security context options on the pod level. ##### `containerSecurityContext` object map\[allowPrivilegeEscalation:false runAsGroup:0 runAsUser:0] [​](#controlPlane-statefulSet-security-containerSecurityContext "Direct link to controlPlane-statefulSet-security-containerSecurityContext")ContainerSecurityContext specifies security context options on the container level. #### `probes` object [​](#controlPlane-statefulSet-probes "Direct link to controlPlane-statefulSet-probes")Probes enables or disables the main container probes. ##### `livenessProbe` object [​](#controlPlane-statefulSet-probes-livenessProbe "Direct link to controlPlane-statefulSet-probes-livenessProbe")LivenessProbe specifies if the liveness probe for the container should be enabled ##### `enabled` boolean true [​](#controlPlane-statefulSet-probes-livenessProbe-enabled "Direct link to controlPlane-statefulSet-probes-livenessProbe-enabled")Enabled defines if this option should be enabled. ##### `failureThreshold` integer 60 [​](#controlPlane-statefulSet-probes-livenessProbe-failureThreshold "Direct link to controlPlane-statefulSet-probes-livenessProbe-failureThreshold")Number of consecutive failures for the probe to be considered failed ##### `initialDelaySeconds` integer 60 [​](#controlPlane-statefulSet-probes-livenessProbe-initialDelaySeconds "Direct link to controlPlane-statefulSet-probes-livenessProbe-initialDelaySeconds")Time (in seconds) to wait before starting the liveness probe ##### `timeoutSeconds` integer 3 [​](#controlPlane-statefulSet-probes-livenessProbe-timeoutSeconds "Direct link to controlPlane-statefulSet-probes-livenessProbe-timeoutSeconds")Maximum duration (in seconds) that the probe will wait for a response. ##### `periodSeconds` integer 2 [​](#controlPlane-statefulSet-probes-livenessProbe-periodSeconds "Direct link to controlPlane-statefulSet-probes-livenessProbe-periodSeconds")Frequency (in seconds) to perform the probe ##### `readinessProbe` object [​](#controlPlane-statefulSet-probes-readinessProbe "Direct link to controlPlane-statefulSet-probes-readinessProbe")ReadinessProbe specifies if the readiness probe for the container should be enabled ##### `enabled` boolean true [​](#controlPlane-statefulSet-probes-readinessProbe-enabled "Direct link to controlPlane-statefulSet-probes-readinessProbe-enabled")Enabled defines if this option should be enabled. ##### `failureThreshold` integer 60 [​](#controlPlane-statefulSet-probes-readinessProbe-failureThreshold "Direct link to controlPlane-statefulSet-probes-readinessProbe-failureThreshold")Number of consecutive failures for the probe to be considered failed ##### `timeoutSeconds` integer 3 [​](#controlPlane-statefulSet-probes-readinessProbe-timeoutSeconds "Direct link to controlPlane-statefulSet-probes-readinessProbe-timeoutSeconds")Maximum duration (in seconds) that the probe will wait for a response. ##### `periodSeconds` integer 2 [​](#controlPlane-statefulSet-probes-readinessProbe-periodSeconds "Direct link to controlPlane-statefulSet-probes-readinessProbe-periodSeconds")Frequency (in seconds) to perform the probe ##### `startupProbe` object [​](#controlPlane-statefulSet-probes-startupProbe "Direct link to controlPlane-statefulSet-probes-startupProbe")StartupProbe specifies if the startup probe for the container should be enabled ##### `enabled` boolean true [​](#controlPlane-statefulSet-probes-startupProbe-enabled "Direct link to controlPlane-statefulSet-probes-startupProbe-enabled")Enabled defines if this option should be enabled. ##### `failureThreshold` integer 300 [​](#controlPlane-statefulSet-probes-startupProbe-failureThreshold "Direct link to controlPlane-statefulSet-probes-startupProbe-failureThreshold")Number of consecutive failures allowed before failing the pod ##### `timeoutSeconds` integer 3 [​](#controlPlane-statefulSet-probes-startupProbe-timeoutSeconds "Direct link to controlPlane-statefulSet-probes-startupProbe-timeoutSeconds")Maximum duration (in seconds) that the probe will wait for a response. ##### `periodSeconds` integer 6 [​](#controlPlane-statefulSet-probes-startupProbe-periodSeconds "Direct link to controlPlane-statefulSet-probes-startupProbe-periodSeconds")Frequency (in seconds) to perform the probe #### `persistence` object [​](#controlPlane-statefulSet-persistence "Direct link to controlPlane-statefulSet-persistence")Persistence defines options around persistence for the statefulSet. ##### `volumeClaim` object [​](#controlPlane-statefulSet-persistence-volumeClaim "Direct link to controlPlane-statefulSet-persistence-volumeClaim")VolumeClaim can be used to configure the persistent volume claim. ##### `enabled` string|boolean auto [​](#controlPlane-statefulSet-persistence-volumeClaim-enabled "Direct link to controlPlane-statefulSet-persistence-volumeClaim-enabled")Enabled enables deploying a persistent volume claim. If auto, vCluster will automatically determine based on the chosen distro and other options if this is required. ##### `accessModes` string\[] \[ReadWriteOnce] [​](#controlPlane-statefulSet-persistence-volumeClaim-accessModes "Direct link to controlPlane-statefulSet-persistence-volumeClaim-accessModes")AccessModes are the persistent volume claim access modes. ##### `retentionPolicy` string Retain [​](#controlPlane-statefulSet-persistence-volumeClaim-retentionPolicy "Direct link to controlPlane-statefulSet-persistence-volumeClaim-retentionPolicy")RetentionPolicy is the persistent volume claim retention policy. ##### `size` string 5Gi [​](#controlPlane-statefulSet-persistence-volumeClaim-size "Direct link to controlPlane-statefulSet-persistence-volumeClaim-size")Size is the persistent volume claim storage size. ##### `storageClass` string [​](#controlPlane-statefulSet-persistence-volumeClaim-storageClass "Direct link to controlPlane-statefulSet-persistence-volumeClaim-storageClass")StorageClass is the persistent volume claim storage class. ##### `volumeClaimTemplates` object\[] \[] [​](#controlPlane-statefulSet-persistence-volumeClaimTemplates "Direct link to controlPlane-statefulSet-persistence-volumeClaimTemplates")VolumeClaimTemplates defines the volumeClaimTemplates for the statefulSet ##### `dataVolume` object\[] \[] [​](#controlPlane-statefulSet-persistence-dataVolume "Direct link to controlPlane-statefulSet-persistence-dataVolume")Allows you to override the dataVolume. Only works correctly if volumeClaim.enabled=false. ##### `binariesVolume` object\[] \[map\[emptyDir:map\[] name:binaries]] [​](#controlPlane-statefulSet-persistence-binariesVolume "Direct link to controlPlane-statefulSet-persistence-binariesVolume")BinariesVolume defines a binaries volume that is used to retrieve distro specific executables to be run by the syncer controller. This volume doesn't need to be persistent. ##### `addVolumes` object\[] \[] [​](#controlPlane-statefulSet-persistence-addVolumes "Direct link to controlPlane-statefulSet-persistence-addVolumes")AddVolumes defines extra volumes for the pod ##### `addVolumeMounts` object\[] [​](#controlPlane-statefulSet-persistence-addVolumeMounts "Direct link to controlPlane-statefulSet-persistence-addVolumeMounts")AddVolumeMounts defines extra volume mounts for the container ##### `name` string [​](#controlPlane-statefulSet-persistence-addVolumeMounts-name "Direct link to controlPlane-statefulSet-persistence-addVolumeMounts-name")This must match the Name of a Volume. ##### `readOnly` boolean [​](#controlPlane-statefulSet-persistence-addVolumeMounts-readOnly "Direct link to controlPlane-statefulSet-persistence-addVolumeMounts-readOnly")Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false. ##### `mountPath` string [​](#controlPlane-statefulSet-persistence-addVolumeMounts-mountPath "Direct link to controlPlane-statefulSet-persistence-addVolumeMounts-mountPath")Path within the container at which the volume should be mounted. Must not contain ':'. ##### `subPath` string [​](#controlPlane-statefulSet-persistence-addVolumeMounts-subPath "Direct link to controlPlane-statefulSet-persistence-addVolumeMounts-subPath")Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root). ##### `mountPropagation` string [​](#controlPlane-statefulSet-persistence-addVolumeMounts-mountPropagation "Direct link to controlPlane-statefulSet-persistence-addVolumeMounts-mountPropagation")mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10. ##### `subPathExpr` string [​](#controlPlane-statefulSet-persistence-addVolumeMounts-subPathExpr "Direct link to controlPlane-statefulSet-persistence-addVolumeMounts-subPathExpr")Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR\_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. #### `enableServiceLinks` boolean true [​](#controlPlane-statefulSet-enableServiceLinks "Direct link to controlPlane-statefulSet-enableServiceLinks")EnableServiceLinks for the StatefulSet pod #### `annotations` object {} [​](#controlPlane-statefulSet-annotations "Direct link to controlPlane-statefulSet-annotations")Annotations are extra annotations for this resource. #### `labels` object {} [​](#controlPlane-statefulSet-labels "Direct link to controlPlane-statefulSet-labels")Labels are extra labels for this resource. #### `pods` object [​](#controlPlane-statefulSet-pods "Direct link to controlPlane-statefulSet-pods")Additional labels or annotations for the statefulSet pods. ##### `annotations` object {} [​](#controlPlane-statefulSet-pods-annotations "Direct link to controlPlane-statefulSet-pods-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#controlPlane-statefulSet-pods-labels "Direct link to controlPlane-statefulSet-pods-labels")Labels are extra labels for this resource. #### `image` object [​](#controlPlane-statefulSet-image "Direct link to controlPlane-statefulSet-image")Image is the image for the controlPlane statefulSet container It defaults to the vCluster pro repository that includes the optional pro modules that are turned off by default. If you still want to use the pure OSS build, set the repository to 'loft-sh/vcluster-oss'. ##### `registry` string ghcr.io [​](#controlPlane-statefulSet-image-registry "Direct link to controlPlane-statefulSet-image-registry")Registry is the registry of the container image, e.g. my-registry.com or ghcr.io. This setting can be globally overridden via the controlPlane.advanced.defaultImageRegistry option. Empty means docker hub. ##### `repository` string loft-sh/vcluster-pro [​](#controlPlane-statefulSet-image-repository "Direct link to controlPlane-statefulSet-image-repository")Repository is the repository of the container image, e.g. my-repo/my-image ##### `tag` string [​](#controlPlane-statefulSet-image-tag "Direct link to controlPlane-statefulSet-image-tag")Tag is the tag of the container image, and is the default version. #### `imagePullPolicy` string [​](#controlPlane-statefulSet-imagePullPolicy "Direct link to controlPlane-statefulSet-imagePullPolicy")ImagePullPolicy is the policy how to pull the image. #### `workingDir` string [​](#controlPlane-statefulSet-workingDir "Direct link to controlPlane-statefulSet-workingDir")WorkingDir specifies in what folder the main process should get started. #### `command` string\[] \[] [​](#controlPlane-statefulSet-command "Direct link to controlPlane-statefulSet-command")Command allows you to override the main command. #### `args` string\[] \[] [​](#controlPlane-statefulSet-args "Direct link to controlPlane-statefulSet-args")Args allows you to override the main arguments. #### `env` object\[] \[] [​](#controlPlane-statefulSet-env "Direct link to controlPlane-statefulSet-env")Env are additional environment variables for the statefulSet container. #### `dnsPolicy` string [​](#controlPlane-statefulSet-dnsPolicy "Direct link to controlPlane-statefulSet-dnsPolicy")Set DNS policy for the pod. #### `dnsConfig` object [​](#controlPlane-statefulSet-dnsConfig "Direct link to controlPlane-statefulSet-dnsConfig")Specifies the DNS parameters of a pod. ##### `nameservers` string\[] [​](#controlPlane-statefulSet-dnsConfig-nameservers "Direct link to controlPlane-statefulSet-dnsConfig-nameservers")A list of DNS name server IP addresses. This will be appended to the base nameservers generated from DNSPolicy. Duplicated nameservers will be removed. ##### `searches` string\[] [​](#controlPlane-statefulSet-dnsConfig-searches "Direct link to controlPlane-statefulSet-dnsConfig-searches")A list of DNS search domains for host-name lookup. This will be appended to the base search paths generated from DNSPolicy. Duplicated search paths will be removed. ##### `options` object\[] [​](#controlPlane-statefulSet-dnsConfig-options "Direct link to controlPlane-statefulSet-dnsConfig-options")A list of DNS resolver options. This will be merged with the base options generated from DNSPolicy. Duplicated entries will be removed. Resolution options given in Options will override those that appear in the base DNSPolicy. ##### `name` string [​](#controlPlane-statefulSet-dnsConfig-options-name "Direct link to controlPlane-statefulSet-dnsConfig-options-name")Required. ##### `value` string [​](#controlPlane-statefulSet-dnsConfig-options-value "Direct link to controlPlane-statefulSet-dnsConfig-options-value") #### `initContainers` object\[] \[] [​](#controlPlane-statefulSet-initContainers "Direct link to controlPlane-statefulSet-initContainers")InitContainers are additional init containers for the statefulSet. #### `sidecarContainers` object\[] \[] [​](#controlPlane-statefulSet-sidecarContainers "Direct link to controlPlane-statefulSet-sidecarContainers")SidecarContainers are additional sidecar containers for the statefulSet. #### `hostAliases` object\[] [​](#controlPlane-statefulSet-hostAliases "Direct link to controlPlane-statefulSet-hostAliases")HostAliases allows you to add custom entries to the /etc/hosts file of each Pod created. ##### `ip` string [​](#controlPlane-statefulSet-hostAliases-ip "Direct link to controlPlane-statefulSet-hostAliases-ip") ##### `hostnames` string\[] [​](#controlPlane-statefulSet-hostAliases-hostnames "Direct link to controlPlane-statefulSet-hostAliases-hostnames") #### `runtimeClassName` string [​](#controlPlane-statefulSet-runtimeClassName "Direct link to controlPlane-statefulSet-runtimeClassName")RuntimeClassName is the runtime class to set for the statefulSet pods. ### `serviceMonitor` object [​](#controlPlane-serviceMonitor "Direct link to controlPlane-serviceMonitor")ServiceMonitor can be used to automatically create a service monitor for vCluster deployment itself. #### `enabled` boolean false [​](#controlPlane-serviceMonitor-enabled "Direct link to controlPlane-serviceMonitor-enabled")Enabled configures if Helm should create the service monitor. #### `labels` object {} [​](#controlPlane-serviceMonitor-labels "Direct link to controlPlane-serviceMonitor-labels")Labels are the extra labels to add to the service monitor. #### `annotations` object {} [​](#controlPlane-serviceMonitor-annotations "Direct link to controlPlane-serviceMonitor-annotations")Annotations are the extra annotations to add to the service monitor. #### `interval` string 30s [​](#controlPlane-serviceMonitor-interval "Direct link to controlPlane-serviceMonitor-interval")Interval is the scrape interval used for every endpoint in the service monitor, e.g. "60s". Defaults to "30s". ### `advanced` object [​](#controlPlane-advanced "Direct link to controlPlane-advanced")Advanced holds additional configuration for the vCluster control plane. #### `defaultImageRegistry` string [​](#controlPlane-advanced-defaultImageRegistry "Direct link to controlPlane-advanced-defaultImageRegistry")DefaultImageRegistry will be used as a prefix for all internal images deployed by vCluster or Helm. This makes it easy to upload all required vCluster images to a single private repository and set this value. Workload images are not affected by this. #### `virtualScheduler` object [​](#controlPlane-advanced-virtualScheduler "Direct link to controlPlane-advanced-virtualScheduler")VirtualScheduler defines if a scheduler should be used within the virtual cluster or the scheduling decision for workloads will be made by the host cluster. Deprecated: Use ControlPlane.Distro.K8S.Scheduler instead. ##### `enabled` boolean false [​](#controlPlane-advanced-virtualScheduler-enabled "Direct link to controlPlane-advanced-virtualScheduler-enabled")Enabled defines if this option should be enabled. #### `serviceAccount` object [​](#controlPlane-advanced-serviceAccount "Direct link to controlPlane-advanced-serviceAccount")ServiceAccount specifies options for the vCluster control plane service account. ##### `enabled` boolean true [​](#controlPlane-advanced-serviceAccount-enabled "Direct link to controlPlane-advanced-serviceAccount-enabled")Enabled specifies if the service account should get deployed. ##### `name` string [​](#controlPlane-advanced-serviceAccount-name "Direct link to controlPlane-advanced-serviceAccount-name")Name specifies what name to use for the service account. ##### `imagePullSecrets` object\[] [​](#controlPlane-advanced-serviceAccount-imagePullSecrets "Direct link to controlPlane-advanced-serviceAccount-imagePullSecrets")ImagePullSecrets defines extra image pull secrets for the service account. ##### `name` string [​](#controlPlane-advanced-serviceAccount-imagePullSecrets-name "Direct link to controlPlane-advanced-serviceAccount-imagePullSecrets-name")Name of the image pull secret to use. ##### `annotations` object {} [​](#controlPlane-advanced-serviceAccount-annotations "Direct link to controlPlane-advanced-serviceAccount-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#controlPlane-advanced-serviceAccount-labels "Direct link to controlPlane-advanced-serviceAccount-labels")Labels are extra labels for this resource. #### `workloadServiceAccount` object [​](#controlPlane-advanced-workloadServiceAccount "Direct link to controlPlane-advanced-workloadServiceAccount")WorkloadServiceAccount specifies options for the service account that will be used for the workloads that run within the virtual cluster. ##### `enabled` boolean true [​](#controlPlane-advanced-workloadServiceAccount-enabled "Direct link to controlPlane-advanced-workloadServiceAccount-enabled")Enabled specifies if the service account for the workloads should get deployed. ##### `name` string [​](#controlPlane-advanced-workloadServiceAccount-name "Direct link to controlPlane-advanced-workloadServiceAccount-name")Name specifies what name to use for the service account for the virtual cluster workloads. ##### `imagePullSecrets` object\[] [​](#controlPlane-advanced-workloadServiceAccount-imagePullSecrets "Direct link to controlPlane-advanced-workloadServiceAccount-imagePullSecrets")ImagePullSecrets defines extra image pull secrets for the workload service account. ##### `name` string [​](#controlPlane-advanced-workloadServiceAccount-imagePullSecrets-name "Direct link to controlPlane-advanced-workloadServiceAccount-imagePullSecrets-name")Name of the image pull secret to use. ##### `annotations` object {} [​](#controlPlane-advanced-workloadServiceAccount-annotations "Direct link to controlPlane-advanced-workloadServiceAccount-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#controlPlane-advanced-workloadServiceAccount-labels "Direct link to controlPlane-advanced-workloadServiceAccount-labels")Labels are extra labels for this resource. #### `headlessService` object [​](#controlPlane-advanced-headlessService "Direct link to controlPlane-advanced-headlessService")HeadlessService specifies options for the headless service used for the vCluster StatefulSet. ##### `annotations` object {} [​](#controlPlane-advanced-headlessService-annotations "Direct link to controlPlane-advanced-headlessService-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#controlPlane-advanced-headlessService-labels "Direct link to controlPlane-advanced-headlessService-labels")Labels are extra labels for this resource. #### `konnectivity` object [​](#controlPlane-advanced-konnectivity "Direct link to controlPlane-advanced-konnectivity")Konnectivity holds dedicated konnectivity configuration. This is only available when privateNodes.enabled is true. ##### `server` object [​](#controlPlane-advanced-konnectivity-server "Direct link to controlPlane-advanced-konnectivity-server")Server holds configuration for the konnectivity server. ##### `enabled` boolean true [​](#controlPlane-advanced-konnectivity-server-enabled "Direct link to controlPlane-advanced-konnectivity-server-enabled")Enabled defines if the konnectivity server should be enabled. ##### `extraArgs` string\[] \[] [​](#controlPlane-advanced-konnectivity-server-extraArgs "Direct link to controlPlane-advanced-konnectivity-server-extraArgs")ExtraArgs are additional arguments to pass to the konnectivity server. ##### `agent` object [​](#controlPlane-advanced-konnectivity-agent "Direct link to controlPlane-advanced-konnectivity-agent")Agent holds configuration for the konnectivity agent. ##### `enabled` boolean true [​](#controlPlane-advanced-konnectivity-agent-enabled "Direct link to controlPlane-advanced-konnectivity-agent-enabled")Enabled defines if the konnectivity agent should be enabled. ##### `replicas` integer 1 [​](#controlPlane-advanced-konnectivity-agent-replicas "Direct link to controlPlane-advanced-konnectivity-agent-replicas")Replicas is the number of replicas for the konnectivity agent. ##### `image` string [​](#controlPlane-advanced-konnectivity-agent-image "Direct link to controlPlane-advanced-konnectivity-agent-image")Image is the image for the konnectivity agent. ##### `imagePullPolicy` string [​](#controlPlane-advanced-konnectivity-agent-imagePullPolicy "Direct link to controlPlane-advanced-konnectivity-agent-imagePullPolicy")ImagePullPolicy is the policy how to pull the image. ##### `nodeSelector` object {} [​](#controlPlane-advanced-konnectivity-agent-nodeSelector "Direct link to controlPlane-advanced-konnectivity-agent-nodeSelector")NodeSelector is the node selector for the konnectivity agent. ##### `priorityClassName` string [​](#controlPlane-advanced-konnectivity-agent-priorityClassName "Direct link to controlPlane-advanced-konnectivity-agent-priorityClassName")PriorityClassName is the priority class name for the konnectivity agent. ##### `tolerations` object\[] \[] [​](#controlPlane-advanced-konnectivity-agent-tolerations "Direct link to controlPlane-advanced-konnectivity-agent-tolerations")Tolerations is the tolerations for the konnectivity agent. ##### `extraEnv` object\[] \[] [​](#controlPlane-advanced-konnectivity-agent-extraEnv "Direct link to controlPlane-advanced-konnectivity-agent-extraEnv")ExtraEnv is the extra environment variables for the konnectivity agent. ##### `extraArgs` string\[] \[] [​](#controlPlane-advanced-konnectivity-agent-extraArgs "Direct link to controlPlane-advanced-konnectivity-agent-extraArgs")ExtraArgs are additional arguments to pass to the konnectivity agent. #### `registry` object [​](#controlPlane-advanced-registry "Direct link to controlPlane-advanced-registry")Registry allows enabling an embedded docker image registry in vCluster. This is useful for air-gapped environments or when you don't have a public registry available to distribute images. ##### `enabled` boolean false [​](#controlPlane-advanced-registry-enabled "Direct link to controlPlane-advanced-registry-enabled")Enabled defines if the embedded registry should be enabled. ##### `anonymousPull` boolean true [​](#controlPlane-advanced-registry-anonymousPull "Direct link to controlPlane-advanced-registry-anonymousPull")AnonymousPull allows enabling anonymous pull for the embedded registry. This allows anybody to pull images from the registry without authentication. ##### `config` object {} [​](#controlPlane-advanced-registry-config "Direct link to controlPlane-advanced-registry-config")Config is the regular docker registry config. See for more details. #### `cloudControllerManager` object [​](#controlPlane-advanced-cloudControllerManager "Direct link to controlPlane-advanced-cloudControllerManager")CloudControllerManager holds configuration for the embedded cloud controller manager. This is only available when private nodes are enabled. The cloud controller manager is responsible for setting the node's ip addresses as well as the provider id for the node and other node metadata. ##### `enabled` boolean true [​](#controlPlane-advanced-cloudControllerManager-enabled "Direct link to controlPlane-advanced-cloudControllerManager-enabled")Enabled defines if the embedded cloud controller manager should be enabled. This defaults to true, but can be disabled if you want to use an external cloud controller manager such as AWS or GCP. The cloud controller manager is responsible for setting the node's ip addresses as well as the provider id for the node and other node metadata. ##### `kubeOvn` object [​](#controlPlane-advanced-cloudControllerManager-kubeOvn "Direct link to controlPlane-advanced-cloudControllerManager-kubeOvn")KubeOvn holds configuration for the kube-ovn integration of the embedded cloud controller manager. When enabled, services of type LoadBalancer are translated into kube-ovn RouterLBRule and OvnEip resources in the host cluster. ##### `enabled` boolean [​](#controlPlane-advanced-cloudControllerManager-kubeOvn-enabled "Direct link to controlPlane-advanced-cloudControllerManager-kubeOvn-enabled")Enabled defines if the kube-ovn integration of the embedded cloud controller manager should be enabled. This requires kube-ovn to be deployed in the host cluster and private nodes to be enabled. ##### `vpc` string [​](#controlPlane-advanced-cloudControllerManager-kubeOvn-vpc "Direct link to controlPlane-advanced-cloudControllerManager-kubeOvn-vpc")VPC is the name of the kube-ovn VPC that the integration operates on. It is shared by loadBalancers and podRoutes. ##### `loadBalancers` object [​](#controlPlane-advanced-cloudControllerManager-kubeOvn-loadBalancers "Direct link to controlPlane-advanced-cloudControllerManager-kubeOvn-loadBalancers")LoadBalancers holds configuration for translating services of type LoadBalancer into kube-ovn resources in the host cluster. ##### `externalSubnet` string [​](#controlPlane-advanced-cloudControllerManager-kubeOvn-loadBalancers-externalSubnet "Direct link to controlPlane-advanced-cloudControllerManager-kubeOvn-loadBalancers-externalSubnet")ExternalSubnet is the name of the kube-ovn subnet that the load balancer VIPs are allocated from. ##### `backendSubnet` string [​](#controlPlane-advanced-cloudControllerManager-kubeOvn-loadBalancers-backendSubnet "Direct link to controlPlane-advanced-cloudControllerManager-kubeOvn-loadBalancers-backendSubnet")BackendSubnet is the name of a kube-ovn subnet in the load balancer's VPC. It is stamped as the generated Service's ovn.kubernetes.io/logical\_switch so kube-ovn places the VIP on the VPC's load balancer; the value only needs to be a real subnet in that VPC. ##### `podRoutes` object [​](#controlPlane-advanced-cloudControllerManager-kubeOvn-podRoutes "Direct link to controlPlane-advanced-cloudControllerManager-kubeOvn-podRoutes")PodRoutes creates routes for the nodes' podCIDRs in the VPC, which enables direct routing for LoadBalancers. ##### `enabled` boolean [​](#controlPlane-advanced-cloudControllerManager-kubeOvn-podRoutes-enabled "Direct link to controlPlane-advanced-cloudControllerManager-kubeOvn-podRoutes-enabled")Enabled selects direct-to-pod mode, preserving the client source IP. #### `globalMetadata` object [​](#controlPlane-advanced-globalMetadata "Direct link to controlPlane-advanced-globalMetadata")GlobalMetadata is metadata that will be added to all resources deployed by Helm. ##### `annotations` object {} [​](#controlPlane-advanced-globalMetadata-annotations "Direct link to controlPlane-advanced-globalMetadata-annotations")Annotations are extra annotations for this resource. #### `kubeVip` object [​](#controlPlane-advanced-kubeVip "Direct link to controlPlane-advanced-kubeVip")KubeVip holds configuration for embedded kube-vip that announces the virtual cluster endpoint IP on layer 2. ##### `enabled` boolean false [​](#controlPlane-advanced-kubeVip-enabled "Direct link to controlPlane-advanced-kubeVip-enabled")Enabled defines if embedded kube-vip should be enabled. ##### `interface` string [​](#controlPlane-advanced-kubeVip-interface "Direct link to controlPlane-advanced-kubeVip-interface")Interface is the network interface on which the VIP is announced. ##### `gateway` string [​](#controlPlane-advanced-kubeVip-gateway "Direct link to controlPlane-advanced-kubeVip-gateway")Gateway is the gateway address in CIDR notation (e.g., 10.100.0.1/24). This is used to configure policy-based routing for the VIP and must include the subnet prefix. #### `podDisruptionBudget` object [​](#controlPlane-advanced-podDisruptionBudget "Direct link to controlPlane-advanced-podDisruptionBudget")PodDisruptionBudget limits how many pods of an application can be voluntarily disrupted at once to ensure availability during maintenance or scaling operations. ##### `enabled` boolean false [​](#controlPlane-advanced-podDisruptionBudget-enabled "Direct link to controlPlane-advanced-podDisruptionBudget-enabled")Enabled defines if the pod disruption budget should be enabled. ##### `minAvailable` object [​](#controlPlane-advanced-podDisruptionBudget-minAvailable "Direct link to controlPlane-advanced-podDisruptionBudget-minAvailable")MinAvailable describes the minimal number or percentage of available pods. ##### `maxUnavailable` object [​](#controlPlane-advanced-podDisruptionBudget-maxUnavailable "Direct link to controlPlane-advanced-podDisruptionBudget-maxUnavailable")MaxUnavailable describes the minimal number or percentage of unavailable pods. ##### `unhealthyPodEvictionPolicy` string [​](#controlPlane-advanced-podDisruptionBudget-unhealthyPodEvictionPolicy "Direct link to controlPlane-advanced-podDisruptionBudget-unhealthyPodEvictionPolicy")UnhealthyPodEvictionPolicy defines the criteria when unhealthy pods should be considered for eviction. Currently supported values are:- IfHealthyBudget - pods that are in the Running phase but not yet healthy are considered disrupted and may be evicted even if the PodDisruptionBudget criteria are not met. - AlwaysAllow - pods that are in the Running phase but not yet healthy are considered disrupted and can be evicted regardless of whether the criteria in a PDB is met. --- # Backing Store Each tenant cluster requires a backing store and vCluster provides two different types of backing store options: * [etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd.md) * [database](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database.md) There are different ways to deploy the type of backing store. By default, vCluster uses an embedded SQLite database as the backing store. This option is great for smaller sandbox tenant clusters, but for production, it's recommended to use a different backing store option. Backing store migration has limited support vCluster doesn't support changing the backing store type after initial deployment by editing configuration in place. A tenant cluster that already runs on deployed etcd also has no [snapshot or restore support](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md#back-up-and-restore) for backing itself up or recovering in place, so choose a backing store type before you deploy to production. Two exceptions exist: * Migration from [deployed etcd to embedded etcd](https://vcluster.com/docs/vcluster/manage/migrate-etcd-backing-store.md). Both modes use etcd as the backing store technology. * Restoring an embedded etcd snapshot onto a tenant cluster configured with an embedded database, external database, or deployed etcd backing store. vCluster converts the snapshot automatically before restoring it. See [Migrate from embedded etcd to a different backing store](https://vcluster.com/docs/vcluster/manage/backup-restore/restore.md#migrate-from-embedded-etcd-to-a-different-backing-store). Migrating in other directions, such as from a database backing store to [etcd](https://etcd.io/), isn't documented or tested. vCluster Standalone vCluster Standalone has limited backing store options. For single control plane nodes setup, * Embedded database (sqlite) * Embedded etcd are supported. For multiple control plane nodes (HA) setup, * Embedded etcd is supported. ## Config reference[​](#config-reference "Direct link to Config reference") ## `backingStore` object [​](#backingStore "Direct link to backingStore")BackingStore defines which backing store to use for virtual cluster. If not defined will use embedded database as a default backing store. ### `etcd` object [​](#backingStore-etcd "Direct link to backingStore-etcd")Etcd defines that etcd should be used as the backend for the virtual cluster #### `embedded` object [​](#backingStore-etcd-embedded "Direct link to backingStore-etcd-embedded")Embedded defines to use embedded etcd as a storage backend for the virtual cluster ##### `enabled` boolean false [​](#backingStore-etcd-embedded-enabled "Direct link to backingStore-etcd-embedded-enabled")Enabled defines if the embedded etcd should be used. ##### `migrateFromDeployedEtcd` boolean false [​](#backingStore-etcd-embedded-migrateFromDeployedEtcd "Direct link to backingStore-etcd-embedded-migrateFromDeployedEtcd")MigrateFromDeployedEtcd signals that vCluster should migrate from the deployed external etcd to embedded etcd. ##### `snapshotCount` integer [​](#backingStore-etcd-embedded-snapshotCount "Direct link to backingStore-etcd-embedded-snapshotCount")SnapshotCount defines the number of snapshots to keep for the embedded etcd. Defaults to 10000 if less than 1. ##### `extraArgs` string\[] \[] [​](#backingStore-etcd-embedded-extraArgs "Direct link to backingStore-etcd-embedded-extraArgs")ExtraArgs are additional arguments to pass to the embedded etcd. #### `deploy` object [​](#backingStore-etcd-deploy "Direct link to backingStore-etcd-deploy")Deploy defines to use an external etcd that is deployed by the helm chart ##### `enabled` boolean false [​](#backingStore-etcd-deploy-enabled "Direct link to backingStore-etcd-deploy-enabled")Enabled defines that an external etcd should be deployed. ##### `statefulSet` object [​](#backingStore-etcd-deploy-statefulSet "Direct link to backingStore-etcd-deploy-statefulSet")StatefulSet holds options for the external etcd statefulSet. ##### `enabled` boolean true [​](#backingStore-etcd-deploy-statefulSet-enabled "Direct link to backingStore-etcd-deploy-statefulSet-enabled")Enabled defines if the statefulSet should be deployed ##### `enableServiceLinks` boolean true [​](#backingStore-etcd-deploy-statefulSet-enableServiceLinks "Direct link to backingStore-etcd-deploy-statefulSet-enableServiceLinks")EnableServiceLinks for the StatefulSet pod ##### `image` object [​](#backingStore-etcd-deploy-statefulSet-image "Direct link to backingStore-etcd-deploy-statefulSet-image")Image is the image to use for the external etcd statefulSet ##### `registry` string registry.k8s.io [​](#backingStore-etcd-deploy-statefulSet-image-registry "Direct link to backingStore-etcd-deploy-statefulSet-image-registry")Registry is the registry of the container image, e.g. my-registry.com or ghcr.io. This setting can be globally overridden via the controlPlane.advanced.defaultImageRegistry option. Empty means docker hub. ##### `repository` string etcd [​](#backingStore-etcd-deploy-statefulSet-image-repository "Direct link to backingStore-etcd-deploy-statefulSet-image-repository")Repository is the repository of the container image, e.g. my-repo/my-image ##### `tag` string 3.6.16-0 [​](#backingStore-etcd-deploy-statefulSet-image-tag "Direct link to backingStore-etcd-deploy-statefulSet-image-tag")Tag is the tag of the container image, and is the default version. ##### `imagePullPolicy` string [​](#backingStore-etcd-deploy-statefulSet-imagePullPolicy "Direct link to backingStore-etcd-deploy-statefulSet-imagePullPolicy")ImagePullPolicy is the pull policy for the external etcd image ##### `env` object\[] \[] [​](#backingStore-etcd-deploy-statefulSet-env "Direct link to backingStore-etcd-deploy-statefulSet-env")Env are extra environment variables ##### `extraArgs` string\[] \[] [​](#backingStore-etcd-deploy-statefulSet-extraArgs "Direct link to backingStore-etcd-deploy-statefulSet-extraArgs")ExtraArgs are appended to the etcd command. ##### `resources` object [​](#backingStore-etcd-deploy-statefulSet-resources "Direct link to backingStore-etcd-deploy-statefulSet-resources")Resources the etcd can consume ##### `limits` object [​](#backingStore-etcd-deploy-statefulSet-resources-limits "Direct link to backingStore-etcd-deploy-statefulSet-resources-limits")Limits are resource limits for the container ##### `requests` object map\[cpu:20m memory:150Mi] [​](#backingStore-etcd-deploy-statefulSet-resources-requests "Direct link to backingStore-etcd-deploy-statefulSet-resources-requests")Requests are minimal resources that will be consumed by the container ##### `pods` object [​](#backingStore-etcd-deploy-statefulSet-pods "Direct link to backingStore-etcd-deploy-statefulSet-pods")Pods defines extra metadata for the etcd pods. ##### `annotations` object {} [​](#backingStore-etcd-deploy-statefulSet-pods-annotations "Direct link to backingStore-etcd-deploy-statefulSet-pods-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#backingStore-etcd-deploy-statefulSet-pods-labels "Direct link to backingStore-etcd-deploy-statefulSet-pods-labels")Labels are extra labels for this resource. ##### `highAvailability` object [​](#backingStore-etcd-deploy-statefulSet-highAvailability "Direct link to backingStore-etcd-deploy-statefulSet-highAvailability")HighAvailability are high availability options ##### `replicas` integer 1 [​](#backingStore-etcd-deploy-statefulSet-highAvailability-replicas "Direct link to backingStore-etcd-deploy-statefulSet-highAvailability-replicas")Replicas are the amount of pods to use. ##### `scheduling` object [​](#backingStore-etcd-deploy-statefulSet-scheduling "Direct link to backingStore-etcd-deploy-statefulSet-scheduling")Scheduling options for the etcd pods. ##### `nodeSelector` object {} [​](#backingStore-etcd-deploy-statefulSet-scheduling-nodeSelector "Direct link to backingStore-etcd-deploy-statefulSet-scheduling-nodeSelector")NodeSelector is the node selector to apply to the pod. ##### `affinity` object {} [​](#backingStore-etcd-deploy-statefulSet-scheduling-affinity "Direct link to backingStore-etcd-deploy-statefulSet-scheduling-affinity")Affinity is the affinity to apply to the pod. ##### `tolerations` object\[] \[] [​](#backingStore-etcd-deploy-statefulSet-scheduling-tolerations "Direct link to backingStore-etcd-deploy-statefulSet-scheduling-tolerations")Tolerations are the tolerations to apply to the pod. ##### `priorityClassName` string [​](#backingStore-etcd-deploy-statefulSet-scheduling-priorityClassName "Direct link to backingStore-etcd-deploy-statefulSet-scheduling-priorityClassName")PriorityClassName is the priority class name for the the pod. ##### `podManagementPolicy` string Parallel [​](#backingStore-etcd-deploy-statefulSet-scheduling-podManagementPolicy "Direct link to backingStore-etcd-deploy-statefulSet-scheduling-podManagementPolicy")PodManagementPolicy is the statefulSet pod management policy. ##### `topologySpreadConstraints` object\[] \[] [​](#backingStore-etcd-deploy-statefulSet-scheduling-topologySpreadConstraints "Direct link to backingStore-etcd-deploy-statefulSet-scheduling-topologySpreadConstraints")TopologySpreadConstraints are the topology spread constraints for the pod. ##### `security` object [​](#backingStore-etcd-deploy-statefulSet-security "Direct link to backingStore-etcd-deploy-statefulSet-security")Security options for the etcd pods. ##### `podSecurityContext` object {} [​](#backingStore-etcd-deploy-statefulSet-security-podSecurityContext "Direct link to backingStore-etcd-deploy-statefulSet-security-podSecurityContext")PodSecurityContext specifies security context options on the pod level. ##### `containerSecurityContext` object {} [​](#backingStore-etcd-deploy-statefulSet-security-containerSecurityContext "Direct link to backingStore-etcd-deploy-statefulSet-security-containerSecurityContext")ContainerSecurityContext specifies security context options on the container level. ##### `persistence` object [​](#backingStore-etcd-deploy-statefulSet-persistence "Direct link to backingStore-etcd-deploy-statefulSet-persistence")Persistence options for the etcd pods. ##### `volumeClaim` object [​](#backingStore-etcd-deploy-statefulSet-persistence-volumeClaim "Direct link to backingStore-etcd-deploy-statefulSet-persistence-volumeClaim")VolumeClaim can be used to configure the persistent volume claim. ##### `enabled` boolean true [​](#backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-enabled "Direct link to backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-enabled")Enabled enables deploying a persistent volume claim. ##### `accessModes` string\[] \[ReadWriteOnce] [​](#backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-accessModes "Direct link to backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-accessModes")AccessModes are the persistent volume claim access modes. ##### `retentionPolicy` string Retain [​](#backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-retentionPolicy "Direct link to backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-retentionPolicy")RetentionPolicy is the persistent volume claim retention policy. ##### `size` string 5Gi [​](#backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-size "Direct link to backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-size")Size is the persistent volume claim storage size. ##### `storageClass` string [​](#backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-storageClass "Direct link to backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-storageClass")StorageClass is the persistent volume claim storage class. ##### `volumeClaimTemplates` object\[] \[] [​](#backingStore-etcd-deploy-statefulSet-persistence-volumeClaimTemplates "Direct link to backingStore-etcd-deploy-statefulSet-persistence-volumeClaimTemplates")VolumeClaimTemplates defines the volumeClaimTemplates for the statefulSet ##### `addVolumes` object\[] \[] [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumes "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumes")AddVolumes defines extra volumes for the pod ##### `addVolumeMounts` object\[] [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts")AddVolumeMounts defines extra volume mounts for the container ##### `name` string [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-name "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-name")This must match the Name of a Volume. ##### `readOnly` boolean [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-readOnly "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-readOnly")Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false. ##### `mountPath` string [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-mountPath "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-mountPath")Path within the container at which the volume should be mounted. Must not contain ':'. ##### `subPath` string [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-subPath "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-subPath")Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root). ##### `mountPropagation` string [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-mountPropagation "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-mountPropagation")mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10. ##### `subPathExpr` string [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-subPathExpr "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-subPathExpr")Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR\_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. ##### `annotations` object {} [​](#backingStore-etcd-deploy-statefulSet-annotations "Direct link to backingStore-etcd-deploy-statefulSet-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#backingStore-etcd-deploy-statefulSet-labels "Direct link to backingStore-etcd-deploy-statefulSet-labels")Labels are extra labels for this resource. ##### `service` object [​](#backingStore-etcd-deploy-service "Direct link to backingStore-etcd-deploy-service")Service holds options for the external etcd service. ##### `enabled` boolean true [​](#backingStore-etcd-deploy-service-enabled "Direct link to backingStore-etcd-deploy-service-enabled")Enabled defines if the etcd service should be deployed ##### `annotations` object {} [​](#backingStore-etcd-deploy-service-annotations "Direct link to backingStore-etcd-deploy-service-annotations")Annotations are extra annotations for the external etcd service ##### `headlessService` object [​](#backingStore-etcd-deploy-headlessService "Direct link to backingStore-etcd-deploy-headlessService")HeadlessService holds options for the external etcd headless service. ##### `annotations` object {} [​](#backingStore-etcd-deploy-headlessService-annotations "Direct link to backingStore-etcd-deploy-headlessService-annotations")Annotations are extra annotations for the external etcd headless service #### `external` object [​](#backingStore-etcd-external "Direct link to backingStore-etcd-external")External defines to use a self-hosted external etcd that is not deployed by the helm chart ##### `enabled` boolean false [​](#backingStore-etcd-external-enabled "Direct link to backingStore-etcd-external-enabled")Enabled defines if the external etcd should be used. ##### `endpoint` string [​](#backingStore-etcd-external-endpoint "Direct link to backingStore-etcd-external-endpoint")Endpoint holds the endpoint of the external etcd server, e.g. my-example-service:2379 ##### `tls` object [​](#backingStore-etcd-external-tls "Direct link to backingStore-etcd-external-tls")TLS defines the tls configuration for the external etcd server ##### `caFile` string [​](#backingStore-etcd-external-tls-caFile "Direct link to backingStore-etcd-external-tls-caFile")CaFile is the path to the ca file ##### `certFile` string [​](#backingStore-etcd-external-tls-certFile "Direct link to backingStore-etcd-external-tls-certFile")CertFile is the path to the cert file ##### `keyFile` string [​](#backingStore-etcd-external-tls-keyFile "Direct link to backingStore-etcd-external-tls-keyFile")KeyFile is the path to the key file ### `database` object [​](#backingStore-database "Direct link to backingStore-database")Database defines that a database backend should be used as the backend for the virtual cluster. This uses a project called kine under the hood which is a shim for bridging Kubernetes and relational databases. #### `embedded` object [​](#backingStore-database-embedded "Direct link to backingStore-database-embedded")Embedded defines that an embedded database (sqlite) should be used as the backend for the virtual cluster ##### `enabled` boolean false [​](#backingStore-database-embedded-enabled "Direct link to backingStore-database-embedded-enabled")Enabled defines if the database should be used. ##### `dataSource` string [​](#backingStore-database-embedded-dataSource "Direct link to backingStore-database-embedded-dataSource")DataSource is the kine dataSource to use for the database. This depends on the database format. This is optional for the external database. Examples:- mysql: mysql://username:password\@tcp(hostname:3306)/vcluster - postgres: postgres\://username:password\@hostname:5432/vcluster ##### `identityProvider` string [​](#backingStore-database-embedded-identityProvider "Direct link to backingStore-database-embedded-identityProvider")IdentityProvider is the kine identity provider to use when generating temporary authentication tokens for enhanced security. This is optional for the external database. Examples:- aws: RDS IAM Authentication ##### `keyFile` string [​](#backingStore-database-embedded-keyFile "Direct link to backingStore-database-embedded-keyFile")KeyFile is the key file to use for the database. This is optional. ##### `certFile` string [​](#backingStore-database-embedded-certFile "Direct link to backingStore-database-embedded-certFile")CertFile is the cert file to use for the database. This is optional. ##### `caFile` string [​](#backingStore-database-embedded-caFile "Direct link to backingStore-database-embedded-caFile")CaFile is the ca file to use for the database. This is optional. ##### `extraArgs` string\[] \[] [​](#backingStore-database-embedded-extraArgs "Direct link to backingStore-database-embedded-extraArgs")ExtraArgs are additional arguments to pass to Kine. #### `external` object [​](#backingStore-database-external "Direct link to backingStore-database-external")External defines that an external database should be used as the backend for the virtual cluster ##### `enabled` boolean false [​](#backingStore-database-external-enabled "Direct link to backingStore-database-external-enabled")Enabled defines if the database should be used. ##### `dataSource` string [​](#backingStore-database-external-dataSource "Direct link to backingStore-database-external-dataSource")DataSource is the kine dataSource to use for the database. This depends on the database format. This is optional for the external database. Examples:- mysql: mysql://username:password\@tcp(hostname:3306)/vcluster - postgres: postgres\://username:password\@hostname:5432/vcluster ##### `identityProvider` string [​](#backingStore-database-external-identityProvider "Direct link to backingStore-database-external-identityProvider")IdentityProvider is the kine identity provider to use when generating temporary authentication tokens for enhanced security. This is optional for the external database. Examples:- aws: RDS IAM Authentication ##### `keyFile` string [​](#backingStore-database-external-keyFile "Direct link to backingStore-database-external-keyFile")KeyFile is the key file to use for the database. This is optional. ##### `certFile` string [​](#backingStore-database-external-certFile "Direct link to backingStore-database-external-certFile")CertFile is the cert file to use for the database. This is optional. ##### `caFile` string [​](#backingStore-database-external-caFile "Direct link to backingStore-database-external-caFile")CaFile is the ca file to use for the database. This is optional. ##### `extraArgs` string\[] \[] [​](#backingStore-database-external-extraArgs "Direct link to backingStore-database-external-extraArgs")ExtraArgs are additional arguments to pass to Kine. ##### `connector` string [​](#backingStore-database-external-connector "Direct link to backingStore-database-external-connector")Connector specifies a secret located in a connected vCluster Platform that contains database server connection information to be used by Platform to create a database and database user for the vCluster. and non-privileged user. A kine endpoint should be created using the database and user on Platform registration. This is optional. --- Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. # Database When using a database as the backing store, you can choose between two deployment options: * [Embedded](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/sqlite.md): The database is deployed inside the tenant cluster control plane. * [External](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md): The database runs outside the tenant cluster control plane. You can either: * Connect to an existing database. * Connect to a database server. (This option requires vCluster Platform.) vCluster Standalone vCluster Standalone can only use embedded database (sqlite) for a single control plane node or embedded etcd for HA.

## [📄️embedded](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/sqlite.md) [Configure SQLite as the tenant cluster's backing store.](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/sqlite.md) ## [📄️external](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md) [Configure an external database as the tenant cluster's backing store for enhanced performance and scalability.](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md) --- # External database | | | Enterprise | | | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [External Database](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md) | | | | | | [External Database Kerberos Authentication](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md#authenticate-to-postgresql-with-kerberos-gssapi) | | | | | | [External Database RDS IAM Authentication](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md) | | | | | | [Platform External Database](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. ## Introduction[​](#introduction "Direct link to Introduction") This guide explains how to configure an external database as the backing store for a tenant cluster. A backing store is a persistent storage solution that maintains the state and data of the tenant cluster. Using an external database can provide better performance, scalability, and data persistence compared to the default embedded storage. Configure this feature to use an external database such as MySQL or PostgreSQL for your tenant cluster's backing store. External database configuration ``` controlPlane: backingStore: database: external: enabled: true dataSource: CONNECTION_STRING connector: CONNECTOR_STRING ``` ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before configuring an external database for your tenant cluster, ensure you have the following prerequisites based on your chosen method: ### Data source (direct connection)[​](#data-source-direct-connection "Direct link to Data source (direct connection)") * A running MySQL or PostgreSQL database server * admin credentials and connection string for the database * Network connectivity from the tenant cluster to the database server ### Connector (platform-managed)[​](#connector-platform-managed "Direct link to Connector (platform-managed)") Platform Setup Required The connector method requires the vCluster Platform to be installed and properly configured. This must be completed by an administrator before using external databases. 1. [**vCluster Platform installation**](https://vcluster.com/docs/platform/install/quick-start-guide.md): The platform must be installed and accessible in your Kubernetes cluster 2. [**Database connector secret**](https://vcluster.com/docs/platform/administer/connector/database.md): A platform administrator must create a database connector secret 3. [**Platform API key**](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/platform.md): Your tenant cluster must be connected to the platform info The **connector** option provides automatic database provisioning, credential management, and cleanup when the tenant cluster is deleted. The **dataSource** option gives you direct control but requires manual database and user management. ## Connector and data source[​](#connector-and-data-source "Direct link to Connector and data source") There are two mutually exclusive options for using an external backing store. `dataSource`: assign a connection string to `dataSource` that the tenant cluster uses for its control plane backing store. `connector`: assign a name of a [connector secret](https://vcluster.com/docs/platform/administer/connector/database.md) that exists in an instance of the platform in the installed namespace. The platform uses the secret to automatically provision a separate database within the database server for the tenant cluster. It also creates a non-privileged user that can only interact with the tenant cluster's database. The tenant cluster receives a connection string built from the user and database. | | Connector | DataSource | | ---------------------------- | --------- | ---------- | | MySQL Support | Yes | Yes | | PostgreSQL Support | Yes | Yes | | Share Across tenant clusters | Yes | No | | Automatic DB Cleanup | Yes | No | | Credential stored in secret | Yes | No | Snapshot and restore `vcluster snapshot create` and `vcluster restore` work with an external database, whether configured with `dataSource` or `connector`, but they operate on the tenant cluster's Kubernetes key/value data through kine, not as a native database backup. vCluster doesn't back up or roll back the external database itself. See [Restore limitations](https://vcluster.com/docs/vcluster/manage/backup-restore/restore.md#limitations). ### Datasource configuration[​](#datasource-configuration "Direct link to Datasource configuration") Replace `CONNECTION_STRING` with the connection string for your database. Examples: * PostgreSQL: `postgres://username:password@hostname:5432/vcluster-db` * MySQL: `mysql://root:password@tcp(192.168.86.9:3306)/vcluster` For passwordless authentication to PostgreSQL with a mounted keytab, see [Authenticate to PostgreSQL with Kerberos](#authenticate-to-postgresql-with-kerberos-gssapi). ### Connector configuration[​](#connector-configuration "Direct link to Connector configuration") After completing the [prerequisites](#connector-platform-managed), reference the connector secret in your tenant cluster configuration: Connector configuration ``` controlPlane: backingStore: database: external: enabled: true connector: my-database-connector # Name of the connector secret created above ``` note The tenant cluster must be [connected to the platform](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/platform.md) to use the connector. This enables centralized management and monitoring of tenant clusters. ## Authenticate to PostgreSQL with Kerberos (GSSAPI)[​](#authenticate-to-postgresql-with-kerberos-gssapi "Direct link to Authenticate to PostgreSQL with Kerberos (GSSAPI)") Licensing Kerberos/GSSAPI authentication ships in the OSS `kine` binary, so it doesn't need a license feature of its own or a vCluster Platform connection. Like every external database, it still requires the external database feature, which is in the Prod and Scale tiers. Modify the following with your specific values to replace on the whole page and generate copyable commands: REALMEXAMPLE.COM KDC\_HOSTkdc.example.com PRINCIPALkrb5-db-user PG\_FQDNpostgres.example.com DB\_NAMEkubernetes When you use an external PostgreSQL `dataSource`, the tenant cluster's `kine` connects to your PostgreSQL backend. Instead of a static password in the connection string, that connection can authenticate with **Kerberos/GSSAPI** using a mounted keytab. This suits enterprises that already operate a Kerberos realm (an MIT KDC or Active Directory) and require Kerberos for all database access. Authentication relies on existing vcluster.yaml values. A passwordless `dataSource` carries the client principal and points at a mounted keytab. The generic `controlPlane.statefulSet` volume, mount, and environment values mount the keytab and `krb5.conf` into the control plane pod. No Kerberos-specific fields are required. note This applies to the **data source** path against **PostgreSQL** only. Kerberos is not used by the [connector](#connector-configuration) path, and GSSAPI is a PostgreSQL feature, so it does not apply to MySQL. Kubernetes distro version GSSAPI support ships in the `kine` binary carried by the **stock** Kubernetes distro image from **v1.36** onward. The vCluster release itself is unchanged: set `controlPlane.distro.k8s.image` to a `v1.36` or later tag, as shown in [Step 4](#step-4---configure-the-vclusteryaml-file). Earlier distro images carry a `kine` build without GSSAPI and fail against a Kerberos data source. ### Prerequisites[​](#prerequisites-1 "Direct link to Prerequisites") Before you begin, ensure you have: * A tenant cluster running the Kubernetes distro image at **v1.36 or later** (see the admonition above). * An existing **Kerberos environment**: * A KDC and realm (for example `EXAMPLE.COM`). * A **client principal** for the tenant cluster, for example `krb5-db-user@EXAMPLE.COM`, that you can export to a keytab. * A **PostgreSQL server built with GSSAPI support**, reachable from the control plane cluster at a stable fully qualified domain name (FQDN), with: * The service principal `postgres/@REALM` present in the server's keytab and referenced by `krb_server_keyfile`. * A `pg_hba.conf` rule that authenticates the connection with `gss`. * A login role and database that the client principal maps to (see [Step 2](#step-2---create-the-postgresql-role-and-database)). ### Step 1 - Create the database principal and export a keytab[​](#step-1---create-the-database-principal-and-export-a-keytab "Direct link to Step 1 - Create the database principal and export a keytab") Create the client principal in your KDC and export it to a keytab file. With MIT Kerberos, use `kadmin`: ``` kadmin -q "addprinc -randkey krb5-db-user@EXAMPLE.COM" kadmin -q "ktadd -k krb5-db-user.keytab krb5-db-user@EXAMPLE.COM" ``` This produces a `krb5-db-user.keytab` file containing credentials for the principal. Keep this file secure. Anyone holding it can authenticate as the principal. ### Step 2 - Create the PostgreSQL role and database[​](#step-2---create-the-postgresql-role-and-database "Direct link to Step 2 - Create the PostgreSQL role and database") Create a passwordless login role whose name matches the principal's short name and a database the role can use. Because the `pg_hba.conf` rule below uses `include_realm=0`, PostgreSQL strips the `@REALM` suffix, so the principal `krb5-db-user@EXAMPLE.COM` maps to the role `krb5-db-user`. ``` CREATE ROLE "krb5-db-user" LOGIN; CREATE DATABASE kubernetes OWNER "krb5-db-user"; ``` Point PostgreSQL at its service keytab in `postgresql.conf`: ``` krb_server_keyfile = '/etc/postgres-keytab/krb5.keytab' ``` Add a `gss` rule to `pg_hba.conf` so remote TCP connections from the tenant cluster authenticate over Kerberos. PostgreSQL uses the first matching `pg_hba.conf` rule, so place this rule before broader `host` rules that use another authentication method. Scope the address range to the tenant cluster network where possible: ``` host all all 0.0.0.0/0 gss include_realm=0 krb_realm=EXAMPLE.COM ``` info `krbsrvname` in the data source (Step 4) selects the `postgres` service principal, and `krb_realm` restricts which realm's principals the rule accepts. Keep `include_realm=0` and the role name aligned, otherwise the mapped role will not match and the connection is rejected. ### Step 3 - Create the keytab Secret and krb5.conf ConfigMap[​](#step-3---create-the-keytab-secret-and-krb5conf-configmap "Direct link to Step 3 - Create the keytab Secret and krb5.conf ConfigMap") Create a Kubernetes Secret holding the client keytab and a ConfigMap holding the client `krb5.conf`, both in the tenant cluster's host namespace (for example `vcluster-my-vcluster`). The control plane pod mounts them in [Step 4](#step-4---configure-the-vclusteryaml-file). ``` kubectl create secret generic kine-keytab \ --from-file=krb5.keytab=krb5-db-user.keytab \ --namespace vcluster-my-vcluster ``` Write a local `krb5.conf` file that points at your KDC. When your service DNS names lack matching PTR records, disable DNS canonicalization and reverse lookups so the service principal name matches the host the tenant cluster connects to: ``` [libdefaults] default_realm = EXAMPLE.COM dns_lookup_realm = false dns_lookup_kdc = false dns_canonicalize_hostname = false rdns = false [realms] EXAMPLE.COM = { kdc = kdc.example.com admin_server = kdc.example.com } ``` Create the ConfigMap from that file: ``` kubectl create configmap krb5-conf \ --from-file=krb5.conf=krb5.conf \ --namespace vcluster-my-vcluster ``` ### Step 4 - Configure the vcluster.yaml file[​](#step-4---configure-the-vclusteryaml-file "Direct link to Step 4 - Configure the vcluster.yaml file") Pin the distro image to a GSSAPI-capable build, set the passwordless Kerberos data source, and mount the keytab and `krb5.conf` into the control plane pod through `controlPlane.statefulSet`. The data source carries **no password**. The principal lives in the user part of the connection string, `krb5_keytab` points at the mounted keytab, and `krbsrvname` selects the PostgreSQL service principal. ``` controlPlane: distro: k8s: image: registry: ghcr.io repository: loft-sh/kubernetes # GSSAPI-capable kine ships in the stock distro from v1.36. tag: v1.36.1 backingStore: database: external: enabled: true # Passwordless GSSAPI data source. The principal in the user part is # URL-encoded (@ becomes %40); krb5_keytab points at the mounted keytab. dataSource: "postgres://krb5-db-user%40EXAMPLE.COM@postgres.example.com:5432/kubernetes?sslmode=disable&krbsrvname=postgres&krb5_keytab=/etc/kine-keytab/krb5.keytab" statefulSet: # KRB5_CONFIG tells the GSSAPI client where to read the realm configuration. env: - name: KRB5_CONFIG value: /etc/krb5.conf # Mount the client keytab and krb5.conf into the control plane pod. persistence: addVolumes: - name: kine-keytab secret: secretName: kine-keytab - name: krb5-conf configMap: name: krb5-conf addVolumeMounts: - name: kine-keytab mountPath: /etc/kine-keytab readOnly: true - name: krb5-conf mountPath: /etc/krb5.conf subPath: krb5.conf readOnly: true ``` warning The principal in the data source user part must be **URL-encoded**. Write the `@` between the principal and the realm as `%40` (for example `krb5-db-user%40EXAMPLE.COM`). The data source **host must exactly match** the host in the PostgreSQL service principal (`postgres/@REALM`) baked into the server keytab. A mismatch makes the GSSAPI handshake fail even when every other value is correct. note Kerberos authenticates the database connection, but it does not replace TLS. The example uses `sslmode=disable` for simplicity. For production, add the PostgreSQL TLS options your environment requires to the data source, such as `sslmode=verify-full`. ### Step 5 - Create the tenant cluster and confirm authentication[​](#step-5---create-the-tenant-cluster-and-confirm-authentication "Direct link to Step 5 - Create the tenant cluster and confirm authentication") Create the tenant cluster with your configuration: ``` vcluster create my-vcluster --namespace vcluster-my-vcluster -f vcluster.yaml ``` The tenant cluster becomes ready once `kine` completes the GSSAPI handshake and connects to PostgreSQL. The data source carries no password, so Kerberos is the only authentication path available. If the tenant cluster stays unready, inspect the control plane pod logs: ``` kubectl logs -n vcluster-my-vcluster -l app=vcluster --tail=100 ``` A working setup logs the GSSAPI provider registration for the principal, followed by a successful connection. A wrong or absent principal logs the provider registration followed by `failed GSS auth`, which proves Kerberos is enforced rather than bypassed. Confirm that the principal in the data source exactly matches the one exported into the mounted keytab, and that the data source host matches the PostgreSQL service principal. ## Network policies[​](#network-policies "Direct link to Network policies") If you are using [network policies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md), the tenant cluster control plane traffic to the external database must be allowed. vcluster.yaml ``` policies: networkPolicy: controlPlane: egress: # Allow outgoing traffic to the mysql database server. - to: - ipBlock: cidr: 192.168.86.9/32 ports: - port: 3306 protocol: TCP # Allow outgoing traffic to the postgres database server private subnets. - to: - ipBlock: cidr: 10.0.0.0/24 - ipBlock: cidr: 10.0.1.0/24 - ipBlock: cidr: 10.0.2.0/24 ports: - port: 5432 protocol: TCP ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `external` object [​](#external "Direct link to external")External defines that an external database should be used as the backend for the virtual cluster ### `enabled` boolean false [​](#external-enabled "Direct link to external-enabled")Enabled defines if the database should be used. ### `dataSource` string [​](#external-dataSource "Direct link to external-dataSource")DataSource is the kine dataSource to use for the database. This depends on the database format. This is optional for the external database. Examples:- mysql: mysql://username:password\@tcp(hostname:3306)/vcluster - postgres: postgres\://username:password\@hostname:5432/vcluster ### `identityProvider` string [​](#external-identityProvider "Direct link to external-identityProvider")IdentityProvider is the kine identity provider to use when generating temporary authentication tokens for enhanced security. This is optional for the external database. Examples:- aws: RDS IAM Authentication ### `keyFile` string [​](#external-keyFile "Direct link to external-keyFile")KeyFile is the key file to use for the database. This is optional. ### `certFile` string [​](#external-certFile "Direct link to external-certFile")CertFile is the cert file to use for the database. This is optional. ### `caFile` string [​](#external-caFile "Direct link to external-caFile")CaFile is the ca file to use for the database. This is optional. ### `extraArgs` string\[] \[] [​](#external-extraArgs "Direct link to external-extraArgs")ExtraArgs are additional arguments to pass to Kine. ### `connector` string [​](#external-connector "Direct link to external-connector")Connector specifies a secret located in a connected vCluster Platform that contains database server connection information to be used by Platform to create a database and database user for the vCluster. and non-privileged user. A kine endpoint should be created using the database and user on Platform registration. This is optional. --- # Embedded SQLite Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. By default, vCluster uses embedded [SQLite](https://www.sqlite.org/index.html) with a PersistentVolume resource to store data. You can [turn off PersistentVolume sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/persistent-volumes.md) for testing, where losing data on restarts is acceptable. ## Disable SQLite[​](#disable-sqlite "Direct link to Disable SQLite") ``` controlPlane: backingStore: database: embedded: enabled: false ``` Disable SQLite if you want to use a different database. See [External database](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md) for configuration details. ## Config reference[​](#config-reference "Direct link to Config reference") ## `embedded` object [​](#embedded "Direct link to embedded")Embedded defines that an embedded database (sqlite) should be used as the backend for the virtual cluster ### `enabled` boolean false [​](#embedded-enabled "Direct link to embedded-enabled")Enabled defines if the database should be used. ### `dataSource` string [​](#embedded-dataSource "Direct link to embedded-dataSource")DataSource is the kine dataSource to use for the database. This depends on the database format. This is optional for the external database. Examples:- mysql: mysql://username:password\@tcp(hostname:3306)/vcluster - postgres: postgres\://username:password\@hostname:5432/vcluster ### `identityProvider` string [​](#embedded-identityProvider "Direct link to embedded-identityProvider")IdentityProvider is the kine identity provider to use when generating temporary authentication tokens for enhanced security. This is optional for the external database. Examples:- aws: RDS IAM Authentication ### `keyFile` string [​](#embedded-keyFile "Direct link to embedded-keyFile")KeyFile is the key file to use for the database. This is optional. ### `certFile` string [​](#embedded-certFile "Direct link to embedded-certFile")CertFile is the cert file to use for the database. This is optional. ### `caFile` string [​](#embedded-caFile "Direct link to embedded-caFile")CaFile is the ca file to use for the database. This is optional. ### `extraArgs` string\[] \[] [​](#embedded-extraArgs "Direct link to embedded-extraArgs")ExtraArgs are additional arguments to pass to Kine. --- # etcd If you choose etcd as your backing store, there are two deployment options: * [Embedded](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/embedded.md): etcd is deployed as part of the tenant cluster control plane pod. * [Deploy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md): etcd is deployed as a separate component.

## [📄️embedded](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/embedded.md) [Configure an embedded etcd instance as the tenant cluster's backing store.](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/embedded.md) ## [📄️deploy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md) [Configure a deployed etcd instance as the tenant cluster's backing store.](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md) --- # Deployed etcd Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** When using this backing store option, [etcd](https://etcd.io/) is deployed on the control plane cluster in the same namespace as the vCluster control plane pod. vCluster deploys etcd with a StatefulSet, Service, and headless Service. ``` controlPlane: backingStore: etcd: deploy: enabled: true ``` Snapshots aren't supported with deployed etcd `vcluster snapshot create`, `vcluster restore`, and [scheduled snapshots](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/snapshots.md) aren't supported for backing up or restoring a tenant cluster that already runs on this backing store. Deployed etcd is a supported *target* when migrating an embedded etcd snapshot. See [Back up and restore](#back-up-and-restore). ## Customize the resources[​](#customize-the-resources "Direct link to Customize the resources") You can customize the resources that is deployed for etcd. Here are some basic examples, but more options exist in the configuration. ### Set resource requests for the StatefulSet[​](#set-resource-requests-for-the-statefulset "Direct link to Set resource requests for the StatefulSet") ``` controlPlane: backingStore: etcd: deploy: enabled: true statefulSet: resources: requests: cpu: 20m memory: 150Mi ``` ### Add annotations to each etcd component[​](#add-annotations-to-each-etcd-component "Direct link to Add annotations to each etcd component") ``` controlPlane: backingStore: etcd: deploy: enabled: true statefulSet: annotations: app.kubernetes.io/part-of: "etcd" cluster-autoscaler.kubernetes.io/safe-to-evict: "false" service: annotations: app.kubernetes.io/part-of: "etcd" headlessService: annotations: app.kubernetes.io/part-of: "etcd" ``` ### Set the security context of the StatefulSet[​](#set-the-security-context-of-the-statefulset "Direct link to Set the security context of the StatefulSet") ``` controlPlane: backingStore: etcd: deploy: enabled: true statefulSet: security: podSecurityContext: allowPrivilegeEscalation: false ``` ## Back up and restore[​](#back-up-and-restore "Direct link to Back up and restore") vCluster snapshot and restore aren't supported for backing up or restoring a tenant cluster that already runs on deployed etcd. `vcluster snapshot create` isn't blocked and can appear to succeed, but the result isn't validated, so don't rely on it to recover a tenant cluster. The same applies to `vcluster restore` and [scheduled snapshots](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/snapshots.md). Deployed etcd is, however, a supported target when migrating an embedded etcd snapshot onto a new tenant cluster. See [Migrate from embedded etcd to a different backing store](https://vcluster.com/docs/vcluster/manage/backup-restore/restore.md#migrate-from-embedded-etcd-to-a-different-backing-store). To move off this backing store, see [Migration options](#migration-options). ## Migration options[​](#migration-options "Direct link to Migration options") ### Migrate to embedded etcd[​](#migrate-to-embedded-etcd "Direct link to Migrate to embedded etcd") You can migrate from deployed [etcd](https://etcd.io/) to [embedded etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/embedded.md) without data loss. This migration simplifies your deployment and reduces resource consumption. Migration guide available For step-by-step instructions on migrating from deployed to embedded etcd while preserving all your data, see the **[complete migration guide](https://vcluster.com/docs/vcluster/manage/migrate-etcd-backing-store.md)**. vcluster.yaml ``` controlPlane: backingStore: etcd: embedded: enabled: true migrateFromDeployedEtcd: true ``` Deploying with `migrateFromDeployedEtcd: true` retains the deployed etcd StatefulSet to perform the migration. After a successful migration, you should see the `Successfully migrated etcd database to embedded etcd` log message in vCluster. To remove the deployed etcd after migration, delete `migrateFromDeployedEtcd: true` from the `vcluster.yaml`: vcluster.yaml after etcd migration ``` controlPlane: backingStore: etcd: embedded: enabled: true ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `deploy` object [​](#deploy "Direct link to deploy")Deploy defines to use an external etcd that is deployed by the helm chart ### `enabled` boolean false [​](#deploy-enabled "Direct link to deploy-enabled")Enabled defines that an external etcd should be deployed. ### `statefulSet` object [​](#deploy-statefulSet "Direct link to deploy-statefulSet")StatefulSet holds options for the external etcd statefulSet. #### `enabled` boolean true [​](#deploy-statefulSet-enabled "Direct link to deploy-statefulSet-enabled")Enabled defines if the statefulSet should be deployed #### `enableServiceLinks` boolean true [​](#deploy-statefulSet-enableServiceLinks "Direct link to deploy-statefulSet-enableServiceLinks")EnableServiceLinks for the StatefulSet pod #### `image` object [​](#deploy-statefulSet-image "Direct link to deploy-statefulSet-image")Image is the image to use for the external etcd statefulSet ##### `registry` string registry.k8s.io [​](#deploy-statefulSet-image-registry "Direct link to deploy-statefulSet-image-registry")Registry is the registry of the container image, e.g. my-registry.com or ghcr.io. This setting can be globally overridden via the controlPlane.advanced.defaultImageRegistry option. Empty means docker hub. ##### `repository` string etcd [​](#deploy-statefulSet-image-repository "Direct link to deploy-statefulSet-image-repository")Repository is the repository of the container image, e.g. my-repo/my-image ##### `tag` string 3.6.16-0 [​](#deploy-statefulSet-image-tag "Direct link to deploy-statefulSet-image-tag")Tag is the tag of the container image, and is the default version. #### `imagePullPolicy` string [​](#deploy-statefulSet-imagePullPolicy "Direct link to deploy-statefulSet-imagePullPolicy")ImagePullPolicy is the pull policy for the external etcd image #### `env` object\[] \[] [​](#deploy-statefulSet-env "Direct link to deploy-statefulSet-env")Env are extra environment variables #### `extraArgs` string\[] \[] [​](#deploy-statefulSet-extraArgs "Direct link to deploy-statefulSet-extraArgs")ExtraArgs are appended to the etcd command. #### `resources` object [​](#deploy-statefulSet-resources "Direct link to deploy-statefulSet-resources")Resources the etcd can consume ##### `limits` object [​](#deploy-statefulSet-resources-limits "Direct link to deploy-statefulSet-resources-limits")Limits are resource limits for the container ##### `requests` object map\[cpu:20m memory:150Mi] [​](#deploy-statefulSet-resources-requests "Direct link to deploy-statefulSet-resources-requests")Requests are minimal resources that will be consumed by the container #### `pods` object [​](#deploy-statefulSet-pods "Direct link to deploy-statefulSet-pods")Pods defines extra metadata for the etcd pods. ##### `annotations` object {} [​](#deploy-statefulSet-pods-annotations "Direct link to deploy-statefulSet-pods-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#deploy-statefulSet-pods-labels "Direct link to deploy-statefulSet-pods-labels")Labels are extra labels for this resource. #### `highAvailability` object [​](#deploy-statefulSet-highAvailability "Direct link to deploy-statefulSet-highAvailability")HighAvailability are high availability options ##### `replicas` integer 1 [​](#deploy-statefulSet-highAvailability-replicas "Direct link to deploy-statefulSet-highAvailability-replicas")Replicas are the amount of pods to use. #### `scheduling` object [​](#deploy-statefulSet-scheduling "Direct link to deploy-statefulSet-scheduling")Scheduling options for the etcd pods. ##### `nodeSelector` object {} [​](#deploy-statefulSet-scheduling-nodeSelector "Direct link to deploy-statefulSet-scheduling-nodeSelector")NodeSelector is the node selector to apply to the pod. ##### `affinity` object {} [​](#deploy-statefulSet-scheduling-affinity "Direct link to deploy-statefulSet-scheduling-affinity")Affinity is the affinity to apply to the pod. ##### `tolerations` object\[] \[] [​](#deploy-statefulSet-scheduling-tolerations "Direct link to deploy-statefulSet-scheduling-tolerations")Tolerations are the tolerations to apply to the pod. ##### `priorityClassName` string [​](#deploy-statefulSet-scheduling-priorityClassName "Direct link to deploy-statefulSet-scheduling-priorityClassName")PriorityClassName is the priority class name for the the pod. ##### `podManagementPolicy` string Parallel [​](#deploy-statefulSet-scheduling-podManagementPolicy "Direct link to deploy-statefulSet-scheduling-podManagementPolicy")PodManagementPolicy is the statefulSet pod management policy. ##### `topologySpreadConstraints` object\[] \[] [​](#deploy-statefulSet-scheduling-topologySpreadConstraints "Direct link to deploy-statefulSet-scheduling-topologySpreadConstraints")TopologySpreadConstraints are the topology spread constraints for the pod. #### `security` object [​](#deploy-statefulSet-security "Direct link to deploy-statefulSet-security")Security options for the etcd pods. ##### `podSecurityContext` object {} [​](#deploy-statefulSet-security-podSecurityContext "Direct link to deploy-statefulSet-security-podSecurityContext")PodSecurityContext specifies security context options on the pod level. ##### `containerSecurityContext` object {} [​](#deploy-statefulSet-security-containerSecurityContext "Direct link to deploy-statefulSet-security-containerSecurityContext")ContainerSecurityContext specifies security context options on the container level. #### `persistence` object [​](#deploy-statefulSet-persistence "Direct link to deploy-statefulSet-persistence")Persistence options for the etcd pods. ##### `volumeClaim` object [​](#deploy-statefulSet-persistence-volumeClaim "Direct link to deploy-statefulSet-persistence-volumeClaim")VolumeClaim can be used to configure the persistent volume claim. ##### `enabled` boolean true [​](#deploy-statefulSet-persistence-volumeClaim-enabled "Direct link to deploy-statefulSet-persistence-volumeClaim-enabled")Enabled enables deploying a persistent volume claim. ##### `accessModes` string\[] \[ReadWriteOnce] [​](#deploy-statefulSet-persistence-volumeClaim-accessModes "Direct link to deploy-statefulSet-persistence-volumeClaim-accessModes")AccessModes are the persistent volume claim access modes. ##### `retentionPolicy` string Retain [​](#deploy-statefulSet-persistence-volumeClaim-retentionPolicy "Direct link to deploy-statefulSet-persistence-volumeClaim-retentionPolicy")RetentionPolicy is the persistent volume claim retention policy. ##### `size` string 5Gi [​](#deploy-statefulSet-persistence-volumeClaim-size "Direct link to deploy-statefulSet-persistence-volumeClaim-size")Size is the persistent volume claim storage size. ##### `storageClass` string [​](#deploy-statefulSet-persistence-volumeClaim-storageClass "Direct link to deploy-statefulSet-persistence-volumeClaim-storageClass")StorageClass is the persistent volume claim storage class. ##### `volumeClaimTemplates` object\[] \[] [​](#deploy-statefulSet-persistence-volumeClaimTemplates "Direct link to deploy-statefulSet-persistence-volumeClaimTemplates")VolumeClaimTemplates defines the volumeClaimTemplates for the statefulSet ##### `addVolumes` object\[] \[] [​](#deploy-statefulSet-persistence-addVolumes "Direct link to deploy-statefulSet-persistence-addVolumes")AddVolumes defines extra volumes for the pod ##### `addVolumeMounts` object\[] [​](#deploy-statefulSet-persistence-addVolumeMounts "Direct link to deploy-statefulSet-persistence-addVolumeMounts")AddVolumeMounts defines extra volume mounts for the container ##### `name` string [​](#deploy-statefulSet-persistence-addVolumeMounts-name "Direct link to deploy-statefulSet-persistence-addVolumeMounts-name")This must match the Name of a Volume. ##### `readOnly` boolean [​](#deploy-statefulSet-persistence-addVolumeMounts-readOnly "Direct link to deploy-statefulSet-persistence-addVolumeMounts-readOnly")Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false. ##### `mountPath` string [​](#deploy-statefulSet-persistence-addVolumeMounts-mountPath "Direct link to deploy-statefulSet-persistence-addVolumeMounts-mountPath")Path within the container at which the volume should be mounted. Must not contain ':'. ##### `subPath` string [​](#deploy-statefulSet-persistence-addVolumeMounts-subPath "Direct link to deploy-statefulSet-persistence-addVolumeMounts-subPath")Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root). ##### `mountPropagation` string [​](#deploy-statefulSet-persistence-addVolumeMounts-mountPropagation "Direct link to deploy-statefulSet-persistence-addVolumeMounts-mountPropagation")mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10. ##### `subPathExpr` string [​](#deploy-statefulSet-persistence-addVolumeMounts-subPathExpr "Direct link to deploy-statefulSet-persistence-addVolumeMounts-subPathExpr")Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR\_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. #### `annotations` object {} [​](#deploy-statefulSet-annotations "Direct link to deploy-statefulSet-annotations")Annotations are extra annotations for this resource. #### `labels` object {} [​](#deploy-statefulSet-labels "Direct link to deploy-statefulSet-labels")Labels are extra labels for this resource. ### `service` object [​](#deploy-service "Direct link to deploy-service")Service holds options for the external etcd service. #### `enabled` boolean true [​](#deploy-service-enabled "Direct link to deploy-service-enabled")Enabled defines if the etcd service should be deployed #### `annotations` object {} [​](#deploy-service-annotations "Direct link to deploy-service-annotations")Annotations are extra annotations for the external etcd service ### `headlessService` object [​](#deploy-headlessService "Direct link to deploy-headlessService")HeadlessService holds options for the external etcd headless service. #### `annotations` object {} [​](#deploy-headlessService-annotations "Direct link to deploy-headlessService-annotations")Annotations are extra annotations for the external etcd headless service --- # Embedded etcd | | | Enterprise | | | | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Embedded etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/embedded.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. Upgrade Notice An issue exists when upgrading etcd (from version 3.5.1 or later, but earlier than 3.5.20) to version 3.6. This upgrade path can lead to a failed upgrade and cause the tenant cluster to break. etcd version 3.5.20 includes a fix that migrates membership data to the v3 data store. This migration prevents the issue when upgrading to version 3.6. To avoid this issue, vCluster does not upgrade etcd to version 3.6 until vCluster version 0.29.0. Follow [vCluster documentation](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/safely-upgrade-etcd.md) on how to upgrade from v3.5 -> v3.6. Any vCluster running a version earlier than 0.24.2, must first be upgraded to a version between 0.24.2 and 0.28.x, before upgrading to version 0.29.0. For more information, see the [official etcd documentation](https://etcd.io/blog/2025/upgrade_from_3.5_to_3.6_issue/). When using this backing store option, etcd is deployed as part of the vCluster control plane pod to reduce the overall footprint. ``` controlPlane: backingStore: etcd: embedded: enabled: true ``` ## How embedded etcd works[​](#how-embedded-etcd-works "Direct link to How embedded etcd works") Embedded etcd starts the etcd binary with the Kubernetes control plane inside the vCluster pod. This enables vCluster to run in high availability (HA) scenarios without requiring a separate `StatefulSet` or `Deployment`. vCluster fully manages embedded etcd and provides these capabilities: * **Dynamic scaling**: Scales the etcd cluster up or down based on vCluster replica count. * **Migration from deployed etcd**: Migrates from [deployed etcd](https://vcluster.com/docs/vcluster/manage/migrate-etcd-backing-store.md) to embedded etcd using the `migrateFromDeployedEtcd` flag. vCluster doesn't support changing to other backing store types, including through snapshot and restore. * **Simplified deployment**: Requires no additional `StatefulSets` or `Deployments`. ## Scaling behavior[​](#scaling-behavior "Direct link to Scaling behavior") vCluster dynamically builds the etcd cluster based on the number of desired replicas. For example, when you scale vCluster from 1 to 3 replicas, vCluster automatically adds the new replicas as members to the existing single-member cluster. Similarly, vCluster removes etcd members when you scale down the cluster. When scaling down breaks quorum (such as scaling from 3 to 1 replicas), vCluster rebuilds the etcd cluster without data loss or interruption. This enables dynamic scaling up and down of vCluster. ## Disaster recovery[​](#disaster-recovery "Direct link to Disaster recovery") #### Recover a single replica[​](#recover-a-single-replica "Direct link to Recover a single replica") Wait at least 10 minutes before deleting the replica pod and PVC. This action causes vCluster to rejoin the etcd member. #### Recover the entire cluster[​](#recover-the-entire-cluster "Direct link to Recover the entire cluster") In rare cases, the entire etcd cluster requires manual recovery. This occurs when the majority of etcd member replicas become corrupted or deleted simultaneously (such as 2 of 3, 3 of 5, or 4 of 7 replicas). In this scenario, etcd fails to start. note Normal pod restarts or terminations do not require manual recovery. These events trigger automatic leader election within the etcd cluster. Recovery procedures depend on whether the first replica (the pod ending with `-0`) is among the failing replicas. note The recovery procedure for the first replica also depends on your StatefulSet's `podManagementPolicy` configuration (`Parallel` or `OrderedReady`). See the [first replica recovery section](#migrate-to-parallel) for details on migrating between policies if needed. Find your vCluster namespace If using VirtualClusterInstance (platform), the vCluster StatefulSet runs in a different namespace than the VirtualClusterInstance itself. Find the StatefulSet namespace with: ``` kubectl get virtualclusterinstance -n -o jsonpath='{.spec.clusterRef.namespace}' ``` For example, if your VirtualClusterInstance is named `my-vcluster` in the `p-default` namespace, the StatefulSet might be in `vcluster-my-vcluster-p-default`. If using Helm, the namespace is what you specified during installation (e.g., `vcluster-my-team`). Modify the following with your specific values to replace on the whole page and generate copyable commands: VCLUSTER\_NAMEmy-vcluster NAMESPACEvcluster-my-team VCLUSTER\_LABELapp=vcluster Use the following procedures when some replicas are still functioning:
* First replica is not failing * First replica is failing 1. Scale the StatefulSet to one replica: ``` kubectl scale statefulset my-vcluster --replicas=1 -n vcluster-my-team ```
Verify only one pod is running: ``` kubectl get pods -l app=vcluster -n vcluster-my-team ``` 2. Monitor the rebuild process: ``` kubectl logs -f my-vcluster-0 -n vcluster-my-team ```
Watch for log messages indicating etcd is ready and the cluster is in good condition. 3. Scale back up to your target replica count: Modify the following with your specific values to generate a copyable command: REPLICA COUNT3 ``` kubectl scale statefulset my-vcluster --replicas=3 -n vcluster-my-team ```
Verify all replicas are running: ``` kubectl get pods -l app=vcluster -n vcluster-my-team kubectl logs my-vcluster-0 -n vcluster-my-team | grep "cluster is ready" ``` warning Before attempting any recovery procedure, [create a backup](https://vcluster.com/docs/vcluster/manage/backup-restore/backup.md) of your tenant cluster using `vcluster snapshot create`. If the tenant cluster's etcd is in a bad state and the snapshot command fails, you can still back up from the control plane cluster (which has its own functioning etcd). Use your preferred backup solution (e.g., Velero, Kasten, or cloud-native backup tools) to back up the control plane cluster namespace containing the vCluster resources. Ensure the backup includes: * All Kubernetes resources in the vCluster namespace (StatefulSet, Services, etc.) * PersistentVolumeClaims and their associated volume data (contains the tenant cluster's etcd data) * Secrets and ConfigMaps When restored, the vCluster pods will restart and the tenant cluster will be recreated from the backed-up etcd data. If using namespace syncing, back up all synced namespaces on the control plane cluster as well. The recovery procedure depends on your StatefulSet `podManagementPolicy` configuration. vCluster version 0.20 and later use `Parallel` by default. Earlier versions used `OrderedReady`. info If more than one pod is down with `podManagementPolicy: OrderedReady`, you must first [migrate to `Parallel`](#migrate-to-parallel) before attempting recovery. Check your configuration: ``` kubectl get statefulset my-vcluster -n vcluster-my-team -o jsonpath='{.spec.podManagementPolicy}' ``` * Parallel (default) * OrderedReady (legacy) 1. First, identify the PVC for replica-0: ``` kubectl get pvc -l app=vcluster -n vcluster-my-team ```
The PVC name typically follows the pattern `data--0` but may vary if customized in your configuration. Note the exact name from the output above, then delete the corrupted pod and its PVC: Modify the following with your specific values to generate a copyable command: PVC PREFIXdata ``` kubectl delete pod my-vcluster-0 -n vcluster-my-team kubectl delete pvc data-my-vcluster-0 -n vcluster-my-team ```
The pod restarts with a new empty PVC. The initial attempts fail because the new member tries to join the existing etcd cluster but lacks the required data. After 1-3 pod restarts, vCluster's automatic recovery detects the empty member and properly adds it as a new learner, allowing it to sync data from healthy members and join the cluster. 2. Monitor the recovery process: ``` kubectl get pods -l app=vcluster -n vcluster-my-team -w ```
Check the logs to verify the pod rejoins successfully: ``` kubectl logs -f my-vcluster-0 -n vcluster-my-team ``` caution If more than one pod is down with `podManagementPolicy: OrderedReady`, migrate to `Parallel` first before attempting recovery. ### Migrate to Parallel[​](#migrate-to-parallel "Direct link to Migrate to Parallel") 1. Check that the StatefulSet retains PVCs on deletion: ``` kubectl get statefulset my-vcluster -n vcluster-my-team -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy}' ```
The policy should be `Retain`. This is the default but can be overridden by `controlPlane.statefulSet.persistence.volumeClaim.retentionPolicy` in your configuration. 2. Delete the StatefulSet without deleting the pods: ``` kubectl delete statefulset my-vcluster -n vcluster-my-team --cascade=orphan ``` 3. Update your tenant cluster configuration to use `Parallel` pod management policy. If using a VirtualClusterInstance, edit the instance and update the `podManagementPolicy`: ``` kubectl edit virtualclusterinstance my-vcluster -n vcluster-my-team ``` Then add or update this section in the spec: ``` spec: template: helmRelease: values: | controlPlane: statefulSet: scheduling: podManagementPolicy: Parallel ```
If using Helm, update your `values.yaml` to set the pod management policy: values.yaml ``` controlPlane: statefulSet: scheduling: podManagementPolicy: Parallel ``` Then apply the update: ``` helm upgrade my-vcluster vcluster --repo https://charts.loft.sh --namespace vcluster-my-team --reuse-values -f values.yaml ```
The StatefulSet is recreated with `Parallel` policy and pods pick up the existing PVCs. 4. Now follow the same procedure as for `Parallel` mode. First, identify the PVC for replica-0: ``` kubectl get pvc -l app=vcluster -n vcluster-my-team ```
The PVC name typically follows the pattern `data--0` but may vary if customized in your configuration. Note the exact name from the output above, then delete the corrupted pod and its PVC: Modify the following with your specific values to generate a copyable command: PVC PREFIXdata ``` kubectl delete pod my-vcluster-0 -n vcluster-my-team kubectl delete pvc data-my-vcluster-0 -n vcluster-my-team ```
The pod restarts with a new empty PVC. The initial attempts fail because the new member tries to join the existing etcd cluster but lacks the required data. After 1-3 pod restarts, vCluster's automatic recovery detects the empty member and properly adds it as a new learner, allowing it to sync data from healthy members and join the cluster. warning Never clone PVCs from other replicas. Cloning PVCs causes etcd member ID conflicts and results in data loss. ### Complete data loss recovery[​](#complete-data-loss-recovery "Direct link to Complete data loss recovery") warning This recovery method results in data loss up to the last backup point. Only proceed if you have verified that all etcd replicas are corrupted and no working replicas remain. When the majority of etcd member replicas become corrupted or deleted simultaneously, the entire cluster requires recovery from backup. Prerequisites Before starting recovery, ensure you have: * Created a snapshot using `vcluster snapshot create ` * The snapshot location URL (for example, `s3://my-bucket/backup` or `oci://registry/repo:tag`) * Access to the control plane cluster namespace where the vCluster is deployed For detailed snapshot creation instructions, see [Create snapshots](https://vcluster.com/docs/vcluster/manage/backup-restore/backup.md). 1. Verify all PVCs are corrupted or inaccessible: ``` kubectl get pvc -l app=vcluster -n vcluster-my-team ```
Modify the following with your specific values to generate a copyable command: PVC PREFIXdata ``` kubectl describe pvc data-my-vcluster-0 data-my-vcluster-1 data-my-vcluster-2 -n vcluster-my-team ``` 2. Stop all vCluster instances before beginning recovery: ``` kubectl scale statefulset my-vcluster --replicas=0 -n vcluster-my-team ```
Verify all pods have terminated: ``` kubectl get pods -l app=vcluster -n vcluster-my-team ``` 3. PVC deletion timing After scaling down, wait a few seconds to ensure pods have fully terminated before deleting PVCs. If a pod restarts immediately after PVC deletion, the PVC may get stuck in a "Terminating" state. If this happens, delete the pod again to allow the PVC deletion to complete. Delete all corrupted PVCs: Modify the following with your specific values to generate a copyable command: PVC PREFIXdata ``` kubectl delete pvc data-my-vcluster-0 data-my-vcluster-1 data-my-vcluster-2 -n vcluster-my-team ```
Verify PVCs are deleted: ``` kubectl get pvc -l app=vcluster -n vcluster-my-team ``` Expected output: `No resources found` 4. Why scale up before restore? The vCluster CLI requires an accessible vCluster instance to execute the restore command. Scaling up creates a new, empty vCluster that the CLI can connect to. The `vcluster restore` command will then scale it back down automatically, restore the etcd data from the snapshot, and restart the vCluster with restored data. Scale up to the desired number of replicas: Modify the following with your specific values to generate a copyable command: DESIRED REPLICA COUNT3 ``` kubectl scale statefulset my-vcluster --replicas=3 -n vcluster-my-team ```
Wait for pods to be running: ``` kubectl get pods -l app=vcluster -n vcluster-my-team ```
Expected output showing all replicas running: ``` NAME READY STATUS RESTARTS AGE my-vcluster-0 1/1 Running 0 45s my-vcluster-1 1/1 Running 0 43s my-vcluster-2 1/1 Running 0 41s ``` 5. Use the vCluster CLI to restore from your snapshot. The restore process will: 1. Pause the vCluster (scale down to 0) 2. Delete the current PVCs 3. Start a snapshot pod to restore etcd data 4. Resume the vCluster (scale back up) Modify the following with your specific values to generate a copyable command: SNAPSHOT LOCATIONs3://my-bucket/backup ``` vcluster restore my-vcluster s3://my-bucket/backup -n vcluster-my-team ```
Expected output: ``` 16:16:38 info Pausing vCluster my-vcluster 16:16:38 info Scale down statefulSet vcluster-my-team/my-vcluster... 16:16:39 info Deleting vCluster pvc vcluster-my-team/data-my-vcluster-0 16:16:39 info Deleting vCluster pvc vcluster-my-team/data-my-vcluster-1 16:16:39 info Deleting vCluster pvc vcluster-my-team/data-my-vcluster-2 16:16:39 info Starting snapshot pod for vCluster vcluster-my-team/my-vcluster... ... Successfully restored snapshot 16:16:42 info Resuming vCluster my-vcluster ``` Authentication for remote storage If using S3 or OCI registry, ensure you have the appropriate credentials configured: * **S3**: Use AWS CLI credentials or pass credentials in the URL * **OCI**: Use Docker login or pass credentials in the URL See [Create snapshots](https://vcluster.com/docs/vcluster/manage/backup-restore/backup.md) for authentication details. 6. Connect to the vCluster and verify your workloads are restored: ``` vcluster connect my-vcluster -n vcluster-my-team ```
Check that your resources are present: ``` kubectl get pods -A kubectl get pvc -A ```
If everything looks correct, disconnect: ``` vcluster disconnect ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `embedded` object [​](#embedded "Direct link to embedded")Embedded defines to use embedded etcd as a storage backend for the virtual cluster ### `enabled` boolean false [​](#embedded-enabled "Direct link to embedded-enabled")Enabled defines if the embedded etcd should be used. ### `migrateFromDeployedEtcd` boolean false [​](#embedded-migrateFromDeployedEtcd "Direct link to embedded-migrateFromDeployedEtcd")MigrateFromDeployedEtcd signals that vCluster should migrate from the deployed external etcd to embedded etcd. ### `snapshotCount` integer [​](#embedded-snapshotCount "Direct link to embedded-snapshotCount")SnapshotCount defines the number of snapshots to keep for the embedded etcd. Defaults to 10000 if less than 1. ### `extraArgs` string\[] \[] [​](#embedded-extraArgs "Direct link to embedded-extraArgs")ExtraArgs are additional arguments to pass to the embedded etcd. --- # CoreDNS | | | Enterprise | | | | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Built-In CoreDNS](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/coredns.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Each tenant cluster runs its own DNS service using CoreDNS. This allows pods and services inside the tenant cluster to discover each other using standard Kubernetes DNS names such as `my-service.default.svc.cluster.local`. The [syncer](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync) syncs the services necessary for DNS to function as expected. This setup means that applications inside the tenant cluster cannot resolve names for services that only exist in the control plane cluster. Similarly, pods in the control plane cluster cannot discover services that only exist inside a tenant cluster. DNS resolution works entirely within the boundaries of the tenant cluster unless you configure external mappings explicitly, using [replicateServices](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking/replicate-services.md) or [resolveDNS](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking/resolve-dns.md). note CoreDNS in the tenant cluster listens on port 1053, not the DNS port 53. This avoids privilege issues, especially when multiple tenant clusters run on the same control plane cluster. Port 53 is a privileged port and typically requires elevated permissions to bind. Blocked access to port 1053 prevents DNS resolution within the tenant cluster. ## Deployment types[​](#deployment-types "Direct link to Deployment types") CoreDNS can deploy in two methods, either as a separate pod (default) or CoreDNS components can be embedded as part of the vCluster control plane pod. Each deployment type affects how the DNS service operates, how many pods the tenant cluster requires, and whether users can access and customize the CoreDNS configuration from inside the tenant cluster. | Deployment Type | Pod count per replica | Editable from tenant cluster | Corefile customization | Notes | | -------------------- | --------------------- | ---------------------------- | ---------------------- | -------------------------------------------------------------------------- | | Separate (*Default*) | 2 | Yes | Yes | CoreDNS runs as a Deployment | | Embedded | 1 | No | No | CoreDNS runs inside the syncer container in the vCluster control plane pod | ### Deploy separate CoreDNS[​](#deploy-separate-coredns "Direct link to Deploy separate CoreDNS") By default, vCluster creates a dedicated CoreDNS pod. The tenant cluster deployment includes two pods: one pod for the [API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) and syncer, and another for CoreDNS. CoreDNS is deployed as a standard Kubernetes deployment inside the tenant cluster. Users can view and modify the CoreDNS deployment using standard Kubernetes tools. DNS configuration is accessible, and you can apply changes or troubleshoot without restarting the tenant cluster's control plane. Configuration Persistence Configuration changes applied directly to CoreDNS are not persistent. They get overwritten when the vCluster control plane pod restarts and the original `vcluster.yaml` configuration is re-applied. ### Deploy embedded CoreDNS[​](#deploy-embedded-coredns "Direct link to Deploy embedded CoreDNS") Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** The embedded CoreDNS feature allows you to run CoreDNS as part of the syncer, which saves the resources of an external CoreDNS pod. This consolidates all tenant cluster components into a single pod and optimizes resource allocation. It also enables advanced DNS features like custom hostname mapping, cross-vCluster service resolution, and communication with control plane cluster services. The embedded approach requires consideration of operational trade-offs. Troubleshooting becomes more complex since DNS and sync functions share the same container and process space. Updates and restarts affect both DNS and sync capabilities simultaneously, though all CoreDNS features remain supported. ## Scale for large node counts[​](#scale-for-large-node-counts "Direct link to Scale for large node counts") This applies to the separate deployment type (the default) only. With `controlPlane.coredns.embedded: true`, there's no `coredns` Deployment in `kube-system`: vCluster removes it entirely, so `controlPlane.coredns.deployment.replicas` and Deployment-level scaling don't apply. Embedded mode can't add independent CoreDNS Deployment replicas, but two options still add DNS capacity. Raising `controlPlane.statefulSet.resources` resizes the whole control-plane pod rather than CoreDNS specifically, and raising `controlPlane.statefulSet.highAvailability.replicas` adds more active CoreDNS instances, since the `kube-dns` Service points at the syncer pods when embedded is on. By default, CoreDNS runs a single replica. This is sufficient for most tenant clusters, but high query volume, particularly from AI and ML workloads with bursty pod scheduling, can exceed what one replica handles. For why that happens, see [Why does CoreDNS become a DNS bottleneck?](https://vcluster.com/docs/vcluster/understand/coredns-bottlenecks.md). For how to scale it, see [Scale CoreDNS for large node counts](https://vcluster.com/docs/vcluster/deploy/control-plane/coredns-scaling.md). ## Example[​](#example "Direct link to Example") The configuration you choose depends on your deployment model. The following examples show how to configure CoreDNS for each supported mode. * Separate (Default) * Embedded ### Basic CoreDNS configuration[​](#basic-coredns-configuration "Direct link to Basic CoreDNS configuration") Basic CoreDNS configuration ``` controlPlane: coredns: enabled: true deployment: replicas: 1 resources: requests: cpu: 20m memory: 32Mi limits: cpu: 100m memory: 128Mi ``` ### Private registry CoreDNS configuration[​](#private-registry-coredns-configuration "Direct link to Private registry CoreDNS configuration") CoreDNS runs as a synced workload on the control plane cluster, so its image is pulled using the vCluster workload service account. To pull the CoreDNS image from a private registry, set the pull secret on `controlPlane.advanced.workloadServiceAccount.imagePullSecrets`. The CoreDNS deployment has no pull-secret field of its own. Private registry CoreDNS configuration ``` controlPlane: coredns: deployment: image: "registry.company.com/coredns" advanced: workloadServiceAccount: imagePullSecrets: - name: company-registry-credentials ``` note `controlPlane.advanced.serviceAccount.imagePullSecrets` also reaches the CoreDNS image pull, because vCluster merges both service accounts' pull secrets onto the workload service account. Use `workloadServiceAccount` for pull secrets that apply to synced workloads. If `sync.toHost.serviceAccounts.enabled` is `true`, CoreDNS uses its own synced service account instead, and the pull secret belongs there. ### Custom CoreDNS configuration[​](#custom-coredns-configuration "Direct link to Custom CoreDNS configuration") Custom CoreDNS configuration ``` controlPlane: coredns: overwriteConfig: | .:1053 { errors health ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload } ``` ### Embedded CoreDNS configuration[​](#embedded-coredns-configuration "Direct link to Embedded CoreDNS configuration") Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Embedded CoreDNS configuration ``` controlPlane: coredns: enabled: true embedded: true ``` ## Config reference[​](#config-reference "Direct link to Config reference") | Deprecated Field | New Field | | ------------------- | ---------------------------- | | `coredns.config` | `coredns.overwriteConfig` | | `coredns.manifests` | `coredns.overwriteManifests` | ## `coredns` object [​](#coredns "Direct link to coredns")CoreDNS defines everything related to the coredns that is deployed and used within the vCluster. ### `enabled` boolean true [​](#coredns-enabled "Direct link to coredns-enabled")Enabled defines if coredns is enabled ### `embedded` boolean false [​](#coredns-embedded "Direct link to coredns-embedded")Embedded defines if vCluster will start the embedded coredns service within the control-plane and not as a separate deployment. This is a PRO feature. ### `security` object [​](#coredns-security "Direct link to coredns-security")Security defines pod or container security context. #### `podSecurityContext` object {} [​](#coredns-security-podSecurityContext "Direct link to coredns-security-podSecurityContext")PodSecurityContext specifies security context options on the pod level. #### `containerSecurityContext` object {} [​](#coredns-security-containerSecurityContext "Direct link to coredns-security-containerSecurityContext")ContainerSecurityContext specifies security context options on the container level. ### `service` object [​](#coredns-service "Direct link to coredns-service")Service holds extra options for the coredns service deployed within the virtual cluster #### `spec` object map\[type:ClusterIP] [​](#coredns-service-spec "Direct link to coredns-service-spec")Spec holds extra options for the coredns service #### `annotations` object {} [​](#coredns-service-annotations "Direct link to coredns-service-annotations")Annotations are extra annotations for this resource. #### `labels` object {} [​](#coredns-service-labels "Direct link to coredns-service-labels")Labels are extra labels for this resource. ### `deployment` object [​](#coredns-deployment "Direct link to coredns-deployment")Deployment holds extra options for the coredns deployment deployed within the virtual cluster #### `image` string [​](#coredns-deployment-image "Direct link to coredns-deployment-image")Image is the coredns image to use #### `replicas` integer 1 [​](#coredns-deployment-replicas "Direct link to coredns-deployment-replicas")Replicas is the amount of coredns pods to run. #### `nodeSelector` object {} [​](#coredns-deployment-nodeSelector "Direct link to coredns-deployment-nodeSelector")NodeSelector is the node selector to use for coredns. #### `affinity` object {} [​](#coredns-deployment-affinity "Direct link to coredns-deployment-affinity")Affinity is the affinity to apply to the pod. #### `tolerations` object\[] \[] [​](#coredns-deployment-tolerations "Direct link to coredns-deployment-tolerations")Tolerations are the tolerations to apply to the pod. #### `resources` object [​](#coredns-deployment-resources "Direct link to coredns-deployment-resources")Resources are the desired resources for coredns. ##### `limits` object map\[cpu:1000m memory:170Mi] [​](#coredns-deployment-resources-limits "Direct link to coredns-deployment-resources-limits")Limits are resource limits for the container ##### `requests` object map\[cpu:20m memory:64Mi] [​](#coredns-deployment-resources-requests "Direct link to coredns-deployment-resources-requests")Requests are minimal resources that will be consumed by the container #### `pods` object [​](#coredns-deployment-pods "Direct link to coredns-deployment-pods")Pods is additional metadata for the coredns pods. ##### `annotations` object {} [​](#coredns-deployment-pods-annotations "Direct link to coredns-deployment-pods-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#coredns-deployment-pods-labels "Direct link to coredns-deployment-pods-labels")Labels are extra labels for this resource. #### `annotations` object {} [​](#coredns-deployment-annotations "Direct link to coredns-deployment-annotations")Annotations are extra annotations for this resource. #### `labels` object {} [​](#coredns-deployment-labels "Direct link to coredns-deployment-labels")Labels are extra labels for this resource. #### `topologySpreadConstraints` object\[] \[map\[labelSelector:map\[matchLabels:map\[k8s-app:vcluster-kube-dns]] maxSkew:1 topologyKey:kubernetes.io/hostname whenUnsatisfiable:DoNotSchedule]] [​](#coredns-deployment-topologySpreadConstraints "Direct link to coredns-deployment-topologySpreadConstraints")TopologySpreadConstraints are the topology spread constraints for the CoreDNS pod. ### `overwriteConfig` string [​](#coredns-overwriteConfig "Direct link to coredns-overwriteConfig")OverwriteConfig can be used to overwrite the coredns config ### `overwriteManifests` string [​](#coredns-overwriteManifests "Direct link to coredns-overwriteManifests")OverwriteManifests can be used to overwrite the coredns manifests used to deploy coredns ### `priorityClassName` string [​](#coredns-priorityClassName "Direct link to coredns-priorityClassName")PriorityClassName specifies the priority class name for the CoreDNS pods. --- # Kubernetes distro vCluster is deployed with its own Kubernetes control plane that does not need to match with the host cluster's distribution. ### Host cluster compatibility[​](#host-cluster-compatibility "Direct link to Host cluster compatibility") vCluster supports specific Kubernetes versions on host clusters. For more information, see the [Kubernetes compatibility matrix](https://vcluster.com/docs/vcluster/manage/upgrade/supported_versions.md#kubernetes-compatibility-matrix). --- # Kubernetes distribution Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. By default, vCluster uses an embedded SQLite as the [backing store](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store.md) when you use the [K8s](https://kubernetes.io/) distribution. warning After deploying your vCluster, there are limited migration paths to change your backing store. Review the backing store migration options before deploying. ## Compatibility matrix[​](#compatibility-matrix "Direct link to Compatibility matrix") ### Control plane cluster compatibility[​](#control-plane-cluster-compatibility "Direct link to Control plane cluster compatibility") The Kubernetes distro of vCluster is completely independent of the Kubernetes distro of the host cluster. This means you can run vCluster using k8s distribution regardless of the distribution type of the host cluster. vCluster supports specific Kubernetes versions on control plane clusters. For more information, see the [Kubernetes compatibility matrix](https://vcluster.com/docs/vcluster/manage/upgrade/supported_versions.md#kubernetes-compatibility-matrix). ## Configure Kubernetes version and image options[​](#setting-kubernetes-version "Direct link to Configure Kubernetes version and image options") You can configure the container image for the Kubernetes components. vCluster deploys the [API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/), controller manager, and scheduler as a single container from the [loft-sh/kubernetes](https://github.com/loft-sh/kubernetes) repository. You can customize the container image by specifying the registry, repository, and tag in your vCluster configuration: vcluster.yaml ``` controlPlane: distro: k8s: image: registry: ghcr.io # Default: ghcr.io (can be overridden globally with controlPlane.advanced.defaultImageRegistry) repository: loft-sh/kubernetes # Default: loft-sh/kubernetes tag: v1.36.0 # Default: v1.36.0 ``` The `tag` field specifies the Kubernetes version to use. When omitted, vCluster deploys **v1.36.0** by default. note The `controlPlane.distro.k8s.version` field is deprecated. Use `controlPlane.distro.k8s.image.tag` instead to specify the Kubernetes version. ## Configure API server rate limiting[​](#api-server-rate-limiting "Direct link to Configure API server rate limiting") You can configure rate limiting for the vCluster API server to control the maximum number of requests it handles. This helps prevent the tenant cluster from being overwhelmed and ensures stable performance under high load. Use the `extraArgs` field to pass rate limiting flags to the API server: vcluster.yaml ``` controlPlane: distro: k8s: apiServer: extraArgs: - --max-requests-inflight=300 - --max-mutating-requests-inflight=100 ``` ### Rate limiting options[​](#rate-limiting-options "Direct link to Rate limiting options") The [Kubernetes API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) supports these rate limiting flags: * `--max-requests-inflight`: Maximum number of non-mutating requests (GET, LIST) that can be served at the same time. Default is 400. * `--max-mutating-requests-inflight`: Maximum number of mutating requests (POST, PUT, PATCH, DELETE) that can be served at the same time. Default is 200. Impact on control plane cluster requests The `--max-requests-inflight` and `--max-mutating-requests-inflight` flags only indirectly affect the number of requests sent to the control plane cluster. With the exception of Service creation, vCluster handles all resource creation asynchronously through the syncer, which self-throttles at a default rate of 40 QPS. API Priority and Fairness Starting with Kubernetes 1.20, the [API Priority and Fairness](https://kubernetes.io/docs/concepts/cluster-administration/flow-control/) feature changes how these limits work. When enabled, the total concurrency limit becomes the sum of both flags, and requests are distributed across configurable priority levels. ### Choose appropriate values[​](#choosing-values "Direct link to Choose appropriate values") When setting rate limits, consider: * **Workload patterns**: Higher limits for read-heavy workloads, balanced limits for mixed workloads * **Available resources**: Ensure the control plane cluster has sufficient CPU and memory to handle the configured limits * **Client behavior**: Applications with many concurrent API calls may need higher limits * **Platform limits**: Some managed Kubernetes platforms (like [EKS](https://docs.aws.amazon.com/eks/latest/best-practices/scale-control-plane.html)) may scale these values automatically tip Start with conservative values and monitor the API server's performance. Increase limits gradually based on actual usage patterns and available resources. Consider implementing [resource quotas](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/resource-quota.md) and [CPU/memory limits](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/configure-memory-limits.md) alongside rate limiting for comprehensive resource management. You can also configure [admission control policies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/admission-control.md) to further control resource usage. ## Config reference[​](#config-reference "Direct link to Config reference") ## `k8s` object [​](#k8s "Direct link to k8s")K8S holds K8s relevant configuration. ### `enabled` boolean false [​](#k8s-enabled "Direct link to k8s-enabled")Enabled specifies if the K8s distro should be enabled. Only one distro can be enabled at the same time. ### `version` string [​](#k8s-version "Direct link to k8s-version")Version is the Kubernetes version to use. ### `apiServer` object [​](#k8s-apiServer "Direct link to k8s-apiServer")APIServer holds configuration specific to starting the api server. #### `enabled` boolean true [​](#k8s-apiServer-enabled "Direct link to k8s-apiServer-enabled")Enabled signals this container should be enabled. #### `command` string\[] \[] [​](#k8s-apiServer-command "Direct link to k8s-apiServer-command")Command is the command to start the distro binary. This will override the existing command. #### `extraArgs` string\[] \[] [​](#k8s-apiServer-extraArgs "Direct link to k8s-apiServer-extraArgs")ExtraArgs are additional arguments to pass to the distro binary. ### `controllerManager` object [​](#k8s-controllerManager "Direct link to k8s-controllerManager")ControllerManager holds configuration specific to starting the controller manager. #### `enabled` boolean true [​](#k8s-controllerManager-enabled "Direct link to k8s-controllerManager-enabled")Enabled signals this container should be enabled. #### `command` string\[] \[] [​](#k8s-controllerManager-command "Direct link to k8s-controllerManager-command")Command is the command to start the distro binary. This will override the existing command. #### `extraArgs` string\[] \[] [​](#k8s-controllerManager-extraArgs "Direct link to k8s-controllerManager-extraArgs")ExtraArgs are additional arguments to pass to the distro binary. ### `scheduler` object [​](#k8s-scheduler "Direct link to k8s-scheduler")Scheduler holds configuration specific to starting the scheduler. #### `enabled` boolean false [​](#k8s-scheduler-enabled "Direct link to k8s-scheduler-enabled")Enabled signals this container should be enabled. #### `command` string\[] \[] [​](#k8s-scheduler-command "Direct link to k8s-scheduler-command")Command is the command to start the distro binary. This will override the existing command. #### `extraArgs` string\[] \[] [​](#k8s-scheduler-extraArgs "Direct link to k8s-scheduler-extraArgs")ExtraArgs are additional arguments to pass to the distro binary. ### `image` object [​](#k8s-image "Direct link to k8s-image")Image is the distro image #### `registry` string ghcr.io [​](#k8s-image-registry "Direct link to k8s-image-registry")Registry is the registry of the container image, e.g. my-registry.com or ghcr.io. This setting can be globally overridden via the controlPlane.advanced.defaultImageRegistry option. Empty means docker hub. #### `repository` string loft-sh/kubernetes [​](#k8s-image-repository "Direct link to k8s-image-repository")Repository is the repository of the container image, e.g. my-repo/my-image #### `tag` string v1.36.6 [​](#k8s-image-tag "Direct link to k8s-image-tag")Tag is the tag of the container image, and is the default version. ### `imagePullPolicy` string [​](#k8s-imagePullPolicy "Direct link to k8s-imagePullPolicy")ImagePullPolicy is the pull policy for the distro image ### `env` object\[] \[] [​](#k8s-env "Direct link to k8s-env")Env are extra environment variables to use for the main container and NOT the init container. ### `resources` object map\[limits:map\[cpu:100m memory:256Mi] requests:map\[cpu:40m memory:64Mi]] [​](#k8s-resources "Direct link to k8s-resources")Resources for the distro init container ### `securityContext` object {} [​](#k8s-securityContext "Direct link to k8s-securityContext")Security options can be used for the distro init container --- # Host path mapper Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Tenant cluster internal logging uses a separate component called the [HostPath Mapper](https://vcluster.com/docs/platform/maintenance/logging/central-hostpath-mapper.md). This component matches virtual pod and container names to their real physical names. If you don't configure host path mapping, log collectors such as Loki, ELK, and Fluentd incorrectly resolve logs for vCluster pods. ## vCluster Open Source[​](#vcluster-open-source "Direct link to vCluster Open Source") note HostPathMapper v0.2.3 and later versions are incompatible with vCluster that have toHost namespace syncing enabled (`sync.toHost.namespaces`). Version Compatibility Each release of the HostPathMapper tracks a specific vCluster version, with which it is fully-compatible. Before installation, please refer the release notes of your intended HPM version to verify compatibility with your vCluster setup. ### Enable the HostPath Mapper[​](#enable-the-hostpath-mapper "Direct link to Enable the HostPath Mapper") Set `enabled: true` in the vCluster config to allow it to use the HostPath Mapper DaemonSet. vcluster.yaml ``` controlPlane: hostPathMapper: enabled: true ``` ### Deploy the HostPath Mapper DaemonSet[​](#deploy-the-hostpath-mapper-daemonset "Direct link to Deploy the HostPath Mapper DaemonSet") After setting up the vCluster, deploy the HostPath Mapper component. Before you deploy the HostPath Mapper component, make sure the following requirements are met: * Deploy the HostPath Mapper in the same namespace as the vCluster. * Set the `.Values.VclusterReleaseName` field in the Helm chart to match the vCluster name. For example, if the vCluster name is `my-vcluster` and the namespace is `my-namespace`, run the following command: Install the HostPath Mapper daemonset ``` helm install vcluster-hpm vcluster-hpm \ --repo https://charts.loft.sh \ -n my-namespace \ --set VclusterReleaseName=my-vcluster ``` After deployment, the HostPath Mapper DaemonSet starts running on every node that hosts vCluster workloads. ## Enterprise feature[​](#enterprise-feature "Direct link to Enterprise feature") The Central HostPath Mapper enables the following in a tenant cluster: * Container-based logging used by tools such as fluentd and Logstash. * Pod-based logging used by Grafana Loki. * Velero restic backups. * KubeVirt workloads. For more information about configuring vCluster and installing the Central HostPath Mapper, see [Central HostPath Mapper](https://vcluster.com/docs/platform/maintenance/logging/central-hostpath-mapper.md) in the platform documentation. ## Config reference[​](#config-reference "Direct link to Config reference") ## `hostPathMapper` object [​](#hostPathMapper "Direct link to hostPathMapper")HostPathMapper defines if vCluster should rewrite host paths. ### `enabled` boolean [​](#hostPathMapper-enabled "Direct link to hostPathMapper-enabled")Enabled specifies if the host path mapper will be used ### `central` boolean [​](#hostPathMapper-central "Direct link to hostPathMapper-central")Central specifies if the central host path mapper will be used --- # Proxy The Proxy feature lets you configure the address and port that the tenant cluster's API server listens on. vCluster uses these fields when generating the certificate for TLS. ## Config reference[​](#config-reference "Direct link to Config reference") | Deprecated Flag | Config Field | | ---------------- | ------------------- | | `--bind-address` | `proxy.bindAddress` | | `--port` | `proxy.port` | | `--tls-san` | `proxy.extraSANs` | ## `proxy` object [​](#proxy "Direct link to proxy")Proxy defines options for the virtual cluster control plane proxy that is used to do authentication and intercept requests. ### `bindAddress` string 0.0.0.0 [​](#proxy-bindAddress "Direct link to proxy-bindAddress")BindAddress under which vCluster will expose the proxy. ### `port` integer 8443 [​](#proxy-port "Direct link to proxy-port")Port under which vCluster will expose the proxy. Changing port is currently not supported. ### `extraSANs` string\[] \[] [​](#proxy-extraSANs "Direct link to proxy-extraSANs")ExtraSANs are extra hostnames to sign the vCluster proxy certificate for. --- # Standalone Supported Configurations Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. vCluster Standalone is a different architecture model for vCluster where the control plane runs directly on your infrastructure with no dependency on an existing Control Plane Cluster. vCluster is deployed directly onto nodes like other Kubernetes distributions. vCluster Standalone can run on any type of node, whether that is a bare-metal node or VM. It provides the strictest tenant isolation as there is no shared Control Plane Cluster for the control plane or worker nodes. When using vCluster Standalone, the control plane runs on its own independent node rather than as a pod on a shared cluster. Worker nodes must be private nodes.
![Comparison of traditional vCluster (control plane as pod) versus vCluster Standalone (control plane as binary on dedicated nodes)](/docs/media/private-nodes/architecture.png) For more information see the full [Standalone Documentation](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md) ## Config reference[​](#config-reference "Direct link to Config reference") ## `standalone` object [​](#standalone "Direct link to standalone")Standalone holds configuration for standalone mode. Standalone mode is set automatically when no container is detected and also implies privateNodes.enabled. ### `enabled` boolean [​](#standalone-enabled "Direct link to standalone-enabled")Enabled defines if standalone mode should be enabled. ### `dataDir` string /var/lib/vcluster [​](#standalone-dataDir "Direct link to standalone-dataDir")DataDir defines the data directory for the standalone mode. ### `advertiseAddress` string [​](#standalone-advertiseAddress "Direct link to standalone-advertiseAddress")AdvertiseAddress pins the IP address the standalone control plane advertises to the kubernetes and konnectivity Services. Defaults to the VCLUSTER\_STANDALONE\_IP\_ADDRESS env var, then default-route interface detection. Set this on multi-homed hosts where the default route is not reachable from the pod network. ### `autoNodes` object [​](#standalone-autoNodes "Direct link to standalone-autoNodes")AutoNodes automatically deploys nodes for standalone mode. #### `provider` string [​](#standalone-autoNodes-provider "Direct link to standalone-autoNodes-provider")Provider is the node provider of the nodes in this pool. #### `quantity` integer [​](#standalone-autoNodes-quantity "Direct link to standalone-autoNodes-quantity")Quantity is the number of nodes to deploy for standalone mode. #### `nodeTypeSelector` object\[] [​](#standalone-autoNodes-nodeTypeSelector "Direct link to standalone-autoNodes-nodeTypeSelector")NodeTypeSelector filters the types of nodes that can be provisioned by this pool. All requirements must be met for a node type to be eligible. ##### `property` required string [​](#standalone-autoNodes-nodeTypeSelector-property "Direct link to standalone-autoNodes-nodeTypeSelector-property")Property is the property on the node type to select. ##### `operator` string [​](#standalone-autoNodes-nodeTypeSelector-operator "Direct link to standalone-autoNodes-nodeTypeSelector-operator")Operator is the comparison operator, such as "In", "NotIn", "Exists". If empty, defaults to "In". ##### `values` string\[] [​](#standalone-autoNodes-nodeTypeSelector-values "Direct link to standalone-autoNodes-nodeTypeSelector-values")Values is the list of values to use for comparison. This is mutually exclusive with value. ##### `value` string [​](#standalone-autoNodes-nodeTypeSelector-value "Direct link to standalone-autoNodes-nodeTypeSelector-value")Value is the value to use for comparison. This is mutually exclusive with values. ### `joinNode` object [​](#standalone-joinNode "Direct link to standalone-joinNode")JoinNode holds configuration for the standalone control plane node. #### `enabled` boolean true [​](#standalone-joinNode-enabled "Direct link to standalone-joinNode-enabled")Enabled defines if the standalone node should be joined into the cluster. If false, only the control plane binaries will be executed and no node will show up in the actual cluster. #### `preInstallCommands` string\[] [​](#standalone-joinNode-preInstallCommands "Direct link to standalone-joinNode-preInstallCommands")PreInstallCommands are commands that will be executed before containerd, kubelet etc. is installed. #### `preJoinCommands` string\[] [​](#standalone-joinNode-preJoinCommands "Direct link to standalone-joinNode-preJoinCommands")PreJoinCommands are commands that will be executed before kubeadm join is executed. #### `postJoinCommands` string\[] [​](#standalone-joinNode-postJoinCommands "Direct link to standalone-joinNode-postJoinCommands")PostJoinCommands are commands that will be executed after kubeadm join is executed. #### `containerd` object [​](#standalone-joinNode-containerd "Direct link to standalone-joinNode-containerd")Containerd holds configuration for the containerd join process. ##### `enabled` boolean true [​](#standalone-joinNode-containerd-enabled "Direct link to standalone-joinNode-containerd-enabled")Enabled defines if containerd should be installed and configured by vCluster. ##### `registry` object [​](#standalone-joinNode-containerd-registry "Direct link to standalone-joinNode-containerd-registry")Registry holds configuration for how containerd should be configured to use a registries. ##### `configPath` string [​](#standalone-joinNode-containerd-registry-configPath "Direct link to standalone-joinNode-containerd-registry-configPath")ConfigPath is the path to the containerd registry config. ##### `mirrors` {key: object} [​](#standalone-joinNode-containerd-registry-mirrors "Direct link to standalone-joinNode-containerd-registry-mirrors")Mirrors holds configuration for the containerd registry mirrors. E.g. myregistry.io:5000 or docker.io. See for more details. ##### `server` string [​](#standalone-joinNode-containerd-registry-mirrors-server "Direct link to standalone-joinNode-containerd-registry-mirrors-server")Server is the fallback server to use for the containerd registry mirror. E.g. . See for more details. ##### `caCert` string\[] [​](#standalone-joinNode-containerd-registry-mirrors-caCert "Direct link to standalone-joinNode-containerd-registry-mirrors-caCert")CACert are paths to CA certificates to use for the containerd registry mirror. ##### `skipVerify` boolean [​](#standalone-joinNode-containerd-registry-mirrors-skipVerify "Direct link to standalone-joinNode-containerd-registry-mirrors-skipVerify")SkipVerify is a boolean to skip the certificate verification for the containerd registry mirror and allows http connections. ##### `capabilities` string\[] [​](#standalone-joinNode-containerd-registry-mirrors-capabilities "Direct link to standalone-joinNode-containerd-registry-mirrors-capabilities")Capabilities is a list of capabilities to enable for the containerd registry mirror. If empty, will use pull and resolve capabilities. ##### `overridePath` boolean [​](#standalone-joinNode-containerd-registry-mirrors-overridePath "Direct link to standalone-joinNode-containerd-registry-mirrors-overridePath")OverridePath is a boolean to override the path for the containerd registry mirror. ##### `hosts` object\[] [​](#standalone-joinNode-containerd-registry-mirrors-hosts "Direct link to standalone-joinNode-containerd-registry-mirrors-hosts")Hosts holds configuration for the containerd registry mirror hosts. See for more details. ##### `server` string [​](#standalone-joinNode-containerd-registry-mirrors-hosts-server "Direct link to standalone-joinNode-containerd-registry-mirrors-hosts-server")Server is the server to use for the containerd registry mirror host. E.g. . ##### `caCert` string\[] [​](#standalone-joinNode-containerd-registry-mirrors-hosts-caCert "Direct link to standalone-joinNode-containerd-registry-mirrors-hosts-caCert")CACert are paths to CA certificates to use for the containerd registry mirror host. ##### `skipVerify` boolean [​](#standalone-joinNode-containerd-registry-mirrors-hosts-skipVerify "Direct link to standalone-joinNode-containerd-registry-mirrors-hosts-skipVerify")SkipVerify is a boolean to skip the certificate verification for the containerd registry mirror and allows http connections. ##### `capabilities` string\[] [​](#standalone-joinNode-containerd-registry-mirrors-hosts-capabilities "Direct link to standalone-joinNode-containerd-registry-mirrors-hosts-capabilities")Capabilities is a list of capabilities to enable for the containerd registry mirror. If empty, will use pull and resolve capabilities. ##### `overridePath` boolean [​](#standalone-joinNode-containerd-registry-mirrors-hosts-overridePath "Direct link to standalone-joinNode-containerd-registry-mirrors-hosts-overridePath")OverridePath is a boolean to override the path for the containerd registry mirror. ##### `auth` {key: object} [​](#standalone-joinNode-containerd-registry-auth "Direct link to standalone-joinNode-containerd-registry-auth")Auth holds configuration for the containerd registry auth. See for more details. ##### `username` string [​](#standalone-joinNode-containerd-registry-auth-username "Direct link to standalone-joinNode-containerd-registry-auth-username")Username is the username for the containerd registry. ##### `password` string [​](#standalone-joinNode-containerd-registry-auth-password "Direct link to standalone-joinNode-containerd-registry-auth-password")Password is the password for the containerd registry. ##### `identityToken` string [​](#standalone-joinNode-containerd-registry-auth-identityToken "Direct link to standalone-joinNode-containerd-registry-auth-identityToken")IdentityToken is the token for the containerd registry. ##### `auth` string [​](#standalone-joinNode-containerd-registry-auth-auth "Direct link to standalone-joinNode-containerd-registry-auth-auth")Auth is the auth config for the containerd registry. ##### `pauseImage` string [​](#standalone-joinNode-containerd-pauseImage "Direct link to standalone-joinNode-containerd-pauseImage")PauseImage is the image for the pause container. #### `caCertPath` string [​](#standalone-joinNode-caCertPath "Direct link to standalone-joinNode-caCertPath")CACertPath is the path to the SSL certificate authority used to secure communications between node and control-plane. Defaults to "/etc/kubernetes/pki/ca.crt". #### `skipPhases` string\[] [​](#standalone-joinNode-skipPhases "Direct link to standalone-joinNode-skipPhases")SkipPhases is a list of phases to skip during command execution. The list of phases can be obtained with the "kubeadm join --help" command. #### `nodeRegistration` object [​](#standalone-joinNode-nodeRegistration "Direct link to standalone-joinNode-nodeRegistration")NodeRegistration holds configuration for the node registration similar to the kubeadm node registration. ##### `criSocket` string [​](#standalone-joinNode-nodeRegistration-criSocket "Direct link to standalone-joinNode-nodeRegistration-criSocket")CRI socket is the socket for the CRI. ##### `kubeletExtraArgs` object\[] [​](#standalone-joinNode-nodeRegistration-kubeletExtraArgs "Direct link to standalone-joinNode-nodeRegistration-kubeletExtraArgs")KubeletExtraArgs passes through extra arguments to the kubelet. The arguments here are passed to the kubelet command line via the environment file kubeadm writes at runtime for the kubelet to source. This overrides the generic base-level configuration in the kubelet-config ConfigMap Flags have higher priority when parsing. These values are local and specific to the node kubeadm is executing on. An argument name in this list is the flag name as it appears on the command line except without leading dash(es). Extra arguments will override existing default arguments. Duplicate extra arguments are allowed. ##### `name` string [​](#standalone-joinNode-nodeRegistration-kubeletExtraArgs-name "Direct link to standalone-joinNode-nodeRegistration-kubeletExtraArgs-name")Name is the name of the argument. ##### `value` string [​](#standalone-joinNode-nodeRegistration-kubeletExtraArgs-value "Direct link to standalone-joinNode-nodeRegistration-kubeletExtraArgs-value")Value is the value of the argument. ##### `taints` object\[] [​](#standalone-joinNode-nodeRegistration-taints "Direct link to standalone-joinNode-nodeRegistration-taints")Taints are additional taints to set for the kubelet. ##### `key` string [​](#standalone-joinNode-nodeRegistration-taints-key "Direct link to standalone-joinNode-nodeRegistration-taints-key")Required. The taint key to be applied to a node. ##### `value` string [​](#standalone-joinNode-nodeRegistration-taints-value "Direct link to standalone-joinNode-nodeRegistration-taints-value")The taint value corresponding to the taint key. ##### `effect` string [​](#standalone-joinNode-nodeRegistration-taints-effect "Direct link to standalone-joinNode-nodeRegistration-taints-effect")Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. ##### `ignorePreflightErrors` string\[] [​](#standalone-joinNode-nodeRegistration-ignorePreflightErrors "Direct link to standalone-joinNode-nodeRegistration-ignorePreflightErrors")IgnorePreflightErrors provides a slice of pre-flight errors to be ignored when the current node is registered, e.g. 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. ##### `imagePullPolicy` string [​](#standalone-joinNode-nodeRegistration-imagePullPolicy "Direct link to standalone-joinNode-nodeRegistration-imagePullPolicy")ImagePullPolicy specifies the policy for image pulling during kubeadm "init" and "join" operations. The value of this field must be one of "Always", "IfNotPresent" or "Never". If this field is unset kubeadm will default it to "IfNotPresent", or pull the required images if not present on the host. --- # Ingress Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Configure this when you want vCluster to deploy an Ingress resource to expose the tenant cluster's API server at a hostname. The cluster must have an ingress controller running and a LoadBalancer Service. ingress-nginx is deprecated The upstream ingress-nginx project is deprecated. Existing ingress-based configurations continue to work in vCluster v0.35 and vCluster Platform v4.10, but Gateway API is recommended for new endpoint deployments. See [Gateway API](https://vcluster.com/docs/vcluster/key-features/gateway-api.md) and [Migrate from ingress-nginx to Gateway API](https://vcluster.com/docs/vcluster/manage/ingress-nginx-to-gateway-api.md). This feature is not the same as syncing Ingress resources between the control plane cluster and tenant clusters. See [sync.fromHost.ingressClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/ingress-classes.md) and [sync.toHost.ingresses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/ingresses.md). vCluster uses a default rule, which uses the `ingress.host` and `ingress.pathType` configuration. When configure `ingress.spec.rules`, vCluster ignores `ingress.host` and `ingress.pathType`. ## Config reference[​](#config-reference "Direct link to Config reference") ## `ingress` object [​](#ingress "Direct link to ingress")Ingress defines options for vCluster ingress deployed by Helm. ### `enabled` boolean false [​](#ingress-enabled "Direct link to ingress-enabled")Enabled defines if the control plane ingress should be enabled ### `host` string my-host.com [​](#ingress-host "Direct link to ingress-host")Host is the host where vCluster will be reachable ### `pathType` string ImplementationSpecific [​](#ingress-pathType "Direct link to ingress-pathType")PathType is the path type of the ingress ### `spec` object map\[tls:\[]] [​](#ingress-spec "Direct link to ingress-spec")Spec allows you to configure extra ingress options. ### `annotations` object map\[nginx.ingress.kubernetes.io/backend-protocol:HTTPS nginx.ingress.kubernetes.io/ssl-passthrough:true nginx.ingress.kubernetes.io/ssl-redirect:true] [​](#ingress-annotations "Direct link to ingress-annotations")Annotations are extra annotations for this resource. ### `labels` object {} [​](#ingress-labels "Direct link to ingress-labels")Labels are extra labels for this resource. --- # Service Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** This feature defines the Service that backs the control plane's [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) resource. vCluster uses a default Service [spec](https://kubernetes.io/docs/reference/kubernetes-api/service-resources/service-v1/#ServiceSpec) when you don't configure one in `service.spec`. This default spec uses the `service.httpsNodePort` and `service.kubeletNodePort`. If you configure`service.spec`, vCluster ignores `service.httpsNodePort` and `service.kubeletNodePort` values, make sure you configure the ports in your Service spec. note * `networking.advanced.proxyKubelets.byHostname` or `networking.advanced.proxyKubelets.byIP` [configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking/advanced.md) changes the default Service spec to expose the `kubelet` port at 10250. ## Config reference[​](#config-reference "Direct link to Config reference") ## `service` object [​](#service "Direct link to service")Service defines options for vCluster service deployed by Helm. ### `enabled` boolean true [​](#service-enabled "Direct link to service-enabled")Enabled defines if the control plane service should be enabled ### `spec` object map\[type:ClusterIP] [​](#service-spec "Direct link to service-spec")Spec allows you to configure extra service options. ### `kubeletNodePort` integer 0 [​](#service-kubeletNodePort "Direct link to service-kubeletNodePort")KubeletNodePort is the node port where the fake kubelet is exposed. Defaults to 0. ### `httpsNodePort` integer 0 [​](#service-httpsNodePort "Direct link to service-httpsNodePort")HTTPSNodePort is the node port where https is exposed. Defaults to 0. ### `annotations` object {} [​](#service-annotations "Direct link to service-annotations")Annotations are extra annotations for this resource. ### `labels` object {} [​](#service-labels "Direct link to service-labels")Labels are extra labels for this resource. --- # Service monitor Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Enable `serviceMonitor` when you are using the [Prometheus Operator](https://prometheus-operator.dev/docs/prologue/introduction/) to monitor your vCluster instance. Prometheus handles scraping & aggregating metrics (in the case of high availability), and provides a query engine. Visualization tools such as [Grafana](https://grafana.com/) can then use Prometheus as a data source to graph / monitor / alert the vCluster control plane. When enabled, vCluster exposes metrics over HTTPS at a `/metrics` endpoint. ## Config reference[​](#config-reference "Direct link to Config reference") ## `serviceMonitor` object [​](#serviceMonitor "Direct link to serviceMonitor")ServiceMonitor can be used to automatically create a service monitor for vCluster deployment itself. ### `enabled` boolean false [​](#serviceMonitor-enabled "Direct link to serviceMonitor-enabled")Enabled configures if Helm should create the service monitor. ### `labels` object {} [​](#serviceMonitor-labels "Direct link to serviceMonitor-labels")Labels are the extra labels to add to the service monitor. ### `annotations` object {} [​](#serviceMonitor-annotations "Direct link to serviceMonitor-annotations")Annotations are the extra annotations to add to the service monitor. ### `interval` string 30s [​](#serviceMonitor-interval "Direct link to serviceMonitor-interval")Interval is the scrape interval used for every endpoint in the service monitor, e.g. "60s". Defaults to "30s". --- # StatefulSet Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** The vCluster control plane runs as a StatefulSet when it keeps data of its own. That's the case with the default embedded SQLite backing store and with embedded etcd, which store state on a PersistentVolumeClaim. `controlPlane.statefulSet.persistence.volumeClaim.enabled` defaults to `auto`, which creates the claim only for those two backing stores. The control plane runs as a Deployment instead when no claim is created and embedded etcd is off. That happens with a deployed etcd, external etcd, or external database backing store, which keep the data outside the control plane. It also happens if you set `volumeClaim.enabled: false` with embedded SQLite. The data then goes into an `emptyDir` volume and is lost when the pod is deleted or rescheduled. ## Configure high availability[​](#configure-high-availability "Direct link to Configure high availability") Use the `highAvailability` settings to run multiple replicas of the vCluster control plane. One pod acts as the leader, while the others remain on standby: * If the leader pod crashes, becomes unhealthy, or restarts, another pod automatically takes over leadership. * The number of available replicas determines how quickly a new leader can be elected. * You can fine-tune the leader election process using the following settings: * `leaseDuration`: Maximum time a leader can hold leadership without renewal. * `renewDeadline`: Time a pod waits to renew its leadership before it gives up. * `retryPeriod`: Interval between retries when attempting to acquire or renew leadership. ## Schedule tenant cluster pods[​](#schedule-tenant-cluster-pods "Direct link to Schedule tenant cluster pods") You can configure how and where tenant cluster pods are scheduled within the Kubernetes cluster. Configure the scheduling field with the following options to control pod placement in the cluster: * **`nodeSelector`**
Match specific node labels to guide the scheduler toward preferred nodes. Examples include: * Targeting nodes in a specific region * Selecting nodes with a specific architecture or machine class * **`affinity`** Set rules to control how pods are scheduled in relation to other pods. * Use anti-affinity (repel the pod) to keep tenant cluster pods on separate nodes. A common technique is to make tenant cluster pods repel each other so that they are not scheduled on the same nodes. This increases resiliency if a node is scaled down or replaced. * Use affinity (attract the pod) to group related pods together and reduce network latency for critical services. * **`tolerations`**
Allow pods to run on nodes with specific taints. Use this to schedule tenant cluster pods on nodes that would otherwise reject them. A common pattern is to taint nodes for non-tenant-cluster workloads and configure tenant cluster pods to tolerate the taint. This keeps tenant clusters separate from critical workloads. See [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) for more information. * **`priorityClassName`**
Set the pod’s priority to control scheduling order and preemption. See the [Pod Priority and Preemption](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/) documentation for details. * **`podManagementPolicy`** Control how StatefulSets create and delete pods. See [Pod Management Policies](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-management-policies) for details. * **`topologySpreadConstraints`** Spread pods evenly across zones, nodes, or other topology domains. See [Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/) and the [comparison with `podAffinity` and `podAntiAffinity`](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#comparison-with-podaffinity-podantiaffinity) for more information. ## Reuse an existing PersistentVolumeClaim[​](#reuse-an-existing-persistentvolumeclaim "Direct link to Reuse an existing PersistentVolumeClaim") You can pre-provision a PersistentVolumeClaim, and then configure vCluster to use it. To do this, turn off the generated claim and set the existing claim name in `controlPlane.statefulSet.persistence.dataVolume` under `persistentVolumeClaim.claimName`. `dataVolume` only works when `volumeClaim.enabled` is `false`. Otherwise vCluster also generates its own `data` claim. ``` controlPlane: statefulSet: persistence: volumeClaim: enabled: false dataVolume: - name: data persistentVolumeClaim: claimName: my-existing-pvc ``` ## vCluster image options[​](#vcluster-image-options "Direct link to vCluster image options") There are currently 3 vCluster image builds you can use in `statefulSet.image.repository`: * `loft-sh/vcluster-pro`: The default image for the Helm chart. This image works for all use cases. * `loft-sh/vcluster-oss`: An open source build of vCluster * `loft-sh/vcluster`: Deprecated. Use `loft-sh/vcluster-oss` instead. ## Config reference[​](#config-reference "Direct link to Config reference") ## `statefulSet` object [​](#statefulSet "Direct link to statefulSet")StatefulSet defines options for vCluster statefulSet deployed by Helm. ### `highAvailability` object [​](#statefulSet-highAvailability "Direct link to statefulSet-highAvailability")HighAvailability holds options related to high availability. #### `replicas` integer 1 [​](#statefulSet-highAvailability-replicas "Direct link to statefulSet-highAvailability-replicas")Replicas is the amount of replicas to use for the statefulSet. #### `leaseDuration` integer 60 [​](#statefulSet-highAvailability-leaseDuration "Direct link to statefulSet-highAvailability-leaseDuration")LeaseDuration is the time to lease for the leader. #### `renewDeadline` integer 40 [​](#statefulSet-highAvailability-renewDeadline "Direct link to statefulSet-highAvailability-renewDeadline")RenewDeadline is the deadline to renew a lease for the leader. #### `retryPeriod` integer 15 [​](#statefulSet-highAvailability-retryPeriod "Direct link to statefulSet-highAvailability-retryPeriod")RetryPeriod is the time until a replica will retry to get a lease. ### `resources` object [​](#statefulSet-resources "Direct link to statefulSet-resources")Resources are the resource requests and limits for the statefulSet container. #### `limits` object map\[ephemeral-storage:10Gi memory:4Gi] [​](#statefulSet-resources-limits "Direct link to statefulSet-resources-limits")Limits are resource limits for the container #### `requests` object map\[cpu:200m ephemeral-storage:1Gi memory:256Mi] [​](#statefulSet-resources-requests "Direct link to statefulSet-resources-requests")Requests are minimal resources that will be consumed by the container ### `scheduling` object [​](#statefulSet-scheduling "Direct link to statefulSet-scheduling")Scheduling holds options related to scheduling. #### `nodeSelector` object {} [​](#statefulSet-scheduling-nodeSelector "Direct link to statefulSet-scheduling-nodeSelector")NodeSelector is the node selector to apply to the pod. #### `affinity` object {} [​](#statefulSet-scheduling-affinity "Direct link to statefulSet-scheduling-affinity")Affinity is the affinity to apply to the pod. #### `tolerations` object\[] \[] [​](#statefulSet-scheduling-tolerations "Direct link to statefulSet-scheduling-tolerations")Tolerations are the tolerations to apply to the pod. #### `priorityClassName` string [​](#statefulSet-scheduling-priorityClassName "Direct link to statefulSet-scheduling-priorityClassName")PriorityClassName is the priority class name for the the pod. #### `podManagementPolicy` string Parallel [​](#statefulSet-scheduling-podManagementPolicy "Direct link to statefulSet-scheduling-podManagementPolicy")PodManagementPolicy is the statefulSet pod management policy. #### `topologySpreadConstraints` object\[] \[] [​](#statefulSet-scheduling-topologySpreadConstraints "Direct link to statefulSet-scheduling-topologySpreadConstraints")TopologySpreadConstraints are the topology spread constraints for the pod. ### `security` object [​](#statefulSet-security "Direct link to statefulSet-security")Security defines pod or container security context. #### `podSecurityContext` object {} [​](#statefulSet-security-podSecurityContext "Direct link to statefulSet-security-podSecurityContext")PodSecurityContext specifies security context options on the pod level. #### `containerSecurityContext` object map\[allowPrivilegeEscalation:false runAsGroup:0 runAsUser:0] [​](#statefulSet-security-containerSecurityContext "Direct link to statefulSet-security-containerSecurityContext")ContainerSecurityContext specifies security context options on the container level. ### `probes` object [​](#statefulSet-probes "Direct link to statefulSet-probes")Probes enables or disables the main container probes. #### `livenessProbe` object [​](#statefulSet-probes-livenessProbe "Direct link to statefulSet-probes-livenessProbe")LivenessProbe specifies if the liveness probe for the container should be enabled ##### `enabled` boolean true [​](#statefulSet-probes-livenessProbe-enabled "Direct link to statefulSet-probes-livenessProbe-enabled")Enabled defines if this option should be enabled. ##### `failureThreshold` integer 60 [​](#statefulSet-probes-livenessProbe-failureThreshold "Direct link to statefulSet-probes-livenessProbe-failureThreshold")Number of consecutive failures for the probe to be considered failed ##### `initialDelaySeconds` integer 60 [​](#statefulSet-probes-livenessProbe-initialDelaySeconds "Direct link to statefulSet-probes-livenessProbe-initialDelaySeconds")Time (in seconds) to wait before starting the liveness probe ##### `timeoutSeconds` integer 3 [​](#statefulSet-probes-livenessProbe-timeoutSeconds "Direct link to statefulSet-probes-livenessProbe-timeoutSeconds")Maximum duration (in seconds) that the probe will wait for a response. ##### `periodSeconds` integer 2 [​](#statefulSet-probes-livenessProbe-periodSeconds "Direct link to statefulSet-probes-livenessProbe-periodSeconds")Frequency (in seconds) to perform the probe #### `readinessProbe` object [​](#statefulSet-probes-readinessProbe "Direct link to statefulSet-probes-readinessProbe")ReadinessProbe specifies if the readiness probe for the container should be enabled ##### `enabled` boolean true [​](#statefulSet-probes-readinessProbe-enabled "Direct link to statefulSet-probes-readinessProbe-enabled")Enabled defines if this option should be enabled. ##### `failureThreshold` integer 60 [​](#statefulSet-probes-readinessProbe-failureThreshold "Direct link to statefulSet-probes-readinessProbe-failureThreshold")Number of consecutive failures for the probe to be considered failed ##### `timeoutSeconds` integer 3 [​](#statefulSet-probes-readinessProbe-timeoutSeconds "Direct link to statefulSet-probes-readinessProbe-timeoutSeconds")Maximum duration (in seconds) that the probe will wait for a response. ##### `periodSeconds` integer 2 [​](#statefulSet-probes-readinessProbe-periodSeconds "Direct link to statefulSet-probes-readinessProbe-periodSeconds")Frequency (in seconds) to perform the probe #### `startupProbe` object [​](#statefulSet-probes-startupProbe "Direct link to statefulSet-probes-startupProbe")StartupProbe specifies if the startup probe for the container should be enabled ##### `enabled` boolean true [​](#statefulSet-probes-startupProbe-enabled "Direct link to statefulSet-probes-startupProbe-enabled")Enabled defines if this option should be enabled. ##### `failureThreshold` integer 300 [​](#statefulSet-probes-startupProbe-failureThreshold "Direct link to statefulSet-probes-startupProbe-failureThreshold")Number of consecutive failures allowed before failing the pod ##### `timeoutSeconds` integer 3 [​](#statefulSet-probes-startupProbe-timeoutSeconds "Direct link to statefulSet-probes-startupProbe-timeoutSeconds")Maximum duration (in seconds) that the probe will wait for a response. ##### `periodSeconds` integer 6 [​](#statefulSet-probes-startupProbe-periodSeconds "Direct link to statefulSet-probes-startupProbe-periodSeconds")Frequency (in seconds) to perform the probe ### `persistence` object [​](#statefulSet-persistence "Direct link to statefulSet-persistence")Persistence defines options around persistence for the statefulSet. #### `volumeClaim` object [​](#statefulSet-persistence-volumeClaim "Direct link to statefulSet-persistence-volumeClaim")VolumeClaim can be used to configure the persistent volume claim. ##### `enabled` string|boolean auto [​](#statefulSet-persistence-volumeClaim-enabled "Direct link to statefulSet-persistence-volumeClaim-enabled")Enabled enables deploying a persistent volume claim. If auto, vCluster will automatically determine based on the chosen distro and other options if this is required. ##### `accessModes` string\[] \[ReadWriteOnce] [​](#statefulSet-persistence-volumeClaim-accessModes "Direct link to statefulSet-persistence-volumeClaim-accessModes")AccessModes are the persistent volume claim access modes. ##### `retentionPolicy` string Retain [​](#statefulSet-persistence-volumeClaim-retentionPolicy "Direct link to statefulSet-persistence-volumeClaim-retentionPolicy")RetentionPolicy is the persistent volume claim retention policy. ##### `size` string 5Gi [​](#statefulSet-persistence-volumeClaim-size "Direct link to statefulSet-persistence-volumeClaim-size")Size is the persistent volume claim storage size. ##### `storageClass` string [​](#statefulSet-persistence-volumeClaim-storageClass "Direct link to statefulSet-persistence-volumeClaim-storageClass")StorageClass is the persistent volume claim storage class. #### `volumeClaimTemplates` object\[] \[] [​](#statefulSet-persistence-volumeClaimTemplates "Direct link to statefulSet-persistence-volumeClaimTemplates")VolumeClaimTemplates defines the volumeClaimTemplates for the statefulSet #### `dataVolume` object\[] \[] [​](#statefulSet-persistence-dataVolume "Direct link to statefulSet-persistence-dataVolume")Allows you to override the dataVolume. Only works correctly if volumeClaim.enabled=false. #### `binariesVolume` object\[] \[map\[emptyDir:map\[] name:binaries]] [​](#statefulSet-persistence-binariesVolume "Direct link to statefulSet-persistence-binariesVolume")BinariesVolume defines a binaries volume that is used to retrieve distro specific executables to be run by the syncer controller. This volume doesn't need to be persistent. #### `addVolumes` object\[] \[] [​](#statefulSet-persistence-addVolumes "Direct link to statefulSet-persistence-addVolumes")AddVolumes defines extra volumes for the pod #### `addVolumeMounts` object\[] [​](#statefulSet-persistence-addVolumeMounts "Direct link to statefulSet-persistence-addVolumeMounts")AddVolumeMounts defines extra volume mounts for the container ##### `name` string [​](#statefulSet-persistence-addVolumeMounts-name "Direct link to statefulSet-persistence-addVolumeMounts-name")This must match the Name of a Volume. ##### `readOnly` boolean [​](#statefulSet-persistence-addVolumeMounts-readOnly "Direct link to statefulSet-persistence-addVolumeMounts-readOnly")Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false. ##### `mountPath` string [​](#statefulSet-persistence-addVolumeMounts-mountPath "Direct link to statefulSet-persistence-addVolumeMounts-mountPath")Path within the container at which the volume should be mounted. Must not contain ':'. ##### `subPath` string [​](#statefulSet-persistence-addVolumeMounts-subPath "Direct link to statefulSet-persistence-addVolumeMounts-subPath")Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root). ##### `mountPropagation` string [​](#statefulSet-persistence-addVolumeMounts-mountPropagation "Direct link to statefulSet-persistence-addVolumeMounts-mountPropagation")mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10. ##### `subPathExpr` string [​](#statefulSet-persistence-addVolumeMounts-subPathExpr "Direct link to statefulSet-persistence-addVolumeMounts-subPathExpr")Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR\_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. ### `enableServiceLinks` boolean true [​](#statefulSet-enableServiceLinks "Direct link to statefulSet-enableServiceLinks")EnableServiceLinks for the StatefulSet pod ### `annotations` object {} [​](#statefulSet-annotations "Direct link to statefulSet-annotations")Annotations are extra annotations for this resource. ### `labels` object {} [​](#statefulSet-labels "Direct link to statefulSet-labels")Labels are extra labels for this resource. ### `pods` object [​](#statefulSet-pods "Direct link to statefulSet-pods")Additional labels or annotations for the statefulSet pods. #### `annotations` object {} [​](#statefulSet-pods-annotations "Direct link to statefulSet-pods-annotations")Annotations are extra annotations for this resource. #### `labels` object {} [​](#statefulSet-pods-labels "Direct link to statefulSet-pods-labels")Labels are extra labels for this resource. ### `image` object [​](#statefulSet-image "Direct link to statefulSet-image")Image is the image for the controlPlane statefulSet container It defaults to the vCluster pro repository that includes the optional pro modules that are turned off by default. If you still want to use the pure OSS build, set the repository to 'loft-sh/vcluster-oss'. #### `registry` string ghcr.io [​](#statefulSet-image-registry "Direct link to statefulSet-image-registry")Registry is the registry of the container image, e.g. my-registry.com or ghcr.io. This setting can be globally overridden via the controlPlane.advanced.defaultImageRegistry option. Empty means docker hub. #### `repository` string loft-sh/vcluster-pro [​](#statefulSet-image-repository "Direct link to statefulSet-image-repository")Repository is the repository of the container image, e.g. my-repo/my-image #### `tag` string [​](#statefulSet-image-tag "Direct link to statefulSet-image-tag")Tag is the tag of the container image, and is the default version. ### `imagePullPolicy` string [​](#statefulSet-imagePullPolicy "Direct link to statefulSet-imagePullPolicy")ImagePullPolicy is the policy how to pull the image. ### `workingDir` string [​](#statefulSet-workingDir "Direct link to statefulSet-workingDir")WorkingDir specifies in what folder the main process should get started. ### `command` string\[] \[] [​](#statefulSet-command "Direct link to statefulSet-command")Command allows you to override the main command. ### `args` string\[] \[] [​](#statefulSet-args "Direct link to statefulSet-args")Args allows you to override the main arguments. ### `env` object\[] \[] [​](#statefulSet-env "Direct link to statefulSet-env")Env are additional environment variables for the statefulSet container. ### `dnsPolicy` string [​](#statefulSet-dnsPolicy "Direct link to statefulSet-dnsPolicy")Set DNS policy for the pod. ### `dnsConfig` object [​](#statefulSet-dnsConfig "Direct link to statefulSet-dnsConfig")Specifies the DNS parameters of a pod. #### `nameservers` string\[] [​](#statefulSet-dnsConfig-nameservers "Direct link to statefulSet-dnsConfig-nameservers")A list of DNS name server IP addresses. This will be appended to the base nameservers generated from DNSPolicy. Duplicated nameservers will be removed. #### `searches` string\[] [​](#statefulSet-dnsConfig-searches "Direct link to statefulSet-dnsConfig-searches")A list of DNS search domains for host-name lookup. This will be appended to the base search paths generated from DNSPolicy. Duplicated search paths will be removed. #### `options` object\[] [​](#statefulSet-dnsConfig-options "Direct link to statefulSet-dnsConfig-options")A list of DNS resolver options. This will be merged with the base options generated from DNSPolicy. Duplicated entries will be removed. Resolution options given in Options will override those that appear in the base DNSPolicy. ##### `name` string [​](#statefulSet-dnsConfig-options-name "Direct link to statefulSet-dnsConfig-options-name")Required. ##### `value` string [​](#statefulSet-dnsConfig-options-value "Direct link to statefulSet-dnsConfig-options-value") ### `initContainers` object\[] \[] [​](#statefulSet-initContainers "Direct link to statefulSet-initContainers")InitContainers are additional init containers for the statefulSet. ### `sidecarContainers` object\[] \[] [​](#statefulSet-sidecarContainers "Direct link to statefulSet-sidecarContainers")SidecarContainers are additional sidecar containers for the statefulSet. ### `hostAliases` object\[] [​](#statefulSet-hostAliases "Direct link to statefulSet-hostAliases")HostAliases allows you to add custom entries to the /etc/hosts file of each Pod created. #### `ip` string [​](#statefulSet-hostAliases-ip "Direct link to statefulSet-hostAliases-ip") #### `hostnames` string\[] [​](#statefulSet-hostAliases-hostnames "Direct link to statefulSet-hostAliases-hostnames") ### `runtimeClassName` string [​](#statefulSet-runtimeClassName "Direct link to statefulSet-runtimeClassName")RuntimeClassName is the runtime class to set for the statefulSet pods. --- # Advanced ## Config reference[​](#config-reference "Direct link to Config reference") ## `advanced` object [​](#advanced "Direct link to advanced")Advanced holds additional configuration for the vCluster control plane. ### `defaultImageRegistry` string [​](#advanced-defaultImageRegistry "Direct link to advanced-defaultImageRegistry")DefaultImageRegistry will be used as a prefix for all internal images deployed by vCluster or Helm. This makes it easy to upload all required vCluster images to a single private repository and set this value. Workload images are not affected by this. ### `virtualScheduler` object [​](#advanced-virtualScheduler "Direct link to advanced-virtualScheduler")VirtualScheduler defines if a scheduler should be used within the virtual cluster or the scheduling decision for workloads will be made by the host cluster. Deprecated: Use ControlPlane.Distro.K8S.Scheduler instead. #### `enabled` boolean false [​](#advanced-virtualScheduler-enabled "Direct link to advanced-virtualScheduler-enabled")Enabled defines if this option should be enabled. ### `serviceAccount` object [​](#advanced-serviceAccount "Direct link to advanced-serviceAccount")ServiceAccount specifies options for the vCluster control plane service account. #### `enabled` boolean true [​](#advanced-serviceAccount-enabled "Direct link to advanced-serviceAccount-enabled")Enabled specifies if the service account should get deployed. #### `name` string [​](#advanced-serviceAccount-name "Direct link to advanced-serviceAccount-name")Name specifies what name to use for the service account. #### `imagePullSecrets` object\[] [​](#advanced-serviceAccount-imagePullSecrets "Direct link to advanced-serviceAccount-imagePullSecrets")ImagePullSecrets defines extra image pull secrets for the service account. ##### `name` string [​](#advanced-serviceAccount-imagePullSecrets-name "Direct link to advanced-serviceAccount-imagePullSecrets-name")Name of the image pull secret to use. #### `annotations` object {} [​](#advanced-serviceAccount-annotations "Direct link to advanced-serviceAccount-annotations")Annotations are extra annotations for this resource. #### `labels` object {} [​](#advanced-serviceAccount-labels "Direct link to advanced-serviceAccount-labels")Labels are extra labels for this resource. ### `workloadServiceAccount` object [​](#advanced-workloadServiceAccount "Direct link to advanced-workloadServiceAccount")WorkloadServiceAccount specifies options for the service account that will be used for the workloads that run within the virtual cluster. #### `enabled` boolean true [​](#advanced-workloadServiceAccount-enabled "Direct link to advanced-workloadServiceAccount-enabled")Enabled specifies if the service account for the workloads should get deployed. #### `name` string [​](#advanced-workloadServiceAccount-name "Direct link to advanced-workloadServiceAccount-name")Name specifies what name to use for the service account for the virtual cluster workloads. #### `imagePullSecrets` object\[] [​](#advanced-workloadServiceAccount-imagePullSecrets "Direct link to advanced-workloadServiceAccount-imagePullSecrets")ImagePullSecrets defines extra image pull secrets for the workload service account. ##### `name` string [​](#advanced-workloadServiceAccount-imagePullSecrets-name "Direct link to advanced-workloadServiceAccount-imagePullSecrets-name")Name of the image pull secret to use. #### `annotations` object {} [​](#advanced-workloadServiceAccount-annotations "Direct link to advanced-workloadServiceAccount-annotations")Annotations are extra annotations for this resource. #### `labels` object {} [​](#advanced-workloadServiceAccount-labels "Direct link to advanced-workloadServiceAccount-labels")Labels are extra labels for this resource. ### `headlessService` object [​](#advanced-headlessService "Direct link to advanced-headlessService")HeadlessService specifies options for the headless service used for the vCluster StatefulSet. #### `annotations` object {} [​](#advanced-headlessService-annotations "Direct link to advanced-headlessService-annotations")Annotations are extra annotations for this resource. #### `labels` object {} [​](#advanced-headlessService-labels "Direct link to advanced-headlessService-labels")Labels are extra labels for this resource. ### `konnectivity` object [​](#advanced-konnectivity "Direct link to advanced-konnectivity")Konnectivity holds dedicated konnectivity configuration. This is only available when privateNodes.enabled is true. #### `server` object [​](#advanced-konnectivity-server "Direct link to advanced-konnectivity-server")Server holds configuration for the konnectivity server. ##### `enabled` boolean true [​](#advanced-konnectivity-server-enabled "Direct link to advanced-konnectivity-server-enabled")Enabled defines if the konnectivity server should be enabled. ##### `extraArgs` string\[] \[] [​](#advanced-konnectivity-server-extraArgs "Direct link to advanced-konnectivity-server-extraArgs")ExtraArgs are additional arguments to pass to the konnectivity server. #### `agent` object [​](#advanced-konnectivity-agent "Direct link to advanced-konnectivity-agent")Agent holds configuration for the konnectivity agent. ##### `enabled` boolean true [​](#advanced-konnectivity-agent-enabled "Direct link to advanced-konnectivity-agent-enabled")Enabled defines if the konnectivity agent should be enabled. ##### `replicas` integer 1 [​](#advanced-konnectivity-agent-replicas "Direct link to advanced-konnectivity-agent-replicas")Replicas is the number of replicas for the konnectivity agent. ##### `image` string [​](#advanced-konnectivity-agent-image "Direct link to advanced-konnectivity-agent-image")Image is the image for the konnectivity agent. ##### `imagePullPolicy` string [​](#advanced-konnectivity-agent-imagePullPolicy "Direct link to advanced-konnectivity-agent-imagePullPolicy")ImagePullPolicy is the policy how to pull the image. ##### `nodeSelector` object {} [​](#advanced-konnectivity-agent-nodeSelector "Direct link to advanced-konnectivity-agent-nodeSelector")NodeSelector is the node selector for the konnectivity agent. ##### `priorityClassName` string [​](#advanced-konnectivity-agent-priorityClassName "Direct link to advanced-konnectivity-agent-priorityClassName")PriorityClassName is the priority class name for the konnectivity agent. ##### `tolerations` object\[] \[] [​](#advanced-konnectivity-agent-tolerations "Direct link to advanced-konnectivity-agent-tolerations")Tolerations is the tolerations for the konnectivity agent. ##### `extraEnv` object\[] \[] [​](#advanced-konnectivity-agent-extraEnv "Direct link to advanced-konnectivity-agent-extraEnv")ExtraEnv is the extra environment variables for the konnectivity agent. ##### `extraArgs` string\[] \[] [​](#advanced-konnectivity-agent-extraArgs "Direct link to advanced-konnectivity-agent-extraArgs")ExtraArgs are additional arguments to pass to the konnectivity agent. ### `registry` object [​](#advanced-registry "Direct link to advanced-registry")Registry allows enabling an embedded docker image registry in vCluster. This is useful for air-gapped environments or when you don't have a public registry available to distribute images. #### `enabled` boolean false [​](#advanced-registry-enabled "Direct link to advanced-registry-enabled")Enabled defines if the embedded registry should be enabled. #### `anonymousPull` boolean true [​](#advanced-registry-anonymousPull "Direct link to advanced-registry-anonymousPull")AnonymousPull allows enabling anonymous pull for the embedded registry. This allows anybody to pull images from the registry without authentication. #### `config` object {} [​](#advanced-registry-config "Direct link to advanced-registry-config")Config is the regular docker registry config. See for more details. ### `cloudControllerManager` object [​](#advanced-cloudControllerManager "Direct link to advanced-cloudControllerManager")CloudControllerManager holds configuration for the embedded cloud controller manager. This is only available when private nodes are enabled. The cloud controller manager is responsible for setting the node's ip addresses as well as the provider id for the node and other node metadata. #### `enabled` boolean true [​](#advanced-cloudControllerManager-enabled "Direct link to advanced-cloudControllerManager-enabled")Enabled defines if the embedded cloud controller manager should be enabled. This defaults to true, but can be disabled if you want to use an external cloud controller manager such as AWS or GCP. The cloud controller manager is responsible for setting the node's ip addresses as well as the provider id for the node and other node metadata. #### `kubeOvn` object [​](#advanced-cloudControllerManager-kubeOvn "Direct link to advanced-cloudControllerManager-kubeOvn")KubeOvn holds configuration for the kube-ovn integration of the embedded cloud controller manager. When enabled, services of type LoadBalancer are translated into kube-ovn RouterLBRule and OvnEip resources in the host cluster. ##### `enabled` boolean [​](#advanced-cloudControllerManager-kubeOvn-enabled "Direct link to advanced-cloudControllerManager-kubeOvn-enabled")Enabled defines if the kube-ovn integration of the embedded cloud controller manager should be enabled. This requires kube-ovn to be deployed in the host cluster and private nodes to be enabled. ##### `vpc` string [​](#advanced-cloudControllerManager-kubeOvn-vpc "Direct link to advanced-cloudControllerManager-kubeOvn-vpc")VPC is the name of the kube-ovn VPC that the integration operates on. It is shared by loadBalancers and podRoutes. ##### `loadBalancers` object [​](#advanced-cloudControllerManager-kubeOvn-loadBalancers "Direct link to advanced-cloudControllerManager-kubeOvn-loadBalancers")LoadBalancers holds configuration for translating services of type LoadBalancer into kube-ovn resources in the host cluster. ##### `externalSubnet` string [​](#advanced-cloudControllerManager-kubeOvn-loadBalancers-externalSubnet "Direct link to advanced-cloudControllerManager-kubeOvn-loadBalancers-externalSubnet")ExternalSubnet is the name of the kube-ovn subnet that the load balancer VIPs are allocated from. ##### `backendSubnet` string [​](#advanced-cloudControllerManager-kubeOvn-loadBalancers-backendSubnet "Direct link to advanced-cloudControllerManager-kubeOvn-loadBalancers-backendSubnet")BackendSubnet is the name of a kube-ovn subnet in the load balancer's VPC. It is stamped as the generated Service's ovn.kubernetes.io/logical\_switch so kube-ovn places the VIP on the VPC's load balancer; the value only needs to be a real subnet in that VPC. ##### `podRoutes` object [​](#advanced-cloudControllerManager-kubeOvn-podRoutes "Direct link to advanced-cloudControllerManager-kubeOvn-podRoutes")PodRoutes creates routes for the nodes' podCIDRs in the VPC, which enables direct routing for LoadBalancers. ##### `enabled` boolean [​](#advanced-cloudControllerManager-kubeOvn-podRoutes-enabled "Direct link to advanced-cloudControllerManager-kubeOvn-podRoutes-enabled")Enabled selects direct-to-pod mode, preserving the client source IP. ### `globalMetadata` object [​](#advanced-globalMetadata "Direct link to advanced-globalMetadata")GlobalMetadata is metadata that will be added to all resources deployed by Helm. #### `annotations` object {} [​](#advanced-globalMetadata-annotations "Direct link to advanced-globalMetadata-annotations")Annotations are extra annotations for this resource. ### `kubeVip` object [​](#advanced-kubeVip "Direct link to advanced-kubeVip")KubeVip holds configuration for embedded kube-vip that announces the virtual cluster endpoint IP on layer 2. #### `enabled` boolean false [​](#advanced-kubeVip-enabled "Direct link to advanced-kubeVip-enabled")Enabled defines if embedded kube-vip should be enabled. #### `interface` string [​](#advanced-kubeVip-interface "Direct link to advanced-kubeVip-interface")Interface is the network interface on which the VIP is announced. #### `gateway` string [​](#advanced-kubeVip-gateway "Direct link to advanced-kubeVip-gateway")Gateway is the gateway address in CIDR notation (e.g., 10.100.0.1/24). This is used to configure policy-based routing for the VIP and must include the subnet prefix. ### `podDisruptionBudget` object [​](#advanced-podDisruptionBudget "Direct link to advanced-podDisruptionBudget")PodDisruptionBudget limits how many pods of an application can be voluntarily disrupted at once to ensure availability during maintenance or scaling operations. #### `enabled` boolean false [​](#advanced-podDisruptionBudget-enabled "Direct link to advanced-podDisruptionBudget-enabled")Enabled defines if the pod disruption budget should be enabled. #### `minAvailable` object [​](#advanced-podDisruptionBudget-minAvailable "Direct link to advanced-podDisruptionBudget-minAvailable")MinAvailable describes the minimal number or percentage of available pods. #### `maxUnavailable` object [​](#advanced-podDisruptionBudget-maxUnavailable "Direct link to advanced-podDisruptionBudget-maxUnavailable")MaxUnavailable describes the minimal number or percentage of unavailable pods. #### `unhealthyPodEvictionPolicy` string [​](#advanced-podDisruptionBudget-unhealthyPodEvictionPolicy "Direct link to advanced-podDisruptionBudget-unhealthyPodEvictionPolicy")UnhealthyPodEvictionPolicy defines the criteria when unhealthy pods should be considered for eviction. Currently supported values are:- IfHealthyBudget - pods that are in the Running phase but not yet healthy are considered disrupted and may be evicted even if the PodDisruptionBudget criteria are not met. - AlwaysAllow - pods that are in the Running phase but not yet healthy are considered disrupted and can be evicted regardless of whether the criteria in a PDB is met. --- # Default image registry Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Use `defaultImageRegistry` to configure a private image registry. For example: ``` controlPlane: advanced: defaultImageRegistry: my-private-registry:5000/vcluster/ ``` vCluster prepends the image registry to all images used by vCluster, such as syncer, K8s, and CoreDNS. For example, `coredns/coredns:v1.11` becomes `my-private-registry:5000/vcluster/coredns/coredns:v1.11`. You can find a list of all needed images by vCluster in the file `images.txt` at the releases page, as well as two scripts (`download-images.sh` and `push-images.sh`) to pull and push those to your private registry. To use your private registry, you need to [configure `imagePullSecrets`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/service-account.md). ## Config reference[​](#config-reference "Direct link to Config reference") ## `defaultImageRegistry` string [​](#defaultImageRegistry "Direct link to defaultImageRegistry")DefaultImageRegistry will be used as a prefix for all internal images deployed by vCluster or Helm. This makes it easy to upload all required vCluster images to a single private repository and set this value. Workload images are not affected by this. --- # Global metadata Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Use this feature to add annotations to all vCluster components. ## Config reference[​](#config-reference "Direct link to Config reference") ## `globalMetadata` object [​](#globalMetadata "Direct link to globalMetadata")GlobalMetadata is metadata that will be added to all resources deployed by Helm. ### `annotations` object {} [​](#globalMetadata-annotations "Direct link to globalMetadata-annotations")Annotations are extra annotations for this resource. --- # Headless service Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** `StatefulSet` pods require a headless Service, as detailed in [Limitations](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#limitations). Configure labels and annotations on that headless Service. ## Config reference[​](#config-reference "Direct link to Config reference") ## `headlessService` object [​](#headlessService "Direct link to headlessService")HeadlessService specifies options for the headless service used for the vCluster StatefulSet. ### `annotations` object {} [​](#headlessService-annotations "Direct link to headlessService-annotations")Annotations are extra annotations for this resource. ### `labels` object {} [​](#headlessService-labels "Direct link to headlessService-labels")Labels are extra labels for this resource. --- Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** # Kube-vip Kube-vip announces the tenant cluster's control plane endpoint IP address on a specified network interface using ARP (layer 2). This makes the endpoint available to private nodes on the same layer 2 network, allowing them to connect to the control plane using a stable IP address. Kube-vip is intended to be used with high-availability (HA) deployments. When the control plane has multiple replicas and the active pod is terminated, kube-vip automatically sends proactive updates about the new VIP location to neighbors using gratuitous ARP, ensuring continuity without manual intervention. ## How it works[​](#how-it-works "Direct link to How it works") When kube-vip is enabled: 1. A virtual IP address (VIP) is configured for the vCluster control plane through the `controlPlane.endpoint` field 2. Kube-vip uses leader election within the tenant cluster to determine which control plane replica manages the VIP 3. The leader adds the VIP on the specified network interface 4. Worker nodes on the same layer 2 network discover the active control plane instance through ARP 5. If the control plane pod is rescheduled, the new leader sends gratuitous ARP packets to update peers about the new VIP location ## Enable kube-vip[​](#enable-kube-vip "Direct link to Enable kube-vip") Configure kube-vip in your `vcluster.yaml`: ``` # kube-vip is only compatible with private node virtual clusters privateNodes: enabled: true controlPlane: # the endpoint must be specified with the VIP endpoint: 10.100.0.100:8443 advanced: kubeVip: # enable kube-vip enabled: true interface: eth0 gateway: 10.100.0.1/24 ``` ### Configuration options[​](#configuration-options "Direct link to Configuration options") * **`interface`** - The network interface on which the VIP is announced (for example, `eth0`, `ens192`). When using in conjunction with Multus, this is normally `net1`. * **`gateway`** - The gateway address in CIDR notation (for example, `10.100.0.1/24`). This is used to configure policy-based routing for the VIP and must include the subnet prefix. ## Example[​](#example "Direct link to Example") Here's a complete example of a vCluster configuration with kube-vip using [Multus](https://github.com/k8snetworkplumbingwg/multus-cni). Kube-vip is especially useful in scenarios where the normal vCluster service endpoint is not reachable from private nodes. Multus is a meta CNI plugin that can attach Pods to additional networks. It is a good alternative, especially in bare metal deployments, where a load balancer may not be available or be considerable effort to set up. This example assumes: * Multus is installed in the control plane cluster * A bridge named `br-private` exists on the host nodes * The bridge is connected to an isolated network where the private nodes are also being deployed First, create a NetworkAttachmentDefinition for the bridge network: ``` apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: private-vcluster namespace: vcluster-platform spec: config: | { "cniVersion": "0.3.1", "type": "bridge", "bridge": "br-private" } ``` Then configure your vCluster: ``` privateNodes: enabled: true controlPlane: endpoint: 10.100.0.100:8443 statefulSet: annotations: k8s.v1.cni.cncf.io/networks: vcluster-platform/private-vcluster advanced: kubeVip: enabled: true interface: net1 gateway: 10.100.0.1/24 networking: podCIDR: 10.64.0.0/16 serviceCIDR: 10.128.0.0/16 ``` ## Limitations[​](#limitations "Direct link to Limitations") * The network interface specified must exist on the vCluster control plane instances (for example, through Multus) * The VIP must be routable from the worker nodes (layer 3), for example through the specified gateway * Embedded Kube-vip currently only supports ARP mode. Other features such as BGP are not supported or configurable. ## Security considerations[​](#security-considerations "Direct link to Security considerations") When kube-vip is enabled, the control plane pods are automatically granted `NET_ADMIN` (to manage the IP on the interface) and `NET_RAW` capabilities (to send gratuitous ARP packets). These capabilities are required for kube-vip to function and do not need to be configured manually. ## Config reference[​](#config-reference "Direct link to Config reference") ## `kubeVip` object [​](#kubeVip "Direct link to kubeVip")KubeVip holds configuration for embedded kube-vip that announces the virtual cluster endpoint IP on layer 2. ### `enabled` boolean false [​](#kubeVip-enabled "Direct link to kubeVip-enabled")Enabled defines if embedded kube-vip should be enabled. ### `interface` string [​](#kubeVip-interface "Direct link to kubeVip-interface")Interface is the network interface on which the VIP is announced. ### `gateway` string [​](#kubeVip-gateway "Direct link to kubeVip-gateway")Gateway is the gateway address in CIDR notation (e.g., 10.100.0.1/24). This is used to configure policy-based routing for the VIP and must include the subnet prefix. ## Additional resources[​](#additional-resources "Direct link to Additional resources") * [Deploy with private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) * [kube-vip reference](https://kube-vip.io/) * [Multus reference](https://github.com/k8snetworkplumbingwg/multus-cni) --- # Service account Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Modify the vCluster control plane's ServiceAccount. You may need to do this in these situations: * You are using a private image registry. See `defaultImageRegistry` [configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/default-image-registry.md). * You are using a Cloud provider's IAM integration, such as [Google's Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) and AWS's [IAM roles for service accounts(IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). * You need to provide AuthN or AuthZ for cloud provider image registries. ## Config reference[​](#config-reference "Direct link to Config reference") ## `serviceAccount` object [​](#serviceAccount "Direct link to serviceAccount")ServiceAccount specifies options for the vCluster control plane service account. ### `enabled` boolean true [​](#serviceAccount-enabled "Direct link to serviceAccount-enabled")Enabled specifies if the service account should get deployed. ### `name` string [​](#serviceAccount-name "Direct link to serviceAccount-name")Name specifies what name to use for the service account. ### `imagePullSecrets` object\[] [​](#serviceAccount-imagePullSecrets "Direct link to serviceAccount-imagePullSecrets")ImagePullSecrets defines extra image pull secrets for the service account. #### `name` string [​](#serviceAccount-imagePullSecrets-name "Direct link to serviceAccount-imagePullSecrets-name")Name of the image pull secret to use. ### `annotations` object {} [​](#serviceAccount-annotations "Direct link to serviceAccount-annotations")Annotations are extra annotations for this resource. ### `labels` object {} [​](#serviceAccount-labels "Direct link to serviceAccount-labels")Labels are extra labels for this resource. --- Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** # Virtual scheduler controlPlane.advanced.virtualScheduler is deprecated The `controlPlane.advanced.virtualScheduler` field is deprecated since vCluster 0.26. Use [`controlPlane.distro.k8s.scheduler`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/distro/k8s.md#k8s-scheduler-enabled) instead. The `enabled` field works the same way on the replacement, and the deprecated field still works, but new configurations should use the current path. By default, vCluster reuses the control plane cluster scheduler to schedule workloads. This conserves compute resources. If you also [sync PersistentVolumeClaim resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/persistent-volume-claims.md), vCluster mirrors the related `CSIDriver`, `CSINode`, and `CSIStorageCapacity` objects into the tenant cluster. This allows the virtual scheduler to make storage-aware scheduling decisions. In some cases, you might want to label or taint nodes inside the tenant cluster to influence scheduling decisions—for example, using [node affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) or [topology spread constraints](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/). To support these use cases, vCluster allows you to run a scheduler inside the tenant cluster instead of reusing the control plane cluster’s scheduler. ## Enable the virtual scheduler[​](#enable-the-virtual-scheduler "Direct link to Enable the virtual scheduler") To use the virtual scheduler, you must enable the following: * Syncing nodes from the control plane cluster (`sync.fromHost.nodes.enabled`) * The virtual scheduler (`controlPlane.distro.k8s.scheduler.enabled`) When the virtual scheduler is enabled: * vCluster only syncs pods that already have a node assigned in the control plane cluster. * You can label and taint nodes within the tenant cluster without affecting the control plane cluster. Configure your `vcluster.yaml` to sync nodes from the control plane cluster and enable the virtual scheduler: ``` sync: fromHost: nodes: enabled: true controlPlane: distro: k8s: scheduler: enabled: true ``` This setting makes control plane cluster nodes available inside the tenant cluster. It allows the virtual scheduler to make scheduling decisions based on node labels and taints. ## Automatically enable sync features[​](#automatically-enable-sync-features "Direct link to Automatically enable sync features") When the virtual scheduler is enabled, the following resources are automatically synced from the control plane cluster: * [CSI drivers](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-drivers.md) * [CSI nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-nodes.md) * [CSI storage capacities](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-storage-capacities.md) This automatic syncing only occurs if: * Syncing for CSI resources from the control plane cluster is set to auto (the default). * `PersistentVolumeClaim` syncing to the control plane cluster is enabled. info If you [sync PersistentVolumeClaim resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/persistent-volume-claims.md), vCluster mirrors the related `CSIDriver`, `CSINode`, and `CSIStorageCapacity` objects into the tenant cluster. This allows the virtual scheduler to make storage-aware scheduling decisions. Manually disabling these features might result in incorrect pod scheduling behavior. ### Example[​](#example "Direct link to Example") If your `vcluster.yaml` is configured as: ``` controlPlane: distro: k8s: scheduler: enabled: true sync: toHost: persistentVolumeClaims: enabled: true fromHost: # All 'auto' values below are set by default csiDrivers: enabled: auto csiNodes: enabled: auto csiStorageCapacities: enabled: auto ``` Then, vCluster automatically updates the config to: ``` controlPlane: distro: k8s: scheduler: enabled: true sync: toHost: persistentVolumeClaims: enabled: true # vCluster enables these values because the virtual scheduler is active fromHost: csiDrivers: enabled: true csiNodes: enabled: true csiStorageCapacities: enabled: true ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `virtualScheduler` object [​](#virtualScheduler "Direct link to virtualScheduler")VirtualScheduler defines if a scheduler should be used within the virtual cluster or the scheduling decision for workloads will be made by the host cluster. Deprecated: Use ControlPlane.Distro.K8S.Scheduler instead. ### `enabled` boolean false [​](#virtualScheduler-enabled "Direct link to virtualScheduler-enabled")Enabled defines if this option should be enabled. ## Additional resources[​](#additional-resources "Direct link to Additional resources") * [Sync `persistentVolumeClaims`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/persistent-volume-claims.md) * [Sync `csiStorageCapacities`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-storage-capacities.md) * [Sync `csiNodes`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-nodes.md) * [Sync `csiDrivers`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-drivers.md) --- # Workload service account Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Private Nodes For private nodes, workload pods run on dedicated nodes and do not sync to the control plane cluster namespace. Configure image pull authentication at the node level using [`privateNodes.joinNode.containerd.registry`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes.md#privateNodes-joinNode-containerd-registry). The workloadServiceAccount allows you to enforce the use of a specific ServiceAccount for all vCluster workload pods running in the control plane cluster. For example, you can attach a shared imagePullSecret to the ServiceAccount so that all synced pods use the same secret when pulling container images. ``` controlPlane: advanced: workloadServiceAccount: enabled: true name: image-pull-sa imagePullSecrets: - name: workload-service-account-image-pull-secret ``` note The pod specification (spec) of the synced workloads should not define its own imagePullSecrets. This is because imagePullSecrets are only copied from the ServiceAccount to the pod when the pod specification does not already include them. The referenced imagePullSecrets must also exist in the host namespace where the vCluster’s workload pods are synced. An easy way to achieve this is by using [Virtual Cluster Templates](https://vcluster.com/docs/platform/administer/templates/create-templates.md). In addition to allowing you to add arbitrary Kubernetes objects inside the tenant cluster, Virtual Cluster Templates also enable you to create arbitrary Kubernetes objects in the control plane cluster namespace where the vCluster’s StatefulSet is deployed. If [syncing ServiceAccount resources from the tenant cluster to the control plane cluster](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/service-accounts.md) is enabled, the `workloadServiceAccount` setting is ignored. ## Config reference[​](#config-reference "Direct link to Config reference") ## `workloadServiceAccount` object [​](#workloadServiceAccount "Direct link to workloadServiceAccount")WorkloadServiceAccount specifies options for the service account that will be used for the workloads that run within the virtual cluster. ### `enabled` boolean true [​](#workloadServiceAccount-enabled "Direct link to workloadServiceAccount-enabled")Enabled specifies if the service account for the workloads should get deployed. ### `name` string [​](#workloadServiceAccount-name "Direct link to workloadServiceAccount-name")Name specifies what name to use for the service account for the virtual cluster workloads. ### `imagePullSecrets` object\[] [​](#workloadServiceAccount-imagePullSecrets "Direct link to workloadServiceAccount-imagePullSecrets")ImagePullSecrets defines extra image pull secrets for the workload service account. #### `name` string [​](#workloadServiceAccount-imagePullSecrets-name "Direct link to workloadServiceAccount-imagePullSecrets-name")Name of the image pull secret to use. ### `annotations` object {} [​](#workloadServiceAccount-annotations "Direct link to workloadServiceAccount-annotations")Annotations are extra annotations for this resource. ### `labels` object {} [​](#workloadServiceAccount-labels "Direct link to workloadServiceAccount-labels")Labels are extra labels for this resource. --- # Configure automatic deletion Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. Tenant cluster deletion can't be undone. To prevent deletion, set `deletion.prevent` or the `loft.sh/non-deletable` annotation to `true`. note To delete this tenant cluster later, reverse whichever setting you used: set `deletion.prevent` to `false`, or remove the `loft.sh/non-deletable` annotation. Then delete the cluster as usual. Configure `deletion.auto` to automatically delete a tenant cluster after a period of inactivity. This is useful for cleanup after single-shot batch jobs, so you don't have to delete the cluster manually. important Automatic deletion requires an agent installed on the control plane cluster. ## Configure deletion[​](#configure-deletion "Direct link to Configure deletion") * vcluster.yaml * Platform UI * Custom Resource ### Prevent deletion[​](#prevent-deletion "Direct link to Prevent deletion") To prevent a tenant cluster from being deleted, add the following configuration to your `vcluster.yaml`: Prevent Deletion ``` deletion: prevent: true ``` ### Deletion after inactivity[​](#deletion-after-inactivity "Direct link to Deletion after inactivity") To enable automatic deletion after a certain inactivity duration, add the following configuration to your `vcluster.yaml`: Automatic Deletion ``` deletion: auto: afterInactivity: 1m # Uses Go duration format ``` 1. Select the project that contains the tenant cluster. 2. Click **Tenant Clusters** in the sidebar. 3. Click **Edit** for the tenant cluster. 4. Expand **Auto-Sleep and Auto-Delete**. 5. Enable **Prevent Deletion**, or set **Delete after inactivity** to configure automatic deletion. 6. Click **Save**. For a tenant cluster created from a template, edit the template instead of the instance. You can prevent tenant cluster deletion by setting the annotation `loft.sh/non-deletable: "true"` on the tenant cluster instance's YAML manifest. ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterInstance metadata: name: my-vcluster annotations: loft.sh/non-deletable: "true" spec: templateRef: name: isolated-vcluster clusterRef: cluster: loft-cluster ``` ## Configuration reference[​](#configuration-reference "Direct link to Configuration reference") ## `deletion` object [​](#deletion "Direct link to deletion")Deletion holds configuration for automatic vCluster deletion. ### `prevent` boolean [​](#deletion-prevent "Direct link to deletion-prevent")Prevent prevents the vCluster from being deleted ### `auto` object [​](#deletion-auto "Direct link to deletion-auto")Auto holds automatic deletion configuration #### `afterInactivity` string [​](#deletion-auto-afterInactivity "Direct link to deletion-auto-afterInactivity")AfterInactivity specifies after how long of inactivity the tenant cluster will be deleted. Uses Go duration format (e.g., "720h" for 30 days). --- # Deploy vCluster AddOns ## Configure addons[​](#configure-addons "Direct link to Configure addons") vCluster supports addons that extend the capabilities of your tenant cluster. You can configure these addons during deployment to adjust networking, observability, and other features for your environment and requirements. ### Metrics server[​](#metrics-server "Direct link to Metrics server") Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** vCluster can install the [Kubernetes Metrics Server](https://github.com/kubernetes-sigs/metrics-server) into the tenant cluster. Enable it when you need `kubectl top` output or want the Horizontal Pod Autoscaler to use CPU and memory metrics. Enable Metrics Server ``` deploy: metricsServer: enabled: true ``` ### CNI[​](#cni "Direct link to CNI") Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** vCluster installs [Flannel](https://github.com/flannel-io/flannel) as the Container Network Interface (CNI) by default. You can disable it and install your own CNI. Disable default Flannel CNI ``` deploy: cni: flannel: enabled: false ``` User-managed component When you disable Flannel, you are responsible for installing and operating a CNI in the tenant cluster. vCluster support does not cover the configuration or behavior of user-managed CNI installations. ### Kube proxy[​](#kube-proxy "Direct link to Kube proxy") Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** vCluster installs [kube-proxy](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) by default to configure services on worker nodes. If your CNI implements its own service proxy (for example, Cilium in kube-proxy replacement mode), you can disable it. Disable kube-proxy ``` deploy: kubeProxy: enabled: false ``` To customize kube-proxy behavior without disabling it, use the `config` field. Values merge into the default KubeProxyConfiguration. See the [Kubernetes kube-proxy config API reference](https://kubernetes.io/docs/reference/config-api/kube-proxy-config.v1alpha1/) for available fields. Custom kube-proxy config ``` deploy: kubeProxy: config: mode: ipvs ipvs: scheduler: rr ``` Use `extraArgs` to pass additional flags directly to the kube-proxy binary. ### MetalLB[​](#metallb "Direct link to MetalLB") Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** vCluster can install [MetalLB](https://metallb.io/) into the tenant cluster. Enable this with the following YAML: Enable MetalLB ``` deploy: metallb: enabled: true ipAddressPool: addresses: - 10.X.X.1-10.X.X.100 # your IPs here ``` ### Local path provisioner[​](#local-path-provisioner "Direct link to Local path provisioner") Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** vCluster installs the [Local Path Provisioner](https://github.com/rancher/local-path-provisioner) by default to provide a default StorageClass for stateful workloads. If you bring your own storage solution, you can disable it. Disable Local Path Provisioner ``` deploy: localPathProvisioner: enabled: false ``` User-managed component When you disable the Local Path Provisioner and replace it with your own StorageClass, vCluster support does not cover your storage configuration. Replacing the provisioner does not affect existing PersistentVolumes, However, new PersistentVolumeClaims do not bind until a replacement provisioner is available. ### KubeVirt CSI[​](#kubevirt-csi "Direct link to KubeVirt CSI") Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** vCluster can deploy a [KubeVirt](https://kubevirt.io/) CSI driver into the tenant cluster. It provisions PersistentVolumes as CDI DataVolumes in the vCluster's namespace on the control plane cluster. It then hot-plugs them into the KubeVirt virtual machines backing the tenant cluster's private nodes. This is a storage integration, unrelated to the [KubeVirt integration](https://vcluster.com/docs/vcluster/integrations/kubevirt.md), which syncs KubeVirt VM resources from the tenant cluster to the control plane cluster. See [How this differs from KubeVirt CSI](https://vcluster.com/docs/vcluster/integrations/kubevirt.md#how-this-differs-from-kubevirt-csi) for the distinction. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * [Private nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes.md) enabled. KubeVirt CSI is rejected when `controlPlane.standalone.enabled` is `true`, since volumes are provisioned in a control-plane-cluster namespace that standalone deployments don't have. * [KubeVirt Containerized Data Importer (CDI)](https://kubevirt.io/user-guide/storage/containerized_data_importer/) installed on the control plane cluster. Volumes are provisioned as CDI `DataVolume` objects rather than plain PersistentVolumeClaims, because KubeVirt's volume hot-plug expects CDI to have already created the underlying disk image. * The private nodes backing this tenant cluster must be KubeVirt virtual machines in the same namespace as the tenant cluster. You can provision them through a [KubeVirt node provider](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md) and [auto nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/auto-nodes.md). The controller first matches a virtual machine to a node by name, then falls back to matching internal IP addresses. It retries until a matching virtual machine is available. The driver doesn't create virtual machines itself. #### Enable KubeVirt CSI[​](#enable-kubevirt-csi "Direct link to Enable KubeVirt CSI") Enable the KubeVirt CSI driver ``` deploy: # Disable the default Local Path Provisioner to avoid two default StorageClasses. localPathProvisioner: enabled: false csi: kubeVirt: enabled: true ``` #### Storage class behavior[​](#storage-class-behavior "Direct link to Storage class behavior") By default, vCluster creates a StorageClass named `kubevirt` inside the tenant cluster and marks it as the default StorageClass. Disable either behavior with `storageClass.enabled: false` or `storageClass.default: false`. A default KubeVirt CSI StorageClass conflicts with [Local path provisioner](#local-path-provisioner), since both would create a default StorageClass in the tenant cluster. Disable one of them if you enable the other. Use a specific host StorageClass and disk bus ``` deploy: csi: kubeVirt: enabled: true storageClass: name: kubevirt-fast # Keep the Local Path Provisioner as the tenant cluster's default. default: false hostStorageClassName: fast-ssd bus: virtio ``` Leave `hostStorageClassName` empty to use the control plane cluster's default StorageClass for the underlying CDI DataVolume. `bus` defaults to `scsi` and is passed through to KubeVirt's disk configuration as-is. vCluster doesn't validate it, so any bus your KubeVirt installation supports (for example `virtio`, `sata`, or `scsi`) is accepted. ### Platform stacks[​](#platform-stacks "Direct link to Platform stacks") When a tenant cluster is managed by vCluster Platform 4.12, vCluster 0.37 can request StackInstances through `deploy.stacks`. Each Stack deploys to the tenant cluster after Platform creates it. Deploy a Stack ``` deploy: stacks: - name: observability # Resolve a StackTemplate from vCluster Platform. templateRef: name: observability parameters: retention: 7d # Keep children when a later template removes their tasks. prunePolicy: Retain ``` Each entry requires a stable `name` and exactly one definition source: * `templateRef.name` references a cluster-scoped StackTemplate. * `template` defines the parameters, tasks, and published outputs inline. The optional `parameters` map supplies `.Values` used by the tasks. `defaults.taskTimeout` sets the default task timeout. `prunePolicy` is `Retain` by default and can be set to `Prune`. This field requires vCluster Platform. The Platform UI hides the Stacks form for vCluster versions that don't support it. See [Use a Stack](https://vcluster.com/docs/platform/use-platform/apps/use-stacks.md) for the user workflow and status model. ## Control plane components[​](#control-plane-components "Direct link to Control plane components") CoreDNS and Konnectivity are control plane components, not deploy addons. Configure them under `controlPlane` in your vcluster.yaml: * **CoreDNS**: see [CoreDNS configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/coredns.md) * **Konnectivity**: configure or disable under `controlPlane.advanced.konnectivity` ## Config reference[​](#config-reference "Direct link to Config reference") ## `deploy` object [​](#deploy "Direct link to deploy")Deploy holds configuration for the deployment of vCluster. ### `kubeProxy` object [​](#deploy-kubeProxy "Direct link to deploy-kubeProxy")KubeProxy holds dedicated kube proxy configuration. #### `enabled` boolean true [​](#deploy-kubeProxy-enabled "Direct link to deploy-kubeProxy-enabled")Enabled defines if the kube proxy should be enabled. #### `image` string [​](#deploy-kubeProxy-image "Direct link to deploy-kubeProxy-image")Image is the image for the kube-proxy. #### `imagePullPolicy` string [​](#deploy-kubeProxy-imagePullPolicy "Direct link to deploy-kubeProxy-imagePullPolicy")ImagePullPolicy is the policy how to pull the image. #### `nodeSelector` object {} [​](#deploy-kubeProxy-nodeSelector "Direct link to deploy-kubeProxy-nodeSelector")NodeSelector is the node selector for the kube-proxy. #### `priorityClassName` string [​](#deploy-kubeProxy-priorityClassName "Direct link to deploy-kubeProxy-priorityClassName")PriorityClassName is the priority class name for the kube-proxy. #### `tolerations` object\[] \[] [​](#deploy-kubeProxy-tolerations "Direct link to deploy-kubeProxy-tolerations")Tolerations is the tolerations for the kube-proxy. #### `extraEnv` object\[] \[] [​](#deploy-kubeProxy-extraEnv "Direct link to deploy-kubeProxy-extraEnv")ExtraEnv is the extra environment variables for the kube-proxy. #### `extraArgs` string\[] \[] [​](#deploy-kubeProxy-extraArgs "Direct link to deploy-kubeProxy-extraArgs")ExtraArgs are additional arguments to pass to the kube-proxy. #### `config` object {} [​](#deploy-kubeProxy-config "Direct link to deploy-kubeProxy-config")Config is the config for the kube-proxy that will be merged into the default kube-proxy config. More information can be found here: ### `metallb` object [​](#deploy-metallb "Direct link to deploy-metallb")Metallb holds dedicated metallb configuration. #### `enabled` boolean false [​](#deploy-metallb-enabled "Direct link to deploy-metallb-enabled")Enabled defines if metallb should be enabled. #### `controllerImage` string [​](#deploy-metallb-controllerImage "Direct link to deploy-metallb-controllerImage")ControllerImage is the image for metallb controller. #### `speakerImage` string [​](#deploy-metallb-speakerImage "Direct link to deploy-metallb-speakerImage")SpeakerImage is the image for metallb speaker. #### `ipAddressPool` object [​](#deploy-metallb-ipAddressPool "Direct link to deploy-metallb-ipAddressPool")IPAddressPool is the IP address pool to use for metallb. ##### `addresses` string\[] \[] [​](#deploy-metallb-ipAddressPool-addresses "Direct link to deploy-metallb-ipAddressPool-addresses")Addresses is a list of IP addresses to use for the IP address pool. ##### `l2Advertisement` boolean true [​](#deploy-metallb-ipAddressPool-l2Advertisement "Direct link to deploy-metallb-ipAddressPool-l2Advertisement")L2Advertisement defines if L2 advertisement should be enabled for the IP address pool. ### `cni` object [​](#deploy-cni "Direct link to deploy-cni")CNI holds dedicated CNI configuration. #### `flannel` object [​](#deploy-cni-flannel "Direct link to deploy-cni-flannel")Flannel holds dedicated Flannel configuration. ##### `enabled` boolean true [​](#deploy-cni-flannel-enabled "Direct link to deploy-cni-flannel-enabled")Enabled defines if Flannel should be enabled. ##### `image` string [​](#deploy-cni-flannel-image "Direct link to deploy-cni-flannel-image")Image is the image for Flannel main container. ##### `initImage` string [​](#deploy-cni-flannel-initImage "Direct link to deploy-cni-flannel-initImage")InitImage is the image for Flannel init container. ##### `imagePullPolicy` string [​](#deploy-cni-flannel-imagePullPolicy "Direct link to deploy-cni-flannel-imagePullPolicy")ImagePullPolicy is the policy how to pull the image. ### `csi` object [​](#deploy-csi "Direct link to deploy-csi")CSI holds dedicated CSI configuration. #### `kubeVirt` object [​](#deploy-csi-kubeVirt "Direct link to deploy-csi-kubeVirt")KubeVirt holds dedicated kubevirt CSI configuration. ##### `enabled` boolean false [​](#deploy-csi-kubeVirt-enabled "Direct link to deploy-csi-kubeVirt-enabled")Enabled defines if kubevirt CSI should be enabled. This allows workloads within this virtual cluster to use persistent volumes that vCluster provisions within its own namespace in the host cluster and hot plugs into the kubevirt virtual machines that back the nodes of this virtual cluster. The virtual machines have to run within the same namespace as this virtual cluster and vCluster annotates the nodes with the virtual machine they run on, which the CSI node plugin needs. The volumes are provisioned as CDI data volumes, so CDI has to be installed within the host cluster. This is only supported in private nodes mode. ##### `image` string [​](#deploy-csi-kubeVirt-image "Direct link to deploy-csi-kubeVirt-image")Image is the image for the kubevirt CSI node plugin. The controller side is part of the vCluster control plane, so this is the only image that is deployed. ##### `imagePullPolicy` string [​](#deploy-csi-kubeVirt-imagePullPolicy "Direct link to deploy-csi-kubeVirt-imagePullPolicy")ImagePullPolicy is the policy how to pull the image. ##### `hostLabels` object {} [​](#deploy-csi-kubeVirt-hostLabels "Direct link to deploy-csi-kubeVirt-hostLabels")HostLabels are additional labels vCluster adds to the persistent volume claims it creates within the namespace of this virtual cluster in the host cluster. ##### `storageClass` object [​](#deploy-csi-kubeVirt-storageClass "Direct link to deploy-csi-kubeVirt-storageClass")StorageClass holds configuration for the storage class that is created within this virtual cluster. ##### `enabled` boolean true [​](#deploy-csi-kubeVirt-storageClass-enabled "Direct link to deploy-csi-kubeVirt-storageClass-enabled")Enabled defines if the storage class should be created within this virtual cluster. ##### `name` string kubevirt [​](#deploy-csi-kubeVirt-storageClass-name "Direct link to deploy-csi-kubeVirt-storageClass-name")Name is the name of the storage class. ##### `default` boolean true [​](#deploy-csi-kubeVirt-storageClass-default "Direct link to deploy-csi-kubeVirt-storageClass-default")Default defines if the storage class should be the default storage class of this virtual cluster. This is not allowed while deploy.localPathProvisioner is enabled, because that creates a default storage class as well and two default storage classes leave it to Kubernetes which provisioner a claim without a storage class ends up on. ##### `hostStorageClassName` string [​](#deploy-csi-kubeVirt-storageClass-hostStorageClassName "Direct link to deploy-csi-kubeVirt-storageClass-hostStorageClassName")HostStorageClassName is the storage class within the host cluster that is used to provision the volumes. If empty, the default storage class of the host cluster is used. ##### `hostVolumeMode` string [​](#deploy-csi-kubeVirt-storageClass-hostVolumeMode "Direct link to deploy-csi-kubeVirt-storageClass-hostVolumeMode")HostVolumeMode pins the volume mode of the volumes that are created within the host cluster. If empty, the storage profile CDI keeps for the storage class of the host cluster decides. ##### `bus` string scsi [​](#deploy-csi-kubeVirt-storageClass-bus "Direct link to deploy-csi-kubeVirt-storageClass-bus")Bus is the bus the volumes are attached with to the virtual machine. ### `localPathProvisioner` object [​](#deploy-localPathProvisioner "Direct link to deploy-localPathProvisioner")LocalPathProvisioner holds dedicated local path provisioner configuration. #### `enabled` boolean true [​](#deploy-localPathProvisioner-enabled "Direct link to deploy-localPathProvisioner-enabled")Enabled defines if LocalPathProvisioner should be enabled. #### `image` string [​](#deploy-localPathProvisioner-image "Direct link to deploy-localPathProvisioner-image")Image is the image for local path provisioner. #### `imagePullPolicy` string [​](#deploy-localPathProvisioner-imagePullPolicy "Direct link to deploy-localPathProvisioner-imagePullPolicy")ImagePullPolicy is the policy how to pull the image. #### `nodePath` string [​](#deploy-localPathProvisioner-nodePath "Direct link to deploy-localPathProvisioner-nodePath")NodePath is the path on the node where to create the persistent volume directories. ### `ingressNginx` object [​](#deploy-ingressNginx "Direct link to deploy-ingressNginx")IngressNginx holds dedicated ingress-nginx configuration. Deprecated: We do not deploy ingress nginx and the project is being deprecated. #### `enabled` boolean false [​](#deploy-ingressNginx-enabled "Direct link to deploy-ingressNginx-enabled")Enabled defines if ingress-nginx should be enabled. #### `defaultIngressClass` boolean true [​](#deploy-ingressNginx-defaultIngressClass "Direct link to deploy-ingressNginx-defaultIngressClass")DefaultIngressClass defines if the deployed ingress class should be the default ingress class. ### `metricsServer` object [​](#deploy-metricsServer "Direct link to deploy-metricsServer")MetricsServer holds dedicated metrics server configuration. #### `enabled` boolean false [​](#deploy-metricsServer-enabled "Direct link to deploy-metricsServer-enabled")Enabled defines if metrics server should be enabled. ### `volumeSnapshotController` object [​](#deploy-volumeSnapshotController "Direct link to deploy-volumeSnapshotController")VolumeSnapshotController holds dedicated CSI snapshot-controller configuration. Deprecated: Removed in 0.36.0. Setting this value is rejected by the Helm chart; remove it from your config. #### `enabled` boolean [​](#deploy-volumeSnapshotController-enabled "Direct link to deploy-volumeSnapshotController-enabled")Enabled defines if the CSI volumes snapshot-controller should be enabled. ### `argoCD` object [​](#deploy-argoCD "Direct link to deploy-argoCD")ArgoCD holds dedicated configuration for argoCD Apps to deploy #### `applications` object\[] [​](#deploy-argoCD-applications "Direct link to deploy-argoCD-applications")Applications specifies the applications to deploy. This requires the argo cd integration to be enabled. ##### `name` string [​](#deploy-argoCD-applications-name "Direct link to deploy-argoCD-applications-name")Name specifies the stable identifier of the argo cd application. It is used to derive generated ArgoCDApplication resource names and the final Argo CD application name. ##### `displayName` string [​](#deploy-argoCD-applications-displayName "Direct link to deploy-argoCD-applications-displayName")DisplayName specifies the display name of the argo cd application. ##### `target` string [​](#deploy-argoCD-applications-target "Direct link to deploy-argoCD-applications-target")Target specifies the target of the argo cd application. This can be "vCluster" or "host". Defaults to "vCluster". ##### `inline` object [​](#deploy-argoCD-applications-inline "Direct link to deploy-argoCD-applications-inline")Inline specifies the inline argo cd application definition. This requires the argo cd integration to be enabled. ##### `template` object [​](#deploy-argoCD-applications-template "Direct link to deploy-argoCD-applications-template")Template specifies the argo cd application template to use. This requires the argo cd integration to be enabled. ##### `name` string [​](#deploy-argoCD-applications-template-name "Direct link to deploy-argoCD-applications-template-name")Name specifies the name of the argo cd application template ##### `parameters` object [​](#deploy-argoCD-applications-template-parameters "Direct link to deploy-argoCD-applications-template-parameters")Parameters specifies the parameters to pass to the argo cd application template. ### `stacks` object\[] [​](#deploy-stacks "Direct link to deploy-stacks")Stack holds dedicated configuration for stacks to deploy #### `name` string [​](#deploy-stacks-name "Direct link to deploy-stacks-name")Name specifies the stable identifier of the stack. #### `displayName` string [​](#deploy-stacks-displayName "Direct link to deploy-stacks-displayName")DisplayName specifies the display name of the stack. #### `description` string [​](#deploy-stacks-description "Direct link to deploy-stacks-description")Description describes the stack. #### `template` object [​](#deploy-stacks-template "Direct link to deploy-stacks-template")Template defines the stack inline: parameter declarations and tasks, the same payload a StackTemplate carries. Mutually exclusive with TemplateRef. ##### `parameters` object\[] [​](#deploy-stacks-template-parameters "Direct link to deploy-stacks-template-parameters")Parameters declares the parameters the tasks may reference as {{ .Values.variable }}. Values are supplied by the stack's parameters field. ##### `variable` string [​](#deploy-stacks-template-parameters-variable "Direct link to deploy-stacks-template-parameters-variable")Variable is the path of the variable. Can be foo or foo.bar for nested objects. ##### `label` string [​](#deploy-stacks-template-parameters-label "Direct link to deploy-stacks-template-parameters-label")Label is the label to show for this parameter ##### `description` string [​](#deploy-stacks-template-parameters-description "Direct link to deploy-stacks-template-parameters-description")Description is the description to show for this parameter ##### `type` string [​](#deploy-stacks-template-parameters-type "Direct link to deploy-stacks-template-parameters-type")Type of the parameter. Can be one of: string, multiline, boolean, number and password ##### `options` string\[] [​](#deploy-stacks-template-parameters-options "Direct link to deploy-stacks-template-parameters-options")Options is a slice of strings, where each string represents a mutually exclusive choice. ##### `min` integer [​](#deploy-stacks-template-parameters-min "Direct link to deploy-stacks-template-parameters-min")Min is the minimum number if type is number ##### `max` integer [​](#deploy-stacks-template-parameters-max "Direct link to deploy-stacks-template-parameters-max")Max is the maximum number if type is number ##### `required` boolean [​](#deploy-stacks-template-parameters-required "Direct link to deploy-stacks-template-parameters-required")Required specifies if this parameter is required ##### `defaultValue` string [​](#deploy-stacks-template-parameters-defaultValue "Direct link to deploy-stacks-template-parameters-defaultValue")DefaultValue is the default value if none is specified ##### `placeholder` string [​](#deploy-stacks-template-parameters-placeholder "Direct link to deploy-stacks-template-parameters-placeholder")Placeholder shown in the UI ##### `invalidation` string [​](#deploy-stacks-template-parameters-invalidation "Direct link to deploy-stacks-template-parameters-invalidation")Invalidation regex that if matched will reject the input ##### `validation` string [​](#deploy-stacks-template-parameters-validation "Direct link to deploy-stacks-template-parameters-validation")Validation regex that if matched will allow the input ##### `section` string [​](#deploy-stacks-template-parameters-section "Direct link to deploy-stacks-template-parameters-section")Section where this app should be displayed. Apps with the same section name will be grouped together ##### `tasks` object\[] [​](#deploy-stacks-template-tasks "Direct link to deploy-stacks-template-tasks")Tasks defines the stack DAG. ##### `name` string [​](#deploy-stacks-template-tasks-name "Direct link to deploy-stacks-template-tasks-name")Name specifies the stable identifier of the task. DNS-label-safe, unique within the stack. ##### `dependsOn` string\[] [​](#deploy-stacks-template-tasks-dependsOn "Direct link to deploy-stacks-template-tasks-dependsOn")DependsOn lists task names that must be healthy before this task starts. ##### `argoCDApplication` object [​](#deploy-stacks-template-tasks-argoCDApplication "Direct link to deploy-stacks-template-tasks-argoCDApplication")ArgoCDApplication specifies the argo cd application this task deploys. Exactly one of ArgoCDApplication or App must be set. ##### `template` object [​](#deploy-stacks-template-tasks-argoCDApplication-template "Direct link to deploy-stacks-template-tasks-argoCDApplication-template")Template specifies the inline argo cd application template definition. Mutually exclusive with TemplateRef. ##### `templateRef` object [​](#deploy-stacks-template-tasks-argoCDApplication-templateRef "Direct link to deploy-stacks-template-tasks-argoCDApplication-templateRef")TemplateRef references an ArgoCDApplicationTemplate (per-application blueprint) by name. Mutually exclusive with Template. ##### `name` string [​](#deploy-stacks-template-tasks-argoCDApplication-templateRef-name "Direct link to deploy-stacks-template-tasks-argoCDApplication-templateRef-name")Name of the ArgoCDApplicationTemplate. ##### `parameters` object [​](#deploy-stacks-template-tasks-argoCDApplication-parameters "Direct link to deploy-stacks-template-tasks-argoCDApplication-parameters")Parameters supplies values to the referenced template's declared parameters. Only valid with TemplateRef; values for an inline template belong in the definition. ##### `app` object [​](#deploy-stacks-template-tasks-app "Direct link to deploy-stacks-template-tasks-app")App specifies the platform app this task deploys. Exactly one of ArgoCDApplication or App must be set. ##### `template` object [​](#deploy-stacks-template-tasks-app-template "Direct link to deploy-stacks-template-tasks-app-template")Template specifies the inline app definition, with metadata and a spec that carries the chart plus a few instance fields (displayName, description, releaseName). Mutually exclusive with TemplateRef. ##### `templateRef` object [​](#deploy-stacks-template-tasks-app-templateRef "Direct link to deploy-stacks-template-tasks-app-templateRef")TemplateRef references a named app by name and version. Mutually exclusive with Template. ##### `name` string [​](#deploy-stacks-template-tasks-app-templateRef-name "Direct link to deploy-stacks-template-tasks-app-templateRef-name")Name of the app to reference. ##### `version` string [​](#deploy-stacks-template-tasks-app-templateRef-version "Direct link to deploy-stacks-template-tasks-app-templateRef-version")Version of the app to deploy. If empty, the latest version is deployed. ##### `parameters` object [​](#deploy-stacks-template-tasks-app-parameters "Direct link to deploy-stacks-template-tasks-app-parameters")Parameters supplies values to the referenced app. Only valid with TemplateRef; values for an inline template belong in the definition. ##### `timeout` string [​](#deploy-stacks-template-tasks-timeout "Direct link to deploy-stacks-template-tasks-timeout")Timeout bounds how long this task may take to become healthy, overriding defaults.taskTimeout. Go duration string, for example "10m". ##### `outputs` object\[] [​](#deploy-stacks-template-tasks-outputs "Direct link to deploy-stacks-template-tasks-outputs")Outputs declares values captured from this task's deployed resources that later tasks reference as {{ .Outputs.task.name }}. ##### `name` string [​](#deploy-stacks-template-tasks-outputs-name "Direct link to deploy-stacks-template-tasks-outputs-name")Name identifies the output. Letters and digits only, unique within the task. ##### `fromSecret` object [​](#deploy-stacks-template-tasks-outputs-fromSecret "Direct link to deploy-stacks-template-tasks-outputs-fromSecret")FromSecret reads the value from a Secret key on the destination cluster. Exactly one of fromSecret or fromResource must be set. ##### `namespace` string [​](#deploy-stacks-template-tasks-outputs-fromSecret-namespace "Direct link to deploy-stacks-template-tasks-outputs-fromSecret-namespace")Namespace of the Secret. Must be a namespace this stack deploys into. ##### `name` string [​](#deploy-stacks-template-tasks-outputs-fromSecret-name "Direct link to deploy-stacks-template-tasks-outputs-fromSecret-name")Name of the Secret. ##### `key` string [​](#deploy-stacks-template-tasks-outputs-fromSecret-key "Direct link to deploy-stacks-template-tasks-outputs-fromSecret-key")Key inside the Secret's data. ##### `fromResource` object [​](#deploy-stacks-template-tasks-outputs-fromResource "Direct link to deploy-stacks-template-tasks-outputs-fromResource")FromResource reads a single field from a resource on the destination cluster. Exactly one of fromSecret or fromResource must be set. ##### `apiVersion` string [​](#deploy-stacks-template-tasks-outputs-fromResource-apiVersion "Direct link to deploy-stacks-template-tasks-outputs-fromResource-apiVersion")APIVersion of the resource, e.g. "v1" or "apps/v1". ##### `kind` string [​](#deploy-stacks-template-tasks-outputs-fromResource-kind "Direct link to deploy-stacks-template-tasks-outputs-fromResource-kind")Kind of the resource, e.g. "Service". ##### `namespace` string [​](#deploy-stacks-template-tasks-outputs-fromResource-namespace "Direct link to deploy-stacks-template-tasks-outputs-fromResource-namespace")Namespace of the resource. Must be a namespace this stack deploys into; cluster-scoped resources cannot be read. ##### `name` string [​](#deploy-stacks-template-tasks-outputs-fromResource-name "Direct link to deploy-stacks-template-tasks-outputs-fromResource-name")Name of the resource. ##### `jsonPath` string [​](#deploy-stacks-template-tasks-outputs-fromResource-jsonPath "Direct link to deploy-stacks-template-tasks-outputs-fromResource-jsonPath")JSONPath selects the field, in the kubectl template syntax, e.g. "{.spec.clusterIP}". It must select exactly one scalar value. ##### `publishedOutputs` object\[] [​](#deploy-stacks-template-publishedOutputs "Direct link to deploy-stacks-template-publishedOutputs")PublishedOutputs selects which captured task outputs the stack exposes to its users, under a public name. They are read through the stack instance's outputs subresource. ##### `name` string [​](#deploy-stacks-template-publishedOutputs-name "Direct link to deploy-stacks-template-publishedOutputs-name")Name specifies the public name of the output. Unique within the stack; it may differ from the task output's name. ##### `fromTask` object [​](#deploy-stacks-template-publishedOutputs-fromTask "Direct link to deploy-stacks-template-publishedOutputs-fromTask")FromTask selects which task output to publish. It must name an existing task and one of its declared outputs. ##### `task` string [​](#deploy-stacks-template-publishedOutputs-fromTask-task "Direct link to deploy-stacks-template-publishedOutputs-fromTask-task")Task specifies the task name. ##### `output` string [​](#deploy-stacks-template-publishedOutputs-fromTask-output "Direct link to deploy-stacks-template-publishedOutputs-fromTask-output")Output specifies the output name declared on that task. #### `templateRef` object [​](#deploy-stacks-templateRef "Direct link to deploy-stacks-templateRef")TemplateRef references a cluster-scoped StackTemplate to resolve the tasks from. Mutually exclusive with Template. ##### `name` string [​](#deploy-stacks-templateRef-name "Direct link to deploy-stacks-templateRef-name")Name specifies the name of the StackTemplate to reference. #### `parameters` object [​](#deploy-stacks-parameters "Direct link to deploy-stacks-parameters")Parameters specifies the values the template's tasks reference. Values the template does not declare are still available to tasks, but get no default and are not validated. #### `defaults` object [​](#deploy-stacks-defaults "Direct link to deploy-stacks-defaults")Defaults are applied to all tasks unless overridden per-task. ##### `taskTimeout` string [​](#deploy-stacks-defaults-taskTimeout "Direct link to deploy-stacks-defaults-taskTimeout")TaskTimeout is the default per-task timeout as a Go duration (e.g. "10m"). #### `prunePolicy` string [​](#deploy-stacks-prunePolicy "Direct link to deploy-stacks-prunePolicy")PrunePolicy controls what happens to an application whose task is removed from the resolved set. One of "Retain" (default) or "Prune". --- # Experimental warning The following features are experimental. They might have breaking changes or be deprecated. ## Experimental features[​](#experimental-features "Direct link to Experimental features") vCluster provides several experimental features that extend its capabilities: ### Available features[​](#available-features "Direct link to Available features") * **[Deploy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/deploy.md)** - Configure deployment settings for experimental features * **[Sync settings](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/sync-settings.md)** - Advanced synchronization configuration * **[Deny proxy requests](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/deny-proxy-requests.md)** - Block proxy requests to the control plane cluster * **[Resource proxy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/proxy.md)** - Proxy custom resources to other tenant clusters ### Configure experimental features[​](#configure-experimental-features "Direct link to Configure experimental features") Experimental features are configured under the `experimental` section of your vCluster configuration. These features might change between releases, so use them with caution in production environments. ## Config reference[​](#config-reference "Direct link to Config reference") ## `experimental` object [​](#experimental "Direct link to experimental")Experimental features for vCluster. Configuration here might change, so be careful with this. ### `deploy` object [​](#experimental-deploy "Direct link to experimental-deploy")Deploy allows you to configure manifests and Helm charts to deploy within the host or virtual cluster. #### `host` object [​](#experimental-deploy-host "Direct link to experimental-deploy-host")Host defines what manifests to deploy into the host cluster ##### `manifests` string [​](#experimental-deploy-host-manifests "Direct link to experimental-deploy-host-manifests")Manifests are raw Kubernetes manifests that should get applied within the host cluster. ##### `manifestsTemplate` string [​](#experimental-deploy-host-manifestsTemplate "Direct link to experimental-deploy-host-manifestsTemplate")ManifestsTemplate is a Kubernetes manifest template that will be rendered with vCluster values before applying it within the host cluster. #### `vcluster` object [​](#experimental-deploy-vcluster "Direct link to experimental-deploy-vcluster")VCluster defines what manifests and charts to deploy into the vCluster ##### `manifests` string [​](#experimental-deploy-vcluster-manifests "Direct link to experimental-deploy-vcluster-manifests")Manifests are raw Kubernetes manifests that should get applied within the virtual cluster. ##### `manifestsTemplate` string [​](#experimental-deploy-vcluster-manifestsTemplate "Direct link to experimental-deploy-vcluster-manifestsTemplate")ManifestsTemplate is a Kubernetes manifest template that will be rendered with vCluster values before applying it within the virtual cluster. ##### `helm` object\[] [​](#experimental-deploy-vcluster-helm "Direct link to experimental-deploy-vcluster-helm")Helm are Helm charts that should get deployed into the virtual cluster ##### `chart` object [​](#experimental-deploy-vcluster-helm-chart "Direct link to experimental-deploy-vcluster-helm-chart")Chart defines what chart should get deployed. ##### `name` string [​](#experimental-deploy-vcluster-helm-chart-name "Direct link to experimental-deploy-vcluster-helm-chart-name") ##### `repo` string [​](#experimental-deploy-vcluster-helm-chart-repo "Direct link to experimental-deploy-vcluster-helm-chart-repo") ##### `insecure` boolean [​](#experimental-deploy-vcluster-helm-chart-insecure "Direct link to experimental-deploy-vcluster-helm-chart-insecure") ##### `version` string [​](#experimental-deploy-vcluster-helm-chart-version "Direct link to experimental-deploy-vcluster-helm-chart-version") ##### `username` string [​](#experimental-deploy-vcluster-helm-chart-username "Direct link to experimental-deploy-vcluster-helm-chart-username") ##### `password` string [​](#experimental-deploy-vcluster-helm-chart-password "Direct link to experimental-deploy-vcluster-helm-chart-password") ##### `release` object [​](#experimental-deploy-vcluster-helm-release "Direct link to experimental-deploy-vcluster-helm-release")Release defines what release should get deployed. ##### `name` string [​](#experimental-deploy-vcluster-helm-release-name "Direct link to experimental-deploy-vcluster-helm-release-name")Name of the release ##### `namespace` string [​](#experimental-deploy-vcluster-helm-release-namespace "Direct link to experimental-deploy-vcluster-helm-release-namespace")Namespace of the release ##### `values` string [​](#experimental-deploy-vcluster-helm-values "Direct link to experimental-deploy-vcluster-helm-values")Values defines what values should get used. ##### `timeout` string [​](#experimental-deploy-vcluster-helm-timeout "Direct link to experimental-deploy-vcluster-helm-timeout")Timeout defines the timeout for Helm ##### `bundle` string [​](#experimental-deploy-vcluster-helm-bundle "Direct link to experimental-deploy-vcluster-helm-bundle")Bundle allows to compress the Helm chart and specify this instead of an online chart ### `syncSettings` object [​](#experimental-syncSettings "Direct link to experimental-syncSettings")SyncSettings are advanced settings for the syncer controller. #### `setOwner` boolean true [​](#experimental-syncSettings-setOwner "Direct link to experimental-syncSettings-setOwner")SetOwner specifies if vCluster should set an owner reference on the synced objects to the vCluster service. This allows for easy garbage collection. #### `hostMetricsBindAddress` string [​](#experimental-syncSettings-hostMetricsBindAddress "Direct link to experimental-syncSettings-hostMetricsBindAddress")HostMetricsBindAddress is the bind address for the local manager #### `virtualMetricsBindAddress` string [​](#experimental-syncSettings-virtualMetricsBindAddress "Direct link to experimental-syncSettings-virtualMetricsBindAddress")VirtualMetricsBindAddress is the bind address for the virtual manager ### `virtualClusterKubeConfig` object [​](#experimental-virtualClusterKubeConfig "Direct link to experimental-virtualClusterKubeConfig")VirtualClusterKubeConfig allows you to override distro specifics and specify where vCluster will find the required certificates and vCluster config. Deprecated: Removed in 0.29.0. #### `kubeConfig` string [​](#experimental-virtualClusterKubeConfig-kubeConfig "Direct link to experimental-virtualClusterKubeConfig-kubeConfig")KubeConfig is the virtual cluster kubeconfig path. #### `serverCAKey` string [​](#experimental-virtualClusterKubeConfig-serverCAKey "Direct link to experimental-virtualClusterKubeConfig-serverCAKey")ServerCAKey is the server ca key path. #### `serverCACert` string [​](#experimental-virtualClusterKubeConfig-serverCACert "Direct link to experimental-virtualClusterKubeConfig-serverCACert")ServerCACert is the server ca cert path. #### `clientCACert` string [​](#experimental-virtualClusterKubeConfig-clientCACert "Direct link to experimental-virtualClusterKubeConfig-clientCACert")ClientCACert is the client ca cert path. #### `clientCAKey` string [​](#experimental-virtualClusterKubeConfig-clientCAKey "Direct link to experimental-virtualClusterKubeConfig-clientCAKey")ClientCAKey is the client ca key path. #### `requestHeaderCACert` string [​](#experimental-virtualClusterKubeConfig-requestHeaderCACert "Direct link to experimental-virtualClusterKubeConfig-requestHeaderCACert")RequestHeaderCACert is the request header ca cert path. ### `denyProxyRequests` object\[] [​](#experimental-denyProxyRequests "Direct link to experimental-denyProxyRequests")DenyProxyRequests denies certain requests in the vCluster proxy. #### `name` string [​](#experimental-denyProxyRequests-name "Direct link to experimental-denyProxyRequests-name")The name of the check. #### `namespaces` string\[] [​](#experimental-denyProxyRequests-namespaces "Direct link to experimental-denyProxyRequests-namespaces")Namespace describe a list of namespaces that will be affected by the check. An empty list means that all namespaces will be affected. In case of ClusterScoped rules, only the Namespace resource is affected. #### `rules` object\[] [​](#experimental-denyProxyRequests-rules "Direct link to experimental-denyProxyRequests-rules")Rules describes on which verbs and on what resources/subresources the webhook is enforced. The webhook is enforced if it matches any Rule. The version of the request must match the rule version exactly. Equivalent matching is not supported. ##### `apiGroups` string\[] [​](#experimental-denyProxyRequests-rules-apiGroups "Direct link to experimental-denyProxyRequests-rules-apiGroups")APIGroups is the API groups the resources belong to. '\*' is all groups. ##### `apiVersions` string\[] [​](#experimental-denyProxyRequests-rules-apiVersions "Direct link to experimental-denyProxyRequests-rules-apiVersions")APIVersions is the API versions the resources belong to. '\*' is all versions. ##### `resources` string\[] [​](#experimental-denyProxyRequests-rules-resources "Direct link to experimental-denyProxyRequests-rules-resources")Resources is a list of resources this rule applies to. ##### `scope` string [​](#experimental-denyProxyRequests-rules-scope "Direct link to experimental-denyProxyRequests-rules-scope")Scope specifies the scope of this rule. ##### `operations` string\[] [​](#experimental-denyProxyRequests-rules-operations "Direct link to experimental-denyProxyRequests-rules-operations")Verb is the kube verb associated with the request for API requests, not the http verb. This includes things like list and watch. For non-resource requests, this is the lowercase http verb. If '\*' is present, the length of the slice must be one. #### `excludedUsers` string\[] [​](#experimental-denyProxyRequests-excludedUsers "Direct link to experimental-denyProxyRequests-excludedUsers")ExcludedUsers describe a list of users for which the checks will be skipped. Impersonation attempts on these users will still be subjected to the checks. ### `proxy` object [​](#experimental-proxy "Direct link to experimental-proxy")Proxy enables vCluster-to-vCluster proxying of resources #### `customResources` {key: object} [​](#experimental-proxy-customResources "Direct link to experimental-proxy-customResources")CustomResources is a map of resource keys (format: "kind.apiGroup/version") to proxy configuration ##### `enabled` boolean [​](#experimental-proxy-customResources-enabled "Direct link to experimental-proxy-customResources-enabled")Enabled defines if this resource proxy should be enabled ##### `targetVirtualCluster` object [​](#experimental-proxy-customResources-targetVirtualCluster "Direct link to experimental-proxy-customResources-targetVirtualCluster")TargetVirtualCluster is the target virtual cluster for the custom resource proxy ##### `name` required string [​](#experimental-proxy-customResources-targetVirtualCluster-name "Direct link to experimental-proxy-customResources-targetVirtualCluster-name")Name is the name of the target virtual cluster. ##### `project` string [​](#experimental-proxy-customResources-targetVirtualCluster-project "Direct link to experimental-proxy-customResources-targetVirtualCluster-project")Project is the project of the target virtual cluster. If empty, defaults to the same project as the source vCluster. ##### `accessResources` string [​](#experimental-proxy-customResources-accessResources "Direct link to experimental-proxy-customResources-accessResources")AccessResources defines which resources should be accessible in the proxy. ### `docker` object [​](#experimental-docker "Direct link to experimental-docker")Docker allows you to configure Docker related settings when deploying a vCluster using Docker. #### `image` string [​](#experimental-docker-image "Direct link to experimental-docker-image")Image defines the image to use for the container. Defaults to ghcr.io/loft-sh/vm-container. #### `ports` string\[] [​](#experimental-docker-ports "Direct link to experimental-docker-ports")Ports defines extra port mappings to be added to the container. #### `volumes` string\[] [​](#experimental-docker-volumes "Direct link to experimental-docker-volumes")Volumes defines extra volumes to be added to the container. #### `env` string\[] [​](#experimental-docker-env "Direct link to experimental-docker-env")Env defines extra environment variables to be added to the container. Use key=value. #### `args` string\[] [​](#experimental-docker-args "Direct link to experimental-docker-args")Args defines extra arguments to be added to the docker run command of the container. #### `enabled` boolean [​](#experimental-docker-enabled "Direct link to experimental-docker-enabled")Enabled defines if the vCluster was deployed using Docker. This is automatically set by vCluster and should not be set by the user. #### `network` string [​](#experimental-docker-network "Direct link to experimental-docker-network")Network defines the network to use for the vCluster. If not specified, the a network will be created for the vCluster. #### `nodes` object\[] [​](#experimental-docker-nodes "Direct link to experimental-docker-nodes")Nodes defines the nodes of the vCluster. ##### `image` string [​](#experimental-docker-nodes-image "Direct link to experimental-docker-nodes-image")Image defines the image to use for the container. Defaults to ghcr.io/loft-sh/vm-container. ##### `ports` string\[] [​](#experimental-docker-nodes-ports "Direct link to experimental-docker-nodes-ports")Ports defines extra port mappings to be added to the container. ##### `volumes` string\[] [​](#experimental-docker-nodes-volumes "Direct link to experimental-docker-nodes-volumes")Volumes defines extra volumes to be added to the container. ##### `env` string\[] [​](#experimental-docker-nodes-env "Direct link to experimental-docker-nodes-env")Env defines extra environment variables to be added to the container. Use key=value. ##### `args` string\[] [​](#experimental-docker-nodes-args "Direct link to experimental-docker-nodes-args")Args defines extra arguments to be added to the docker run command of the container. ##### `name` string [​](#experimental-docker-nodes-name "Direct link to experimental-docker-nodes-name")Name defines the name of the node. If not specified, a random name will be generated. #### `registryProxy` object [​](#experimental-docker-registryProxy "Direct link to experimental-docker-registryProxy")Defines if docker images should be pulled from the host docker daemon. This prevents pulling images again and allows to use purely local images. Only works if containerd image storage is used. For more information, see ##### `enabled` boolean true [​](#experimental-docker-registryProxy-enabled "Direct link to experimental-docker-registryProxy-enabled")Enabled defines if this option should be enabled. #### `loadBalancer` object [​](#experimental-docker-loadBalancer "Direct link to experimental-docker-loadBalancer")Defines if vCluster should configure load balancer services inside the vCluster. This might require sudo access on the host cluster for docker desktop or rancher desktop on macos. ##### `enabled` boolean true [​](#experimental-docker-loadBalancer-enabled "Direct link to experimental-docker-loadBalancer-enabled")Enabled defines if this option should be enabled. ##### `forwardPorts` boolean true [​](#experimental-docker-loadBalancer-forwardPorts "Direct link to experimental-docker-loadBalancer-forwardPorts")ForwardPorts defines if the load balancer ips should be made available locally via port forwarding. This will be only done if necessary for example on macos when using docker desktop. ### `nodeMonitors` object\[] [​](#experimental-nodeMonitors "Direct link to experimental-nodeMonitors")NodeMonitors allows you to create a service monitor for each node. #### `name` string [​](#experimental-nodeMonitors-name "Direct link to experimental-nodeMonitors-name")Name is the name of the monitor. It will be suffixed with the node name. #### `nodeSelector` object [​](#experimental-nodeMonitors-nodeSelector "Direct link to experimental-nodeMonitors-nodeSelector")NodeSelector defines the node selector for the service monitor. #### `endpoints` object\[] [​](#experimental-nodeMonitors-endpoints "Direct link to experimental-nodeMonitors-endpoints")Endpoints is a list of endpoints to add to the service monitor. By default, vCluster will relabel the node and instance label to the node name. ##### `path` string [​](#experimental-nodeMonitors-endpoints-path "Direct link to experimental-nodeMonitors-endpoints-path")Path is the kubelet path of the endpoint. vCluster will prepend /api/v1/nodes/NODE\_NAME to the path. ##### `params` object [​](#experimental-nodeMonitors-endpoints-params "Direct link to experimental-nodeMonitors-endpoints-params")Params allows you to configure extra parameters to add to the endpoint. ##### `extraRelabelings` object\[] [​](#experimental-nodeMonitors-endpoints-extraRelabelings "Direct link to experimental-nodeMonitors-endpoints-extraRelabelings")ExtraRelabelings allows you to configure extra relabelings to add to the endpoint. By default, vCluster will relabel the node and instance label to the node name. ##### `metricsRelabelings` object\[] [​](#experimental-nodeMonitors-endpoints-metricsRelabelings "Direct link to experimental-nodeMonitors-endpoints-metricsRelabelings")MetricsRelabelings allows you to configure extra metrics relabelings to add to the endpoint. ##### `interval` string [​](#experimental-nodeMonitors-endpoints-interval "Direct link to experimental-nodeMonitors-endpoints-interval")Interval is the interval at which to scrape the endpoint. ##### `scrapeTimeout` string [​](#experimental-nodeMonitors-endpoints-scrapeTimeout "Direct link to experimental-nodeMonitors-endpoints-scrapeTimeout")ScrapeTimeout is the timeout for the scrape of the endpoint. #### `spec` object [​](#experimental-nodeMonitors-spec "Direct link to experimental-nodeMonitors-spec")Spec allows you to configure extra service monitor options that will be merged into the spec. #### `annotations` object [​](#experimental-nodeMonitors-annotations "Direct link to experimental-nodeMonitors-annotations")Annotations are extra annotations for this resource. #### `labels` object [​](#experimental-nodeMonitors-labels "Direct link to experimental-nodeMonitors-labels")Labels are extra labels for this resource. --- # Denying proxy requests Experimental Feature This feature is experimental. Breaking changes may occur between releases. It may become Enterprise-only or be deprecated. Not recommended for production use. Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. Configure denying proxy requests based on rules. * Only validation based on matched namespaces, and resource group / kind / version and request verbs are supported. * No mutation of requests, only allow / deny. ## Example[​](#example "Direct link to Example") ``` experimental: denyProxyRequests: - name: "Deny \"protected\" namespace" namespaces: - protected rules: - operations: - delete - update - patch apiGroups: - '' apiVersions: - "*" resources: - "namespaces/*" scope: Cluster - operations: - create - delete - update - patch apiGroups: - "*" apiVersions: - "*" resources: - "*" scope: Namespaced excludedUsers: - system:admin - system:serviceaccount:default:excluded ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `denyProxyRequests` object\[] [​](#denyProxyRequests "Direct link to denyProxyRequests")DenyProxyRequests denies certain requests in the vCluster proxy. ### `name` string [​](#denyProxyRequests-name "Direct link to denyProxyRequests-name")The name of the check. ### `namespaces` string\[] [​](#denyProxyRequests-namespaces "Direct link to denyProxyRequests-namespaces")Namespace describe a list of namespaces that will be affected by the check. An empty list means that all namespaces will be affected. In case of ClusterScoped rules, only the Namespace resource is affected. ### `rules` object\[] [​](#denyProxyRequests-rules "Direct link to denyProxyRequests-rules")Rules describes on which verbs and on what resources/subresources the webhook is enforced. The webhook is enforced if it matches any Rule. The version of the request must match the rule version exactly. Equivalent matching is not supported. #### `apiGroups` string\[] [​](#denyProxyRequests-rules-apiGroups "Direct link to denyProxyRequests-rules-apiGroups")APIGroups is the API groups the resources belong to. '\*' is all groups. #### `apiVersions` string\[] [​](#denyProxyRequests-rules-apiVersions "Direct link to denyProxyRequests-rules-apiVersions")APIVersions is the API versions the resources belong to. '\*' is all versions. #### `resources` string\[] [​](#denyProxyRequests-rules-resources "Direct link to denyProxyRequests-rules-resources")Resources is a list of resources this rule applies to. #### `scope` string [​](#denyProxyRequests-rules-scope "Direct link to denyProxyRequests-rules-scope")Scope specifies the scope of this rule. #### `operations` string\[] [​](#denyProxyRequests-rules-operations "Direct link to denyProxyRequests-rules-operations")Verb is the kube verb associated with the request for API requests, not the http verb. This includes things like list and watch. For non-resource requests, this is the lowercase http verb. If '\*' is present, the length of the slice must be one. ### `excludedUsers` string\[] [​](#denyProxyRequests-excludedUsers "Direct link to denyProxyRequests-excludedUsers")ExcludedUsers describe a list of users for which the checks will be skipped. Impersonation attempts on these users will still be subjected to the checks. --- # Experimental deploy Experimental Feature This feature is experimental. Breaking changes may occur between releases. It may become Enterprise-only or be deprecated. Not recommended for production use. Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** ## Apply manifests on initialization in vCluster[​](#apply-manifests-on-initialization-in-vcluster "Direct link to Apply manifests on initialization in vCluster") Apply Kubernetes manifests when the tenant cluster starts. This is useful for configuring internal vCluster resources. For example: ``` experimental: deploy: vcluster: manifests: |- apiVersion: v1 kind: Service ... apiVersion: v1 kind: ConfigMap ... ``` ## Apply manifest templates on initialization in vCluster[​](#apply-manifest-templates-on-initialization-in-vcluster "Direct link to Apply manifest templates on initialization in vCluster") Add a manifest template that uses Helm values. For example: ``` experimental: deploy: vcluster: manifestsTemplate: |- apiVersion: v1 kind: ConfigMap data: example: {{ .Release.Name }} ``` ## Apply Helm charts on initialization in vCluster[​](#apply-helm-charts-on-initialization-in-vcluster "Direct link to Apply Helm charts on initialization in vCluster") ### Upstream chart[​](#upstream-chart "Direct link to Upstream chart") Apply a Helm chart from public and private upstream chart repositories. This example deploys from public and private: ``` experimental: deploy: vcluster: helm: # public chart - chart: name: cert-manager repo: https://charts.jetstack.io version: v1.8.0 # optional field values: |- installCRDs: true release: name: cert-manager namespace: cert-manager # private chart - chart: name: my-chart repo: https://my-company.com/private-repo/ version: 1.0 username: my-username password: my-password release: name: my-release namespace: my-release-namespace ``` ### Local chart[​](#local-chart "Direct link to Local chart") Apply a chart that exists locally as a compressed `tar` archive. ``` experimental: deploy: vcluster: helm: bundle: COMPRESSED_STRING ``` Replace *`COMPRESSED_STRING`* with a compressed base64 string that has the chart's contents, which are then directly embedded in the vCluster values file. ## Apply manifests on initialization in host[​](#apply-manifests-on-initialization-in-host "Direct link to Apply manifests on initialization in host") Apply Kubernetes manifests when the tenant cluster starts into the host namespace. This is useful for configuring external vCluster resources. For example: ``` experimental: deploy: host: manifests: |- apiVersion: v1 kind: Secret ... apiVersion: v1 kind: ConfigMap ... manifestsTemplate: |- apiVersion: v1 kind: ConfigMap data: example: {{ .Release.Name }} ``` ## Config reference[​](#config-reference "Direct link to Config reference") info The `experimental.deploy.vcluster.helm[].chart.version` scheme only supports absolute versions and not any constraints. ## `deploy` object [​](#deploy "Direct link to deploy")Deploy allows you to configure manifests and Helm charts to deploy within the host or virtual cluster. ### `host` object [​](#deploy-host "Direct link to deploy-host")Host defines what manifests to deploy into the host cluster #### `manifests` string [​](#deploy-host-manifests "Direct link to deploy-host-manifests")Manifests are raw Kubernetes manifests that should get applied within the host cluster. #### `manifestsTemplate` string [​](#deploy-host-manifestsTemplate "Direct link to deploy-host-manifestsTemplate")ManifestsTemplate is a Kubernetes manifest template that will be rendered with vCluster values before applying it within the host cluster. ### `vcluster` object [​](#deploy-vcluster "Direct link to deploy-vcluster")VCluster defines what manifests and charts to deploy into the vCluster #### `manifests` string [​](#deploy-vcluster-manifests "Direct link to deploy-vcluster-manifests")Manifests are raw Kubernetes manifests that should get applied within the virtual cluster. #### `manifestsTemplate` string [​](#deploy-vcluster-manifestsTemplate "Direct link to deploy-vcluster-manifestsTemplate")ManifestsTemplate is a Kubernetes manifest template that will be rendered with vCluster values before applying it within the virtual cluster. #### `helm` object\[] [​](#deploy-vcluster-helm "Direct link to deploy-vcluster-helm")Helm are Helm charts that should get deployed into the virtual cluster ##### `chart` object [​](#deploy-vcluster-helm-chart "Direct link to deploy-vcluster-helm-chart")Chart defines what chart should get deployed. ##### `name` string [​](#deploy-vcluster-helm-chart-name "Direct link to deploy-vcluster-helm-chart-name") ##### `repo` string [​](#deploy-vcluster-helm-chart-repo "Direct link to deploy-vcluster-helm-chart-repo") ##### `insecure` boolean [​](#deploy-vcluster-helm-chart-insecure "Direct link to deploy-vcluster-helm-chart-insecure") ##### `version` string [​](#deploy-vcluster-helm-chart-version "Direct link to deploy-vcluster-helm-chart-version") ##### `username` string [​](#deploy-vcluster-helm-chart-username "Direct link to deploy-vcluster-helm-chart-username") ##### `password` string [​](#deploy-vcluster-helm-chart-password "Direct link to deploy-vcluster-helm-chart-password") ##### `release` object [​](#deploy-vcluster-helm-release "Direct link to deploy-vcluster-helm-release")Release defines what release should get deployed. ##### `name` string [​](#deploy-vcluster-helm-release-name "Direct link to deploy-vcluster-helm-release-name")Name of the release ##### `namespace` string [​](#deploy-vcluster-helm-release-namespace "Direct link to deploy-vcluster-helm-release-namespace")Namespace of the release ##### `values` string [​](#deploy-vcluster-helm-values "Direct link to deploy-vcluster-helm-values")Values defines what values should get used. ##### `timeout` string [​](#deploy-vcluster-helm-timeout "Direct link to deploy-vcluster-helm-timeout")Timeout defines the timeout for Helm ##### `bundle` string [​](#deploy-vcluster-helm-bundle "Direct link to deploy-vcluster-helm-bundle")Bundle allows to compress the Helm chart and specify this instead of an online chart --- # vCluster in Docker (vind) vCluster in Docker (vind) allows you to run virtual Kubernetes clusters directly in Docker containers instead of deploying them to an existing Kubernetes cluster. This provides a lightweight, isolated environment perfect for development, testing, and CI/CD scenarios. ## Overview[​](#overview "Direct link to Overview") **What is vind?** vind creates a complete Kubernetes cluster that runs entirely within Docker containers on your local machine or any Docker-enabled environment. Each vCluster instance gets its own set of containers that simulate Kubernetes nodes, providing full cluster functionality without requiring a host Kubernetes cluster. **Why use vind?** * **Local Development**: Quickly spin up isolated Kubernetes environments for development and testing * **Pull through cache**: No `kind load docker-image` or importing of images needed. vind uses your local docker daemon to pull images * **Automatic load balancers**: Kubernetes LoadBalancer services work out of the box and are reachable locally as well as within the cluster. Requires sudo on some platforms. See [Permissions](#permissions). * **External nodes**: Use [vCluster VPN](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/vpn.md) and the vCluster Platform to join external nodes into the vCluster * **Isolation**: Complete separation between different vCluster instances * **Portability**: Works consistently across different Docker environments * **CI/CD**: Use the [`setup-vind` GitHub Action](https://vcluster.com/docs/vcluster/integrations/setup-vind.md) to provision clusters in GitHub Actions workflows ## How to use[​](#how-to-use "Direct link to How to use") ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Docker installed and running * vCluster CLI installed ### Permissions[​](#permissions "Direct link to Permissions") Sudo required for load balancer services Creating a vCluster with load balancer support requires elevated privileges. Without sudo, the CLI outputs this warning: ``` warn Load balancer type services are not supported inside the vCluster because this command was executed with insufficient privileges. To enable load balancer type services, run this command with sudo ``` Run the `vcluster create` command with `sudo` to enable load balancer services. This is most commonly required on Docker Desktop and Rancher Desktop on macOS. Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster ``` sudo vcluster create my-vcluster --values values.yaml ``` If you don't need load balancer services, you can [disable load balancer services](#disable-load-balancer) and run without sudo. ### Basic usage[​](#basic-usage "Direct link to Basic usage") First, configure vCluster to use the Docker driver: ``` vcluster use driver docker ``` Then create a vCluster with Docker-specific configuration: Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster ``` vcluster create my-vcluster --values values.yaml ``` ### Configuration options[​](#configuration-options "Direct link to Configuration options") The `experimental.docker` section in your vCluster configuration allows you to customize various Docker-specific settings: #### Container customization[​](#container-customization "Direct link to Container customization") * **Image**: Specify custom container images for the vCluster nodes * **Ports**: Expose additional ports from the containers to the host * **Volumes**: Mount host directories or Docker volumes into the containers * **Environment Variables**: Set custom environment variables in the containers * **Docker Arguments**: Pass additional arguments to the Docker run command #### Multi-node setup[​](#multi-node-setup "Direct link to Multi-node setup") * **Additional Nodes**: Configure multiple worker nodes for your vCluster * **Node-Specific Settings**: Customize individual nodes with different images, ports, or volumes #### Advanced features[​](#advanced-features "Direct link to Advanced features") * **Registry Proxy**: Enable pull-through caching to reuse images from the host Docker daemon * **Load Balancer**: Configure automatic load balancer services (may require elevated privileges on some platforms) * **Custom Networks**: Specify Docker networks for inter-container communication ## Examples[​](#examples "Direct link to Examples") ### Basic vCluster with extra ports[​](#basic-vcluster-with-extra-ports "Direct link to Basic vCluster with extra ports") ``` experimental: docker: ports: - "8080:80" - "8443:443" env: - "DEBUG=true" - "LOG_LEVEL=info" ``` ### Multi-node vCluster[​](#multi-node-vcluster "Direct link to Multi-node vCluster") ``` experimental: docker: nodes: - name: "worker-1" ports: - "9090:9090" - name: "worker-2" volumes: - "/host/data:/data" env: - "NODE_ROLE=worker" ``` ### Use vCluster VPN[​](#use-vcluster-vpn "Direct link to Use vCluster VPN") ``` experimental: docker: nodes: - name: my-node privateNodes: vpn: enabled: true nodeToNode: enabled: true ``` ### Set up with volume mounts[​](#set-up-with-volume-mounts "Direct link to Set up with volume mounts") ``` experimental: docker: volumes: - "/home/user/projects:/workspace" ``` ### Disable registry proxy and load balancer[​](#disable-load-balancer "Direct link to Disable registry proxy and load balancer") ``` experimental: docker: registryProxy: enabled: false loadBalancer: enabled: false ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `docker` object [​](#docker "Direct link to docker")Docker allows you to configure Docker related settings when deploying a vCluster using Docker. ### `image` string [​](#docker-image "Direct link to docker-image")Image defines the image to use for the container. Defaults to ghcr.io/loft-sh/vm-container. ### `ports` string\[] [​](#docker-ports "Direct link to docker-ports")Ports defines extra port mappings to be added to the container. ### `volumes` string\[] [​](#docker-volumes "Direct link to docker-volumes")Volumes defines extra volumes to be added to the container. ### `env` string\[] [​](#docker-env "Direct link to docker-env")Env defines extra environment variables to be added to the container. Use key=value. ### `args` string\[] [​](#docker-args "Direct link to docker-args")Args defines extra arguments to be added to the docker run command of the container. ### `enabled` boolean [​](#docker-enabled "Direct link to docker-enabled")Enabled defines if the vCluster was deployed using Docker. This is automatically set by vCluster and should not be set by the user. ### `network` string [​](#docker-network "Direct link to docker-network")Network defines the network to use for the vCluster. If not specified, the a network will be created for the vCluster. ### `nodes` object\[] [​](#docker-nodes "Direct link to docker-nodes")Nodes defines the nodes of the vCluster. #### `image` string [​](#docker-nodes-image "Direct link to docker-nodes-image")Image defines the image to use for the container. Defaults to ghcr.io/loft-sh/vm-container. #### `ports` string\[] [​](#docker-nodes-ports "Direct link to docker-nodes-ports")Ports defines extra port mappings to be added to the container. #### `volumes` string\[] [​](#docker-nodes-volumes "Direct link to docker-nodes-volumes")Volumes defines extra volumes to be added to the container. #### `env` string\[] [​](#docker-nodes-env "Direct link to docker-nodes-env")Env defines extra environment variables to be added to the container. Use key=value. #### `args` string\[] [​](#docker-nodes-args "Direct link to docker-nodes-args")Args defines extra arguments to be added to the docker run command of the container. #### `name` string [​](#docker-nodes-name "Direct link to docker-nodes-name")Name defines the name of the node. If not specified, a random name will be generated. ### `registryProxy` object [​](#docker-registryProxy "Direct link to docker-registryProxy")Defines if docker images should be pulled from the host docker daemon. This prevents pulling images again and allows to use purely local images. Only works if containerd image storage is used. For more information, see #### `enabled` boolean true [​](#docker-registryProxy-enabled "Direct link to docker-registryProxy-enabled")Enabled defines if this option should be enabled. ### `loadBalancer` object [​](#docker-loadBalancer "Direct link to docker-loadBalancer")Defines if vCluster should configure load balancer services inside the vCluster. This might require sudo access on the host cluster for docker desktop or rancher desktop on macos. #### `enabled` boolean true [​](#docker-loadBalancer-enabled "Direct link to docker-loadBalancer-enabled")Enabled defines if this option should be enabled. #### `forwardPorts` boolean true [​](#docker-loadBalancer-forwardPorts "Direct link to docker-loadBalancer-forwardPorts")ForwardPorts defines if the load balancer ips should be made available locally via port forwarding. This will be only done if necessary for example on macos when using docker desktop. --- # Resource Proxy Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. vCluster Platform required This feature requires [vCluster Platform](https://vcluster.com/docs/platform/.md). Both the client and target tenant clusters must be managed as [VirtualClusterInstance](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance/.md) within the platform. The Resource Proxy feature enables vCluster to proxy requests for custom resources defined by CustomResourceDefinitions (CRDs) to other tenant clusters. When enabled, the client tenant cluster transparently forwards resource operations to a target tenant cluster, where the objects are stored and can be managed by controllers running there. This page covers the `vcluster.yaml` fields. For the mechanism, see [How the resource proxy works](https://vcluster.com/docs/vcluster/understand/resource-proxy.md). To set one up, see [Set up the resource proxy](https://vcluster.com/docs/vcluster/learn-how-to/resource-proxy-setup.md). For ongoing operation and monitoring, see [Operate the resource proxy](https://vcluster.com/docs/vcluster/manage/resource-proxy-operations.md). If something's broken, see [Troubleshoot the resource proxy](https://vcluster.com/docs/vcluster/troubleshoot/resource-proxy.md). ## Configuration[​](#configuration "Direct link to Configuration") To enable resource proxying, configure the `experimental.proxy.customResources` section in your `vcluster.yaml`: vcluster.yaml ``` # Basic Resource Proxy configuration # Enables group/version-wide proxying for example.com/v1 to a target cluster. # The explicit resource name is advertised by synthetic fallback discovery. experimental: proxy: customResources: myresources.example.com/v1: enabled: true targetVirtualCluster: name: "target" ``` ### Configuration options[​](#configuration-options "Direct link to Configuration options") | Field | Type | Default | Description | | ---------------------------------------------------- | -------------------------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `customResources` | `map[string]CustomResourceProxy` | `{}` | Map of resource keys to proxy configuration. | | `customResources..enabled` | `boolean` | `false` | Enable the proxy for this resource. | | `customResources..targetVirtualCluster` | `VirtualClusterRef` | - | Reference to the target [VirtualClusterInstance](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance/.md) to proxy requests to. | | `customResources..targetVirtualCluster.name` | `string` | - | Name of the target tenant cluster. Required for every entry, including disabled entries. | | `customResources..targetVirtualCluster.project` | `string` | Same as client | Project of the target tenant cluster. Defaults to the same project as the client vCluster. | | `customResources..accessResources` | `string` | `"owned"` | Resource visibility mode: `owned` or `all`. See [Access modes](#access-modes). | ### Resource key format[​](#resource-key-format "Direct link to Resource key format") The resource key follows the format `resource.apiGroup/version`: * `myresources.example.com/v1` - enables the proxy for the `example.com/v1` group and version and advertises `myresources` in fallback discovery * `otherresources.test.io/v2` - enables the proxy for the `test.io/v2` group and version and advertises `otherresources` in fallback discovery * `additionalresources.acme.org/v1alpha1` - enables the proxy for the `acme.org/v1alpha1` group and version and advertises `additionalresources` in fallback discovery Proxy registration is group/version-wide An enabled entry registers one Kubernetes APIService for the entry's entire API group and version. While the target is reachable, the proxy forwards discovery for that group and version to the target and can route requests for every resource the target advertises. The resource name in an explicit key doesn't restrict live requests to that resource. Use RBAC in the target tenant cluster to restrict the resource kinds and operations available to a client. All enabled entries for the same group and version must use the same target and access mode. For entries that share a group and version, write equivalent settings identically. Validation compares the configured values before applying defaults. Either omit `accessResources` from every entry or set the same explicit value on every entry. Likewise, either omit `targetVirtualCluster.project` from every entry or set the same explicit project on every entry. Mixing an omitted value with its explicit default is rejected. #### Wildcard group and version entry[​](#wildcard-kind-matching "Direct link to Wildcard group and version entry") You can use `*` in place of an explicit resource name: * `*.example.com/v1` - enables the proxy for the `example.com/v1` group and version * `*.test.io/v2` - enables the proxy for the `test.io/v2` group and version Wildcard and explicit entries have the same group/version-wide behavior while the target is reachable. They differ when the target is unavailable. Explicit entries give the proxy resource names for its synthetic fallback discovery document, while a wildcard doesn't. Prefer explicit resource keys over wildcards where you can, and list each resource that clients depend on during an outage. With explicit keys such as `myresources.example.com/v1`, fallback discovery continues to advertise the configured resource names when the target is briefly unreachable. The fallback is synthetic. It reports resources as namespaced, derives singular names and kinds mechanically, and advertises only the `get` verb. Treat only the configured resource names as dependable. With `*.example.com/v1`, discovery degrades to an empty list, which some controllers and CLIs handle poorly. See [Discovery lists resources with only a get verb, or an empty list](https://vcluster.com/docs/vcluster/troubleshoot/resource-proxy.md#discovery-fallback). Wildcard and explicit entries are mutually exclusive A wildcard key (for example `*.example.com/v1`) and an explicit key (for example `myresources.example.com/v1`) cannot both be enabled for the same group and version. vCluster rejects that combination at startup. A disabled explicit entry (`enabled: false`) does not conflict with a wildcard for the same group and version. For worked examples of this configuration, including multi-target and cross-project setups, see [Set up the resource proxy](https://vcluster.com/docs/vcluster/learn-how-to/resource-proxy-setup.md). ## Access modes[​](#access-modes "Direct link to Access modes") The `accessResources` field controls which resources a client tenant cluster can see through the proxied API group and version: Access modes configuration ``` # Access modes configuration examples # Example 1: "owned" mode (default) # The cluster only sees objects with its ownership labels experimental: proxy: customResources: myresources.example.com/v1: enabled: true targetVirtualCluster: name: "target" # accessResources defaults to "owned" - only see resources we created --- # Example 2: "all" mode # The proxy doesn't filter by owner; target RBAC still applies experimental: proxy: customResources: myresources.example.com/v1: enabled: true targetVirtualCluster: name: "target" accessResources: all # Disable ownership filtering ``` ### `owned` mode - default[​](#owned-mode "Direct link to owned-mode") The proxy presents each client tenant cluster only objects with matching ownership labels. It normally adds those labels to objects the client creates. This enables tenant isolation where multiple client tenant clusters can share a target without seeing each other's resources. Stay on `accessResources: owned` unless you have a reason not to. Each client sees and manages only objects with its ownership labels. The proxy normally adds those labels when the client writes an object. Lists and watches are filtered by an injected label selector, and named GET, PUT, PATCH, and DELETE requests for objects it doesn't own return 404. This is visibility isolation, not name allocation. Two clients creating the same object name in the same target namespace still collide, and the second create normally returns 409 from the target. Give each client its own namespace or its own name prefix. ### `all` mode[​](#all-mode "Direct link to all-mode") The proxy doesn't filter objects by ownership. Target RBAC still determines which resource kinds and operations the client can access. This is useful for read-only observers, centralized dashboards, or admin access. Use `all` only when the client must manage objects it didn't create, and treat that client as privileged. The mode removes ownership filtering, so target RBAC can permit the client to read, mutate, and delete objects owned by other tenant clusters. Combine access modes with RBAC The `accessResources` mode controls **visibility** - what resources a tenant cluster can see. RBAC in the target tenant cluster controls **permissions** - what operations the tenant cluster can perform. For example, a tenant cluster with `accessResources: all` but read-only RBAC can see all resources that its target RBAC permits but can't modify any. For a worked multi-client example, see [Set up the resource proxy](https://vcluster.com/docs/vcluster/learn-how-to/resource-proxy-setup.md#multi-client). ## Limitations[​](#limitations "Direct link to Limitations") * All `CustomResources` within the same API `GroupVersion` must use the same target tenant cluster and access mode. Validation enforces that all entries for a group and version agree on `targetVirtualCluster` and `accessResources`. Split API groups if two targets must serve sibling resources. * An enabled API group can't appear in both `experimental.proxy.customResources` and either `sync.toHost.customResources` or `sync.fromHost.customResources`. Validation checks the entire API group, regardless of resource name or version. Disabled entries don't conflict. * Proxy routing is group/version-wide. An explicit resource key doesn't prevent access to sibling resources that the target advertises in that group and version. Restrict sibling resources with target RBAC. * In `owned` mode, named PUT, PATCH, and DELETE requests require target RBAC permission to `get` the resource as well as permission for the requested mutation. The proxy performs a GET first to verify ownership. If that GET is forbidden, the mutation is masked as a 404. * Connection upgrade verbs such as `exec`, `attach`, and `port-forward` aren't supported through the proxy. See [Connection upgrades return 501](https://vcluster.com/docs/vcluster/troubleshoot/resource-proxy.md#connection-upgrades). * When configuring RBAC for status updates, include permissions for the status subresource. ## Find content that moved off this page[​](#moved-content "Direct link to Find content that moved off this page") This page used to include the full explanation, setup walkthrough, monitoring guide, and troubleshooting steps for the resource proxy. That content moved to dedicated pages, so a bookmarked link to a section below may land here instead of on the section itself. Here's where things went: * **How it works, key capabilities, connection path, and peer-health thresholds** → [How the resource proxy works](https://vcluster.com/docs/vcluster/understand/resource-proxy.md) - **Platform RBAC and the setup examples** → [Set up the resource proxy](https://vcluster.com/docs/vcluster/learn-how-to/resource-proxy-setup.md) * **DERP relay tuning and the monitoring guide** → [Operate the resource proxy](https://vcluster.com/docs/vcluster/manage/resource-proxy-operations.md) - **Troubleshooting steps and the log reference** → [Troubleshoot the resource proxy](https://vcluster.com/docs/vcluster/troubleshoot/resource-proxy.md) * **The typed config reference tables** → merged into [Configuration options](#configuration-options) above - **Best practices** → distributed into the sections above: explicit resource keys under [Wildcard group and version entry](#wildcard-kind-matching), staying on `owned` mode under [Access modes](#access-modes), and the rest into [Operate the resource proxy](https://vcluster.com/docs/vcluster/manage/resource-proxy-operations.md) and [Troubleshoot the resource proxy](https://vcluster.com/docs/vcluster/troubleshoot/resource-proxy.md) --- # Experimental sync settings Experimental Feature This feature is experimental. Breaking changes may occur between releases. It may become Enterprise-only or be deprecated. Not recommended for production use. Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** ## Feature matrix[​](#feature-matrix "Direct link to Feature matrix") | Feature | Open Source | Pro | Notes | | ------------------------- | ----------- | --- | ----------------------- | | setOwner | ✅ | ✅ | Replaces `--set-owner`. | | hostMetricsBindAddress | ✅ | ✅ | | | virtualMetricsBindAddress | ✅ | ✅ | | ## Config reference[​](#config-reference "Direct link to Config reference") ## `syncSettings` object [​](#syncSettings "Direct link to syncSettings")SyncSettings are advanced settings for the syncer controller. ### `setOwner` boolean true [​](#syncSettings-setOwner "Direct link to syncSettings-setOwner")SetOwner specifies if vCluster should set an owner reference on the synced objects to the vCluster service. This allows for easy garbage collection. ### `hostMetricsBindAddress` string [​](#syncSettings-hostMetricsBindAddress "Direct link to syncSettings-hostMetricsBindAddress")HostMetricsBindAddress is the bind address for the local manager ### `virtualMetricsBindAddress` string [​](#syncSettings-virtualMetricsBindAddress "Direct link to syncSettings-virtualMetricsBindAddress")VirtualMetricsBindAddress is the bind address for the virtual manager --- # Export kubeconfig Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Customize how vCluster exports the kubeconfig file to a secret for use in tools like ArgoCD, Flux, or Terraform pipelines. By default, vCluster stores the kubeconfig in a secret named `vc-NAME` within the namespace where vCluster is deployed. You can [customize this default secret](#customize-the-default-secret). In addition to the default secret, you can also [export additional kubeconfig secrets](#export-additional-secrets). ## Customize the default secret[​](#customize-the-default-secret "Direct link to Customize the default secret") The following example configures a tenant cluster to use a specific kubeconfig context and server endpoint for the default kubeconfig secret `vc-NAME`: * Set the kubeconfig context name to `my-domain-context`. * Configure the kubeconfig to use `https://my-domain.org:443` as the endpoint for the exposed tenant cluster. ``` exportKubeConfig: context: my-domain-context server: https://my-domain.org:443 ``` Customizing the default secret does not affect the additional kubeconfig secrets that are configured separately with the `exportKubeConfig.additionalSecrets` field. E.g. in the above example, custom context and server values are used only for the default secret `vc-NAME` and do not affect the additional secrets that you may have configured. note The default secret and the secret specified with the deprecated `exportKubeConfig.secret` field are customized with the same set of fields (`exportKubeConfig.context`, `exportKubeConfig.server`, etc), so setting these fields customizes both the default and the deprecated secret in the same way. More details in the [migration section below](#migrate-custom-deprecated). ## Export additional secrets[​](#export-additional-secrets "Direct link to Export additional secrets") You can export additional kubeconfig secrets by specifying `exportKubeConfig.additionalSecrets`. ### Using the default namespace for the additional secret[​](#using-the-default-namespace-for-the-additional-secret "Direct link to Using the default namespace for the additional secret") The following example configures a tenant cluster to export an additional secret `vc-my-domain`, which is created in the namespace where vCluster is deployed. In this example, the additional secret `vc-my-domain` is customized in the following way: * Set the kubeconfig context name to `my-domain-context`. * Configure the kubeconfig to use `https://my-domain.org:443` as the server endpoint for the exposed tenant cluster. ``` exportKubeConfig: additionalSecrets: - name: vc-my-domain context: my-domain-context server: https://my-domain.org:443 ``` You can create multiple additional secrets. Each additional secret is configured separately, and configuring additional secrets does not affect the default kubeconfig secret. E.g. in the above example, custom context and server values are used only for the additional secret `vc-my-domain` and do not affect the default kubeconfig secret, nor any other additional secret that you may have configured. ### Using a new namespace for the additional secret[​](#using-a-new-namespace-for-the-additional-secret "Direct link to Using a new namespace for the additional secret") The following example configures a tenant cluster to store the kubeconfig secret in a separate namespace while maintaining proper access control: * Set the kubeconfig context name to `my-domain-context`. * Configure the kubeconfig to use `https://my-domain.org:443` as the tenant cluster endpoint. * Create a namespace called `kubeconfig-secret-namespace` to store the secret. * Name the secret `vc-my-domain` instead of using the default "vc-NAME" (where `NAME` is the vCluster name). * Additionally, grant access to the new namespace by creating a `Role` and `RoleBinding` for the vCluster service account: * Configure the new role with the same permissions assigned to the vCluster app role in the original namespace. * If needed, copy permissions from the default vCluster deployment. * When vCluster permissions change, update the unmanaged role in the target namespace accordingly. ``` exportKubeConfig: additionalSecrets: - namespace: kubeconfig-secret-namespace name: vc-my-domain context: my-domain-context server: https://my-domain.org:443 ``` ### Deprecated `exportKubeConfig.secret` config[​](#deprecated-exportkubeconfigsecret-config "Direct link to deprecated-exportkubeconfigsecret-config") Starting from version 0.24.0, the `exportKubeConfig.secret` configuration is deprecated and replaced by `exportKubeConfig.additionalSecrets`. #### Migrate a deprecated secret to new additional secrets[​](#migrate-a-deprecated-secret-to-new-additional-secrets "Direct link to Migrate a deprecated secret to new additional secrets") It is recommended to migrate the deprecated `exportKubeConfig.secret` config to the new `exportKubeConfig.additionalSecrets`. If `my-secret` is configured such as: ``` exportKubeConfig: secret: name: my-secret ``` You can migrate the secret to the new additional secret with the following: ``` exportKubeConfig: additionalSecrets: - name: my-secret ``` #### Migrate a customized deprecated secret to new additional secrets[​](#migrate-custom-deprecated "Direct link to Migrate a customized deprecated secret to new additional secrets") The secret defined using the deprecated `exportKubeConfig.secret` field is customized using the same configuration fields (such as `exportKubeConfig.context`, `exportKubeConfig.server`) as the default kubeconfig secret. For example, in the following configuration, vCluster exports a kubeconfig secret named `vc-my-domain` with: * Context: `my-domain-context` * Server endpoint: `https://my-domain.org:443` vCluster also exports the default kubeconfig secret using these same context and server values. ``` exportKubeConfig: context: my-domain-context server: https://my-domain.org:443 secret: name: vc-my-domain ``` When migrating this customized deprecated secret to the new additional secrets, you have the following two options. 1\. Customize both the default and the additional kubeconfig secret. ``` exportKubeConfig: context: my-domain-context server: https://my-domain.org:443 additionalSecrets: - name: vc-my-domain context: my-domain-context server: https://my-domain.org:443 ``` In this example, both the default secret `vc-NAME` and the additional secret `vc-my-domain` use the same custom values for the context and the server, which gives you the same exported kubeconfig secrets as with the old config that used the deprecated `exportKubeConfig.secret`. You might want to export the additional secret with different config values for the context and the server. You can achieve this by customizing the additional secret differently than the default secret: ``` exportKubeConfig: context: my-domain-context server: https://my-domain.org:443 additionalSecrets: - name: vc-my-domain context: my-other-domain-context server: https://my-other-domain.org:443 ``` 2\. Customize only the additional kubeconfig secret. ``` exportKubeConfig: additionalSecrets: - name: vc-my-domain context: my-domain-context server: https://my-domain.org:443 ``` In this example, only the additional secret `vc-my-domain` uses custom values for the context and the server. ## Config reference[​](#config-reference "Direct link to Config reference") ## `exportKubeConfig` object [​](#exportKubeConfig "Direct link to exportKubeConfig")ExportKubeConfig describes how vCluster should export the vCluster kubeConfig file. ### `context` string [​](#exportKubeConfig-context "Direct link to exportKubeConfig-context")Context is the name of the context within the generated kubeconfig to use. ### `server` string [​](#exportKubeConfig-server "Direct link to exportKubeConfig-server")Override the default and specify a custom hostname for the generated kubeconfig. ### `insecure` boolean false [​](#exportKubeConfig-insecure "Direct link to exportKubeConfig-insecure")If tls should get skipped for the server ### `serviceAccount` object [​](#exportKubeConfig-serviceAccount "Direct link to exportKubeConfig-serviceAccount")ServiceAccount can be used to generate a service account token instead of the default certificates. #### `name` string [​](#exportKubeConfig-serviceAccount-name "Direct link to exportKubeConfig-serviceAccount-name")Name of the service account to be used to generate a service account token instead of the default certificates. #### `namespace` string [​](#exportKubeConfig-serviceAccount-namespace "Direct link to exportKubeConfig-serviceAccount-namespace")Namespace of the service account to be used to generate a service account token instead of the default certificates. If omitted, will use the kube-system namespace. #### `clusterRole` string [​](#exportKubeConfig-serviceAccount-clusterRole "Direct link to exportKubeConfig-serviceAccount-clusterRole")ClusterRole to assign to the service account. ### `secret` object [​](#exportKubeConfig-secret "Direct link to exportKubeConfig-secret")Declare in which host cluster secret vCluster should store the generated virtual cluster kubeconfig. If this is not defined, vCluster will create it with `vc-NAME`. If you specify another name, vCluster creates the config in this other secret.Deprecated: Use AdditionalSecrets instead. #### `name` string [​](#exportKubeConfig-secret-name "Direct link to exportKubeConfig-secret-name")Name is the name of the secret where the kubeconfig should get stored. #### `namespace` string [​](#exportKubeConfig-secret-namespace "Direct link to exportKubeConfig-secret-namespace")Namespace where vCluster should store the kubeconfig secret. If this is not equal to the namespace where you deployed vCluster, you need to make sure vCluster has access to this other namespace. ### `additionalSecrets` object\[] [​](#exportKubeConfig-additionalSecrets "Direct link to exportKubeConfig-additionalSecrets")AdditionalSecrets specifies the additional host cluster secrets in which vCluster will store the generated virtual cluster kubeconfigs. #### `context` string [​](#exportKubeConfig-additionalSecrets-context "Direct link to exportKubeConfig-additionalSecrets-context")Context is the name of the context within the generated kubeconfig to use. #### `server` string [​](#exportKubeConfig-additionalSecrets-server "Direct link to exportKubeConfig-additionalSecrets-server")Override the default and specify a custom hostname for the generated kubeconfig. #### `insecure` boolean [​](#exportKubeConfig-additionalSecrets-insecure "Direct link to exportKubeConfig-additionalSecrets-insecure")If tls should get skipped for the server #### `serviceAccount` object [​](#exportKubeConfig-additionalSecrets-serviceAccount "Direct link to exportKubeConfig-additionalSecrets-serviceAccount")ServiceAccount can be used to generate a service account token instead of the default certificates. ##### `name` string [​](#exportKubeConfig-additionalSecrets-serviceAccount-name "Direct link to exportKubeConfig-additionalSecrets-serviceAccount-name")Name of the service account to be used to generate a service account token instead of the default certificates. ##### `namespace` string [​](#exportKubeConfig-additionalSecrets-serviceAccount-namespace "Direct link to exportKubeConfig-additionalSecrets-serviceAccount-namespace")Namespace of the service account to be used to generate a service account token instead of the default certificates. If omitted, will use the kube-system namespace. ##### `clusterRole` string [​](#exportKubeConfig-additionalSecrets-serviceAccount-clusterRole "Direct link to exportKubeConfig-additionalSecrets-serviceAccount-clusterRole")ClusterRole to assign to the service account. #### `name` string [​](#exportKubeConfig-additionalSecrets-name "Direct link to exportKubeConfig-additionalSecrets-name")Name is the name of the secret where the kubeconfig is stored. #### `namespace` string [​](#exportKubeConfig-additionalSecrets-namespace "Direct link to exportKubeConfig-additionalSecrets-namespace")Namespace where vCluster stores the kubeconfig secret. If this is not equal to the namespace where you deployed vCluster, you need to make sure vCluster has access to this other namespace. --- # Integrations Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** ## Config reference[​](#config-reference "Direct link to Config reference") ## `integrations` object [​](#integrations "Direct link to integrations")Integrations holds config for vCluster integrations with other operators or tools running on the host cluster ### `metricsServer` object [​](#integrations-metricsServer "Direct link to integrations-metricsServer")MetricsServer reuses the metrics server from the host cluster within the vCluster. #### `enabled` boolean false [​](#integrations-metricsServer-enabled "Direct link to integrations-metricsServer-enabled")Enabled signals the metrics server integration should be enabled. #### `apiService` object [​](#integrations-metricsServer-apiService "Direct link to integrations-metricsServer-apiService")APIService holds information about where to find the metrics-server service. Defaults to metrics-server/kube-system. ##### `service` object [​](#integrations-metricsServer-apiService-service "Direct link to integrations-metricsServer-apiService-service")Service is a reference to the service for the API server. ##### `name` string [​](#integrations-metricsServer-apiService-service-name "Direct link to integrations-metricsServer-apiService-service-name")Name is the name of the host service of the apiservice. ##### `namespace` string [​](#integrations-metricsServer-apiService-service-namespace "Direct link to integrations-metricsServer-apiService-service-namespace")Namespace is the name of the host service of the apiservice. ##### `port` integer [​](#integrations-metricsServer-apiService-service-port "Direct link to integrations-metricsServer-apiService-service-port")Port is the target port on the host service to connect to. #### `nodes` boolean true [​](#integrations-metricsServer-nodes "Direct link to integrations-metricsServer-nodes")Nodes defines if metrics-server nodes api should get proxied from host to virtual cluster. #### `pods` boolean true [​](#integrations-metricsServer-pods "Direct link to integrations-metricsServer-pods")Pods defines if metrics-server pods api should get proxied from host to virtual cluster. ### `kubeVirt` object [​](#integrations-kubeVirt "Direct link to integrations-kubeVirt")KubeVirt reuses a host kubevirt and makes certain CRDs from it available inside the vCluster #### `enabled` boolean false [​](#integrations-kubeVirt-enabled "Direct link to integrations-kubeVirt-enabled")Enabled signals if the integration should be enabled #### `apiService` object [​](#integrations-kubeVirt-apiService "Direct link to integrations-kubeVirt-apiService")APIService holds information about where to find the virt-api service. Defaults to virt-api/kubevirt. ##### `service` object [​](#integrations-kubeVirt-apiService-service "Direct link to integrations-kubeVirt-apiService-service")Service is a reference to the service for the API server. ##### `name` string [​](#integrations-kubeVirt-apiService-service-name "Direct link to integrations-kubeVirt-apiService-service-name")Name is the name of the host service of the apiservice. ##### `namespace` string [​](#integrations-kubeVirt-apiService-service-namespace "Direct link to integrations-kubeVirt-apiService-service-namespace")Namespace is the name of the host service of the apiservice. ##### `port` integer [​](#integrations-kubeVirt-apiService-service-port "Direct link to integrations-kubeVirt-apiService-service-port")Port is the target port on the host service to connect to. #### `webhook` object [​](#integrations-kubeVirt-webhook "Direct link to integrations-kubeVirt-webhook")Webhook holds configuration for enabling the webhook within the vCluster ##### `enabled` boolean true [​](#integrations-kubeVirt-webhook-enabled "Direct link to integrations-kubeVirt-webhook-enabled")Enabled defines if this option should be enabled. #### `sync` object [​](#integrations-kubeVirt-sync "Direct link to integrations-kubeVirt-sync")Sync holds configuration on what resources to sync ##### `dataVolumes` object [​](#integrations-kubeVirt-sync-dataVolumes "Direct link to integrations-kubeVirt-sync-dataVolumes")If DataVolumes should get synced ##### `enabled` boolean false [​](#integrations-kubeVirt-sync-dataVolumes-enabled "Direct link to integrations-kubeVirt-sync-dataVolumes-enabled")Enabled defines if this option should be enabled. ##### `virtualMachineInstanceMigrations` object [​](#integrations-kubeVirt-sync-virtualMachineInstanceMigrations "Direct link to integrations-kubeVirt-sync-virtualMachineInstanceMigrations")If VirtualMachineInstanceMigrations should get synced ##### `enabled` boolean true [​](#integrations-kubeVirt-sync-virtualMachineInstanceMigrations-enabled "Direct link to integrations-kubeVirt-sync-virtualMachineInstanceMigrations-enabled")Enabled defines if this option should be enabled. ##### `virtualMachineInstances` object [​](#integrations-kubeVirt-sync-virtualMachineInstances "Direct link to integrations-kubeVirt-sync-virtualMachineInstances")If VirtualMachineInstances should get synced ##### `enabled` boolean true [​](#integrations-kubeVirt-sync-virtualMachineInstances-enabled "Direct link to integrations-kubeVirt-sync-virtualMachineInstances-enabled")Enabled defines if this option should be enabled. ##### `virtualMachines` object [​](#integrations-kubeVirt-sync-virtualMachines "Direct link to integrations-kubeVirt-sync-virtualMachines")If VirtualMachines should get synced ##### `enabled` boolean true [​](#integrations-kubeVirt-sync-virtualMachines-enabled "Direct link to integrations-kubeVirt-sync-virtualMachines-enabled")Enabled defines if this option should be enabled. ##### `virtualMachineClones` object [​](#integrations-kubeVirt-sync-virtualMachineClones "Direct link to integrations-kubeVirt-sync-virtualMachineClones")If VirtualMachineClones should get synced ##### `enabled` boolean true [​](#integrations-kubeVirt-sync-virtualMachineClones-enabled "Direct link to integrations-kubeVirt-sync-virtualMachineClones-enabled")Enabled defines if this option should be enabled. ##### `virtualMachinePools` object [​](#integrations-kubeVirt-sync-virtualMachinePools "Direct link to integrations-kubeVirt-sync-virtualMachinePools")If VirtualMachinePools should get synced ##### `enabled` boolean true [​](#integrations-kubeVirt-sync-virtualMachinePools-enabled "Direct link to integrations-kubeVirt-sync-virtualMachinePools-enabled")Enabled defines if this option should be enabled. ### `externalSecrets` object [​](#integrations-externalSecrets "Direct link to integrations-externalSecrets")ExternalSecrets reuses a host external secret operator and makes certain CRDs from it available inside the vCluster.- ExternalSecrets will be synced from the virtual cluster to the host cluster. - SecretStores will be synced from the virtual cluster to the host cluster and then bi-directionally. - ClusterSecretStores will be synced from the host cluster to the virtual cluster. #### `enabled` boolean false [​](#integrations-externalSecrets-enabled "Direct link to integrations-externalSecrets-enabled")Enabled defines whether the external secret integration is enabled or not #### `version` string [​](#integrations-externalSecrets-version "Direct link to integrations-externalSecrets-version")Version defines the version of the external secrets operator to use. If empty, the storage version will be used. #### `webhook` object [​](#integrations-externalSecrets-webhook "Direct link to integrations-externalSecrets-webhook")Webhook defines whether the host webhooks are reused or not ##### `enabled` boolean false [​](#integrations-externalSecrets-webhook-enabled "Direct link to integrations-externalSecrets-webhook-enabled")Enabled defines if this option should be enabled. #### `sync` object [​](#integrations-externalSecrets-sync "Direct link to integrations-externalSecrets-sync")Sync defines the syncing behavior for the integration ##### `toHost` object [​](#integrations-externalSecrets-sync-toHost "Direct link to integrations-externalSecrets-sync-toHost")ToHost defines what resources are synced from the virtual cluster to the host ##### `externalSecrets` object [​](#integrations-externalSecrets-sync-toHost-externalSecrets "Direct link to integrations-externalSecrets-sync-toHost-externalSecrets")ExternalSecrets allows to configure if only a subset of ExternalSecrets matching a label selector should get synced from the virtual cluster to the host cluster. ##### `selector` object [​](#integrations-externalSecrets-sync-toHost-externalSecrets-selector "Direct link to integrations-externalSecrets-sync-toHost-externalSecrets-selector") ##### `matchLabels` object {} [​](#integrations-externalSecrets-sync-toHost-externalSecrets-selector-matchLabels "Direct link to integrations-externalSecrets-sync-toHost-externalSecrets-selector-matchLabels") ##### `matchExpressions` object\[] [​](#integrations-externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions "Direct link to integrations-externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions") ##### `key` string [​](#integrations-externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-key "Direct link to integrations-externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-key") ##### `operator` string [​](#integrations-externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-operator "Direct link to integrations-externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-operator") ##### `values` string\[] [​](#integrations-externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-values "Direct link to integrations-externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-values") ##### `stores` object [​](#integrations-externalSecrets-sync-toHost-stores "Direct link to integrations-externalSecrets-sync-toHost-stores")Stores defines if secret stores should get synced from the virtual cluster to the host cluster and then bi-directionally. ##### `selector` object [​](#integrations-externalSecrets-sync-toHost-stores-selector "Direct link to integrations-externalSecrets-sync-toHost-stores-selector") ##### `matchLabels` object {} [​](#integrations-externalSecrets-sync-toHost-stores-selector-matchLabels "Direct link to integrations-externalSecrets-sync-toHost-stores-selector-matchLabels") ##### `matchExpressions` object\[] [​](#integrations-externalSecrets-sync-toHost-stores-selector-matchExpressions "Direct link to integrations-externalSecrets-sync-toHost-stores-selector-matchExpressions") ##### `key` string [​](#integrations-externalSecrets-sync-toHost-stores-selector-matchExpressions-key "Direct link to integrations-externalSecrets-sync-toHost-stores-selector-matchExpressions-key") ##### `operator` string [​](#integrations-externalSecrets-sync-toHost-stores-selector-matchExpressions-operator "Direct link to integrations-externalSecrets-sync-toHost-stores-selector-matchExpressions-operator") ##### `values` string\[] [​](#integrations-externalSecrets-sync-toHost-stores-selector-matchExpressions-values "Direct link to integrations-externalSecrets-sync-toHost-stores-selector-matchExpressions-values") ##### `enabled` boolean false [​](#integrations-externalSecrets-sync-toHost-stores-enabled "Direct link to integrations-externalSecrets-sync-toHost-stores-enabled")Enabled defines if this option should be enabled. ##### `fromHost` object [​](#integrations-externalSecrets-sync-fromHost "Direct link to integrations-externalSecrets-sync-fromHost")FromHost defines what resources are synced from the host cluster to the virtual cluster ##### `clusterStores` object [​](#integrations-externalSecrets-sync-fromHost-clusterStores "Direct link to integrations-externalSecrets-sync-fromHost-clusterStores")ClusterStores defines if cluster secrets stores should get synced from the host cluster to the virtual cluster. ##### `selector` object [​](#integrations-externalSecrets-sync-fromHost-clusterStores-selector "Direct link to integrations-externalSecrets-sync-fromHost-clusterStores-selector") ##### `matchLabels` object {} [​](#integrations-externalSecrets-sync-fromHost-clusterStores-selector-matchLabels "Direct link to integrations-externalSecrets-sync-fromHost-clusterStores-selector-matchLabels") ##### `matchExpressions` object\[] [​](#integrations-externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions "Direct link to integrations-externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions") ##### `key` string [​](#integrations-externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-key "Direct link to integrations-externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-key") ##### `operator` string [​](#integrations-externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-operator "Direct link to integrations-externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-operator") ##### `values` string\[] [​](#integrations-externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-values "Direct link to integrations-externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-values") ##### `enabled` boolean false [​](#integrations-externalSecrets-sync-fromHost-clusterStores-enabled "Direct link to integrations-externalSecrets-sync-fromHost-clusterStores-enabled")Enabled defines if this option should be enabled. ### `certManager` object [​](#integrations-certManager "Direct link to integrations-certManager")CertManager reuses a host cert-manager and makes its CRDs from it available inside the vCluster.- Certificates and Issuers will be synced from the virtual cluster to the host cluster. - ClusterIssuers will be synced from the host cluster to the virtual cluster. #### `enabled` boolean false [​](#integrations-certManager-enabled "Direct link to integrations-certManager-enabled")Enabled defines if this option should be enabled. #### `sync` object [​](#integrations-certManager-sync "Direct link to integrations-certManager-sync")Sync contains advanced configuration for syncing cert-manager resources. ##### `toHost` object [​](#integrations-certManager-sync-toHost "Direct link to integrations-certManager-sync-toHost") ##### `certificates` object [​](#integrations-certManager-sync-toHost-certificates "Direct link to integrations-certManager-sync-toHost-certificates")Certificates defines if certificates should get synced from the virtual cluster to the host cluster. ##### `enabled` boolean true [​](#integrations-certManager-sync-toHost-certificates-enabled "Direct link to integrations-certManager-sync-toHost-certificates-enabled")Enabled defines if this option should be enabled. ##### `issuers` object [​](#integrations-certManager-sync-toHost-issuers "Direct link to integrations-certManager-sync-toHost-issuers")Issuers defines if issuers should get synced from the virtual cluster to the host cluster. ##### `enabled` boolean true [​](#integrations-certManager-sync-toHost-issuers-enabled "Direct link to integrations-certManager-sync-toHost-issuers-enabled")Enabled defines if this option should be enabled. ##### `fromHost` object [​](#integrations-certManager-sync-fromHost "Direct link to integrations-certManager-sync-fromHost") ##### `clusterIssuers` object [​](#integrations-certManager-sync-fromHost-clusterIssuers "Direct link to integrations-certManager-sync-fromHost-clusterIssuers")ClusterIssuers defines if (and which) cluster issuers should get synced from the host cluster to the virtual cluster. ##### `enabled` boolean true [​](#integrations-certManager-sync-fromHost-clusterIssuers-enabled "Direct link to integrations-certManager-sync-fromHost-clusterIssuers-enabled")Enabled defines if this option should be enabled. ##### `selector` object [​](#integrations-certManager-sync-fromHost-clusterIssuers-selector "Direct link to integrations-certManager-sync-fromHost-clusterIssuers-selector")Selector defines what cluster issuers should be imported. ##### `labels` object {} [​](#integrations-certManager-sync-fromHost-clusterIssuers-selector-labels "Direct link to integrations-certManager-sync-fromHost-clusterIssuers-selector-labels")Labels defines what labels should be looked for ### `istio` object [​](#integrations-istio "Direct link to integrations-istio")Istio syncs DestinationRules, Gateways and VirtualServices from virtual cluster to the host. #### `enabled` boolean false [​](#integrations-istio-enabled "Direct link to integrations-istio-enabled")Enabled defines if this option should be enabled. #### `sync` object [​](#integrations-istio-sync "Direct link to integrations-istio-sync") ##### `toHost` object [​](#integrations-istio-sync-toHost "Direct link to integrations-istio-sync-toHost") ##### `destinationRules` object [​](#integrations-istio-sync-toHost-destinationRules "Direct link to integrations-istio-sync-toHost-destinationRules") ##### `enabled` boolean true [​](#integrations-istio-sync-toHost-destinationRules-enabled "Direct link to integrations-istio-sync-toHost-destinationRules-enabled")Enabled defines if this option should be enabled. ##### `gateways` object [​](#integrations-istio-sync-toHost-gateways "Direct link to integrations-istio-sync-toHost-gateways") ##### `enabled` boolean true [​](#integrations-istio-sync-toHost-gateways-enabled "Direct link to integrations-istio-sync-toHost-gateways-enabled")Enabled defines if this option should be enabled. ##### `virtualServices` object [​](#integrations-istio-sync-toHost-virtualServices "Direct link to integrations-istio-sync-toHost-virtualServices") ##### `enabled` boolean true [​](#integrations-istio-sync-toHost-virtualServices-enabled "Direct link to integrations-istio-sync-toHost-virtualServices-enabled")Enabled defines if this option should be enabled. ### `netris` object [​](#integrations-netris "Direct link to integrations-netris")Netris integration helps configuring netris networking for vCluster. #### `enabled` boolean [​](#integrations-netris-enabled "Direct link to integrations-netris-enabled")Enabled defines if netris integration is enabled #### `connector` string [​](#integrations-netris-connector "Direct link to integrations-netris-connector")Connector specifies the netris connector name #### `kubeVip` object [​](#integrations-netris-kubeVip "Direct link to integrations-netris-kubeVip")KubeVip holds kube-vip configuration for netris ##### `serverCluster` string [​](#integrations-netris-kubeVip-serverCluster "Direct link to integrations-netris-kubeVip-serverCluster")ServerCluster specifies the server cluster name ##### `bridge` string [​](#integrations-netris-kubeVip-bridge "Direct link to integrations-netris-kubeVip-bridge")Bridge specifies the bridge interface name ##### `ipRange` string [​](#integrations-netris-kubeVip-ipRange "Direct link to integrations-netris-kubeVip-ipRange")IPRange specifies the IP range for kube-vip ### `argoCD` object [​](#integrations-argoCD "Direct link to integrations-argoCD")ArgoCD integration helps configuring ArgoCD for vCluster. #### `enabled` boolean [​](#integrations-argoCD-enabled "Direct link to integrations-argoCD-enabled")Enabled defines if argo cd integration is enabled #### `connector` string [​](#integrations-argoCD-connector "Direct link to integrations-argoCD-connector")Connector specifies the argo cd connector name #### `cluster` object [​](#integrations-argoCD-cluster "Direct link to integrations-argoCD-cluster")Cluster holds settings for the cluster entry that is registered in argo cd ##### `metadata` object [​](#integrations-argoCD-cluster-metadata "Direct link to integrations-argoCD-cluster-metadata")Metadata is set on the argo cd cluster, so an ApplicationSet cluster generator can select it. On Akuity, a key removed here stays on the cluster until it is removed in Akuity. ##### `labels` object [​](#integrations-argoCD-cluster-metadata-labels "Direct link to integrations-argoCD-cluster-metadata-labels")Labels are set on the argo cd cluster ##### `annotations` object [​](#integrations-argoCD-cluster-metadata-annotations "Direct link to integrations-argoCD-cluster-metadata-annotations")Annotations are set on the argo cd cluster --- # cert-manager | | | Enterprise | | | | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Cert Manager Integration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/cert-manager.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. You can integrate cert-manager with your tenant cluster to manage TLS certificates across control plane and tenant environments. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Run the command `kubectl auth can-i create clusterrole -A` to verify that your current kube-context has administrative privileges. info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm`: Helm v3.10 is required for deploying the platform. Refer to the [Helm Installation Guide](https://Helm.sh/docs/intro/install/) if you need to install it. * `kubectl`: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. - cert-manager operator installed on your control plane cluster. For setup instructions, see the [cert-manager installation guide](https://cert-manager.io/docs/installation/). ## Enable the integration[​](#enable-the-integration "Direct link to Enable the integration") Enable cert-manager integration in your tenant cluster configuration: Enable cert-manager integration ``` integrations: certManager: enabled: true ``` This configuration enables the integration and configures automatic resource sync: * **Imports**: Cluster-scoped `ClusterIssuers` from your control plane cluster into the tenant cluster * **Exports**: Namespaced `Issuers` and `Certificates` from the tenant cluster to the control plane cluster * **Syncs**: Generated certificate secrets back to the tenant cluster automatically ## Set up cluster contexts[​](#set-up-cluster-contexts "Direct link to Set up cluster contexts") Create or update a tenant cluster following the [vCluster quick start guide](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md). Configure cluster contexts to simplify switching between control plane and tenant clusters: ``` export HOST_CTX="your-host-context" export VCLUSTER_CTX="vcluster-ctx" ``` Find your contexts Run `kubectl config get-contexts` to list available contexts. ## Certificate issuer configurations[​](#certificate-issuer-configurations "Direct link to Certificate issuer configurations") Choose the appropriate certificate issuer based on your security and operational requirements. Self-signed certificates for development Self-signed certificates work well for development environments and internal services that don't require public trust. 1. Tenant Cluster Create a self-signed issuer self-signed-issuer.yaml ``` apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: selfsigned-issuer namespace: default spec: selfSigned: {} ``` Apply the issuer: Apply self-signed issuer ``` kubectl --context=$VCLUSTER_CTX apply -f self-signed-issuer.yaml ``` 2. Tenant Cluster Create a certificate self-signed-certificate.yaml ``` apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: app-tls-cert namespace: default spec: secretName: app-tls-secret duration: 2160h # 90 days renewBefore: 360h # 15 days commonName: myapp.local dnsNames: - myapp.local - api.myapp.local privateKey: algorithm: RSA size: 2048 issuerRef: name: selfsigned-issuer kind: Issuer ``` Apply the certificate: Apply certificate ``` kubectl --context=$VCLUSTER_CTX apply -f self-signed-certificate.yaml ``` warning Self-signed certificates generate browser warnings and should not be used in production environments accessible to external users. Private PKI for enterprise environments Private PKI configurations provide enterprise-grade certificate management with custom certificate authorities. 1. Control Plane Cluster Create a root CA certificate Create a self-signed root CA in the control plane cluster (ClusterIssuers are imported from control plane to tenant cluster): root-ca.yaml ``` apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-issuer spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: root-ca-cert namespace: cert-manager spec: isCA: true commonName: "Enterprise Root CA" subject: organizations: - "Enterprise Corp" organizationalUnits: - "IT Security" countries: - "US" secretName: root-ca-secret duration: 87600h # 10 years renewBefore: 78840h # 9 years privateKey: algorithm: ECDSA size: 256 issuerRef: name: selfsigned-issuer kind: ClusterIssuer ``` Apply in the control plane cluster: Apply root CA in control plane cluster ``` kubectl --context=$HOST_CTX apply -f root-ca.yaml ``` 2. Control Plane Cluster Create a CA issuer ca-issuer.yaml ``` apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: enterprise-ca-issuer spec: ca: secretName: root-ca-secret ``` Apply in the control plane cluster: Apply CA issuer in control plane cluster ``` kubectl --context=$HOST_CTX apply -f ca-issuer.yaml ``` 3. Tenant Cluster Issue certificates from your CA ca-signed-certificate.yaml ``` apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: enterprise-app-cert namespace: default spec: secretName: enterprise-app-tls duration: 8760h # 1 year renewBefore: 720h # 30 days commonName: myapp.enterprise.local dnsNames: - myapp.enterprise.local - api.myapp.enterprise.local subject: organizations: - "Enterprise Corp" organizationalUnits: - "Application Services" usages: - digital signature - key encipherment - server auth issuerRef: name: enterprise-ca-issuer kind: ClusterIssuer ``` Apply in the tenant cluster: Apply certificate in tenant cluster ``` kubectl --context=$VCLUSTER_CTX apply -f ca-signed-certificate.yaml ``` **Security considerations**: * Store root CA private keys securely and offline when possible * Implement proper certificate lifecycle management * Monitor certificate expiration dates * Establish clear revocation procedures ACME for public-facing services [ACME](https://cert-manager.io/docs/configuration/acme/) (Automatic Certificate Management Environment) issuers like [Let's Encrypt](https://letsencrypt.org) provide free, automatically-renewed certificates for public-facing services. 1. Tenant Cluster Create an ACME issuer acme-issuer.yaml ``` apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: letsencrypt-prod namespace: default spec: acme: email: admin@yourdomain.com server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt-prod-account-key solvers: - http01: ingress: ingressClassName: nginx # the ingress-nginx project has been deprecated, we recommend using a different ingress class - dns01: cloudflare: email: admin@yourdomain.com apiKeySecretRef: name: cloudflare-api-key-secret key: api-key ``` 2. Tenant Cluster Create an ACME certificate acme-certificate.yaml ``` apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: public-app-cert namespace: default spec: secretName: public-app-secret duration: 2160h # 90 days renewBefore: 720h # 30 days dnsNames: - myapp.example.com - api.myapp.example.com issuerRef: name: letsencrypt-prod kind: Issuer ``` **Rate limits**: Let's Encrypt enforces rate limits. Use staging environment (`https://acme-staging-v02.api.letsencrypt.org/directory`) for testing. PKI for compliance requirements PKI configurations require specific certificate formats and validation procedures for compliance with standards. gov-compliant-certificate.yaml ``` apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: gov-service-cert namespace: default spec: secretName: gov-service-secret duration: 8760h # 1 year maximum renewBefore: 2160h # 90 days commonName: service.application.1234567890 subject: organizations: - "Organization" organizationalUnits: - "PKI" - "CONTRACTOR" countries: - "US" usages: - digital signature - key encipherment - server auth privateKey: algorithm: RSA size: 2048 # Federal minimum requirement issuerRef: name: gov-pki-issuer kind: ClusterIssuer ``` **Compliance requirements**: * Use hierarchical 3-tier CA structure * Include specific Distinguished Name fields * Maintain Certificate Revocation Lists (CRLs) * Follow Federal PKI Certificate Policy guidelines ## Verify certificate creation[​](#verify-certificate-creation "Direct link to Verify certificate creation") Check that your certificates generate correctly: 1. Tenant Cluster Check certificate status Verify certificate creation ``` kubectl --context=$VCLUSTER_CTX describe certificate -n ``` Look for `Ready: True` in the status conditions. 2. Tenant Cluster Verify secret creation Check certificate secret ``` kubectl --context=$VCLUSTER_CTX get secret -n -o yaml ``` Confirm the secret contains `tls.crt` and `tls.key` data. 3. Control Plane Cluster Confirm synchronization Check control plane cluster resources ``` kubectl --context=$HOST_CTX get certificate,issuer -A ``` Verify resources appear in the control plane cluster namespace. ## Use certificates in applications[​](#use-certificates-in-applications "Direct link to Use certificates in applications") Reference your certificate secrets in Ingress resources: secure-ingress.yaml ``` apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: secure-app-ingress annotations: kubernetes.io/ingress.class: nginx # ingress-nginx has been deprecated nginx.ingress.kubernetes.io/ssl-redirect: "true" spec: tls: - hosts: - myapp.example.com secretName: your-tls-secret # Use the secretName from your Certificate rules: - host: myapp.example.com http: paths: - path: / pathType: Prefix backend: service: name: app-service port: number: 80 ``` tip Replace `your-tls-secret` with the actual `secretName` specified in your Certificate resource (e.g., `app-tls-secret`, `enterprise-app-tls`, or `public-app-secret` from the examples above). ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") Certificate creation issues **Check certificate status**: ``` kubectl --context=$VCLUSTER_CTX describe certificate ``` **Common problems**: * **Pending status**: Check CertificateRequest events * **Failed validation**: Verify DNS/HTTP challenge accessibility * **Rate limiting**: Switch to staging ACME server for testing **Examine certificate requests**: ``` kubectl --context=$VCLUSTER_CTX get certificaterequests kubectl --context=$VCLUSTER_CTX describe certificaterequest ``` Integration synchronization problems **Verify integration status**: Control Plane Cluster Check cert-manager installation ``` kubectl --context=$HOST_CTX -n cert-manager get pods kubectl --context=$HOST_CTX -n cert-manager logs deployment/cert-manager ``` Tenant Cluster Verify integration configuration ``` # Check if integration is enabled kubectl --context=$VCLUSTER_CTX get configmap vcluster-config -o yaml # Check resource synchronization kubectl --context=$VCLUSTER_CTX get events --field-selector reason=Sync ``` **Common synchronization issues**: * RBAC permissions missing on control plane cluster * Network policies blocking communication * Resource naming conflicts between clusters ACME challenge failures For detailed troubleshooting of ACME challenges: **Check challenge status**: ``` kubectl --context=$VCLUSTER_CTX describe order kubectl --context=$VCLUSTER_CTX describe challenge ``` For comprehensive ACME troubleshooting, including HTTP-01 and DNS-01 challenge issues, see the [cert-manager ACME troubleshooting guide](https://cert-manager.io/docs/troubleshooting/acme/). For troubleshooting guidance, see the [cert-manager troubleshooting documentation](https://cert-manager.io/docs/troubleshooting/). ## Advanced configuration[​](#advanced-configuration "Direct link to Advanced configuration") Custom sync configurations Fine-tune resource synchronization behavior: Advanced cert-manager integration ``` integrations: certManager: enabled: true sync: toHost: certificates: enabled: true issuers: enabled: true fromHost: clusterIssuers: enabled: true selector: labels: vcluster.loft.sh/managed: "true" ``` **Configuration options**: * `selector`: Filter which ClusterIssuers to import * Individual resource type controls for granular sync management ## Configuration reference[​](#configuration-reference "Direct link to Configuration reference") ## `certManager` object [​](#certManager "Direct link to certManager")CertManager reuses a host cert-manager and makes its CRDs from it available inside the vCluster.- Certificates and Issuers will be synced from the virtual cluster to the host cluster. - ClusterIssuers will be synced from the host cluster to the virtual cluster. ### `enabled` boolean false [​](#certManager-enabled "Direct link to certManager-enabled")Enabled defines if this option should be enabled. ### `sync` object [​](#certManager-sync "Direct link to certManager-sync")Sync contains advanced configuration for syncing cert-manager resources. #### `toHost` object [​](#certManager-sync-toHost "Direct link to certManager-sync-toHost") ##### `certificates` object [​](#certManager-sync-toHost-certificates "Direct link to certManager-sync-toHost-certificates")Certificates defines if certificates should get synced from the virtual cluster to the host cluster. ##### `enabled` boolean true [​](#certManager-sync-toHost-certificates-enabled "Direct link to certManager-sync-toHost-certificates-enabled")Enabled defines if this option should be enabled. ##### `issuers` object [​](#certManager-sync-toHost-issuers "Direct link to certManager-sync-toHost-issuers")Issuers defines if issuers should get synced from the virtual cluster to the host cluster. ##### `enabled` boolean true [​](#certManager-sync-toHost-issuers-enabled "Direct link to certManager-sync-toHost-issuers-enabled")Enabled defines if this option should be enabled. #### `fromHost` object [​](#certManager-sync-fromHost "Direct link to certManager-sync-fromHost") ##### `clusterIssuers` object [​](#certManager-sync-fromHost-clusterIssuers "Direct link to certManager-sync-fromHost-clusterIssuers")ClusterIssuers defines if (and which) cluster issuers should get synced from the host cluster to the virtual cluster. ##### `enabled` boolean true [​](#certManager-sync-fromHost-clusterIssuers-enabled "Direct link to certManager-sync-fromHost-clusterIssuers-enabled")Enabled defines if this option should be enabled. ##### `selector` object [​](#certManager-sync-fromHost-clusterIssuers-selector "Direct link to certManager-sync-fromHost-clusterIssuers-selector")Selector defines what cluster issuers should be imported. ##### `labels` object {} [​](#certManager-sync-fromHost-clusterIssuers-selector-labels "Direct link to certManager-sync-fromHost-clusterIssuers-selector-labels")Labels defines what labels should be looked for ## Config reference[​](#config-reference "Direct link to Config reference") --- # External secrets | | | Enterprise | | | | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [External Secrets Integration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/eso.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * `kubectl` installed * `external-secrets` operator installed on your control plane cluster. See instructions at External secrets version By default, vCluster uses the same CRD version as the External Secrets Operator installed on your control plane cluster. The `version` field allows you to explicitly set which CRD version to use (for example, `v1beta1` or `v1`). Ensure your chosen version is supported by the External Secrets Operator on your control plane cluster. # External secrets integration To enable the external secret integration, set the following fields: ``` integrations: externalSecrets: enabled: true version: v1 # Optional. If not specified, uses the same CRD version as your host cluster's External Secrets Operator sync: toHost: stores: enabled: true fromHost: clusterStores: enabled: true ``` This enables the integration and the sync for all custom resources: * `ExternalSecret`: namespaced, synced from tenant cluster into control plane cluster and then bi-directionally * `SecretStore`: namespaced, synced from tenant cluster into control plane cluster * `ClusterSecretStore`: cluster scoped, synced from control plane cluster into tenant cluster Once the tenant cluster is up and running, you can create a `SecretStore` inside the tenant cluster. For this guide, you use the `fake` store type, which prefills data instead of connecting to a distant secret store. API Version Compatibility External Secrets Operator `v0.16.2` is the last supporting `v1beta1` API version. For `v0.17.0+` versions, it provides exclusively `v1`. ``` apiVersion: external-secrets.io/v1 kind: SecretStore metadata: name: fake spec: provider: fake: data: - key: "/foo/bar" value: "HELLO1" version: "v1" - key: "/foo/bar" value: "HELLO2" version: "v2" - key: "/foo/baz" value: '{"john": "doe"}' version: "v1" ``` Inside the tenant cluster, create the store with `kubectl apply -f fake.yaml`. This creates a corresponding store in the control plane cluster. You can then create an `ExternalSecret` in the tenant cluster, which references the `SecretStore`. ``` apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: example spec: refreshInterval: 1h secretStoreRef: name: fake kind: SecretStore target: name: secret-to-be-created data: - secretKey: foo_bar remoteRef: key: /foo/bar version: v1 dataFrom: - extract: key: /foo/baz version: v1 ``` After the `ExternalSecret` is created in the tenant cluster, the integration creates a corresponding resource inside the control plane cluster. The external secret operator running in the control plane cluster creates the corresponding Kubernetes secret which the integration imports into the tenant cluster. Running `kubectl get secrets` in the tenant cluster includes the `secret-to-be-created` in its output. ## Config reference[​](#config-reference "Direct link to Config reference") ## `externalSecrets` object [​](#externalSecrets "Direct link to externalSecrets")ExternalSecrets reuses a host external secret operator and makes certain CRDs from it available inside the vCluster.- ExternalSecrets will be synced from the virtual cluster to the host cluster. - SecretStores will be synced from the virtual cluster to the host cluster and then bi-directionally. - ClusterSecretStores will be synced from the host cluster to the virtual cluster. ### `enabled` boolean false [​](#externalSecrets-enabled "Direct link to externalSecrets-enabled")Enabled defines whether the external secret integration is enabled or not ### `version` string [​](#externalSecrets-version "Direct link to externalSecrets-version")Version defines the version of the external secrets operator to use. If empty, the storage version will be used. ### `webhook` object [​](#externalSecrets-webhook "Direct link to externalSecrets-webhook")Webhook defines whether the host webhooks are reused or not #### `enabled` boolean false [​](#externalSecrets-webhook-enabled "Direct link to externalSecrets-webhook-enabled")Enabled defines if this option should be enabled. ### `sync` object [​](#externalSecrets-sync "Direct link to externalSecrets-sync")Sync defines the syncing behavior for the integration #### `toHost` object [​](#externalSecrets-sync-toHost "Direct link to externalSecrets-sync-toHost")ToHost defines what resources are synced from the virtual cluster to the host ##### `externalSecrets` object [​](#externalSecrets-sync-toHost-externalSecrets "Direct link to externalSecrets-sync-toHost-externalSecrets")ExternalSecrets allows to configure if only a subset of ExternalSecrets matching a label selector should get synced from the virtual cluster to the host cluster. ##### `selector` object [​](#externalSecrets-sync-toHost-externalSecrets-selector "Direct link to externalSecrets-sync-toHost-externalSecrets-selector") ##### `matchLabels` object {} [​](#externalSecrets-sync-toHost-externalSecrets-selector-matchLabels "Direct link to externalSecrets-sync-toHost-externalSecrets-selector-matchLabels") ##### `matchExpressions` object\[] [​](#externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions "Direct link to externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions") ##### `key` string [​](#externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-key "Direct link to externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-key") ##### `operator` string [​](#externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-operator "Direct link to externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-operator") ##### `values` string\[] [​](#externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-values "Direct link to externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-values") ##### `stores` object [​](#externalSecrets-sync-toHost-stores "Direct link to externalSecrets-sync-toHost-stores")Stores defines if secret stores should get synced from the virtual cluster to the host cluster and then bi-directionally. ##### `selector` object [​](#externalSecrets-sync-toHost-stores-selector "Direct link to externalSecrets-sync-toHost-stores-selector") ##### `matchLabels` object {} [​](#externalSecrets-sync-toHost-stores-selector-matchLabels "Direct link to externalSecrets-sync-toHost-stores-selector-matchLabels") ##### `matchExpressions` object\[] [​](#externalSecrets-sync-toHost-stores-selector-matchExpressions "Direct link to externalSecrets-sync-toHost-stores-selector-matchExpressions") ##### `key` string [​](#externalSecrets-sync-toHost-stores-selector-matchExpressions-key "Direct link to externalSecrets-sync-toHost-stores-selector-matchExpressions-key") ##### `operator` string [​](#externalSecrets-sync-toHost-stores-selector-matchExpressions-operator "Direct link to externalSecrets-sync-toHost-stores-selector-matchExpressions-operator") ##### `values` string\[] [​](#externalSecrets-sync-toHost-stores-selector-matchExpressions-values "Direct link to externalSecrets-sync-toHost-stores-selector-matchExpressions-values") ##### `enabled` boolean false [​](#externalSecrets-sync-toHost-stores-enabled "Direct link to externalSecrets-sync-toHost-stores-enabled")Enabled defines if this option should be enabled. #### `fromHost` object [​](#externalSecrets-sync-fromHost "Direct link to externalSecrets-sync-fromHost")FromHost defines what resources are synced from the host cluster to the virtual cluster ##### `clusterStores` object [​](#externalSecrets-sync-fromHost-clusterStores "Direct link to externalSecrets-sync-fromHost-clusterStores")ClusterStores defines if cluster secrets stores should get synced from the host cluster to the virtual cluster. ##### `selector` object [​](#externalSecrets-sync-fromHost-clusterStores-selector "Direct link to externalSecrets-sync-fromHost-clusterStores-selector") ##### `matchLabels` object {} [​](#externalSecrets-sync-fromHost-clusterStores-selector-matchLabels "Direct link to externalSecrets-sync-fromHost-clusterStores-selector-matchLabels") ##### `matchExpressions` object\[] [​](#externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions "Direct link to externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions") ##### `key` string [​](#externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-key "Direct link to externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-key") ##### `operator` string [​](#externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-operator "Direct link to externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-operator") ##### `values` string\[] [​](#externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-values "Direct link to externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-values") ##### `enabled` boolean false [​](#externalSecrets-sync-fromHost-clusterStores-enabled "Direct link to externalSecrets-sync-fromHost-clusterStores-enabled")Enabled defines if this option should be enabled. --- # Istio Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. # Istio integration This guide shows how to set up Istio integration with your tenant cluster. This enables you to use one Istio installation from the control plane cluster instead of installing Istio in each tenant cluster. You can include your tenant workloads in the mesh by setting `istio.io/dataplane-mode=ambient` label on the tenant Namespaces or Pods. You can exclude your tenant workloads from the mesh by setting `istio.io/dataplane-mode=none` label either on the Namespace or on the Pod. ## Istio supported versions[​](#istio-supported-versions "Direct link to Istio supported versions") Integration works with following Istio / Gateway API Versions: | Istio Version | Gateway API Version | | ------------- | ------------------- | | v1.29 | 1.4 | | v1.30 | 1.5 | | v1.31 | 1.6 | ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Run the command `kubectl auth can-i create clusterrole -A` to verify that your current kube-context has administrative privileges. info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm`: Helm v3.10 is required for deploying the platform. Refer to the [Helm Installation Guide](https://Helm.sh/docs/intro/install/) if you need to install it. * `kubectl`: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. - `istio` operator installed on your control plane cluster in ambient mode with DNS Capture disabled warning To disable DNS capture, set `values.cni.ambient.dnsCapture: false` in your Istio configuration. This integration works only with Istio in Ambient mode. Sidecar mode is not supported. ## Enable the integration[​](#enable-the-integration "Direct link to Enable the integration") Enable the Istio integration in your tenant cluster configuration: Enable istio integration ``` integrations: istio: enabled: true ``` This configuration: * Enables the integration. * Installs Resource Definitions for `DestinationRules`, `Gateways` and `VirtualServices` into the tenant cluster. * Exports `DestinationRules`, `Gateways` and `VirtualServices` from the tenant cluster to the control plane cluster (and re-writes) service references to the services translated names in the control plane cluster. * Adds `istio.io/dataplane-mode` label to the synced Pods based on the value of this label set in the virtual namespace. warning Only `DestinationRules`, `Gateways`, and `VirtualServices` from `networking.istio.io/v1` API Version are synced to the control plane clusters. Other kinds are not yet supported. Prefer the Kubernetes Gateway API for new gateways Istio [intends to make the Kubernetes Gateway API the default API for traffic management](https://istio.io/latest/docs/tasks/traffic-management/ingress/gateway-api/). For new setups that expose tenant workloads outside the cluster, consider Kubernetes `Gateway` resources instead of Istio `Gateway` (`networking.istio.io`) resources: * To let tenants create their own `Gateway` and `HTTPRoute` resources, enable [Gateway API sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md). * To let tenants attach `HTTPRoute` resources to a shared Gateway in the control plane cluster, see [Imported Gateways and GatewayClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/gateways.md). The integration continues to sync Istio `Gateway` resources, so existing configurations keep working. `VirtualService` and `DestinationRule` resources are still required for the mesh-internal routing shown in this guide. ### Network policies[​](#network-policies "Direct link to Network policies") If you are using [network policies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md), consult the [Ambient and Kubernetes NetworkPolicy](https://istio.io/latest/docs/ambient/usage/networkpolicy/) documentation. Depending on the underlying configuration, the following policies may be required: vcluster.yaml ``` policies: networkPolicy: workload: ingress: - ports: # Allow HBONE traffic per https://istio.io/latest/docs/ambient/usage/networkpolicy/ - port: 15008 protocol: TCP - from: # Allow kubelet health probe per https://istio.io/latest/docs/ambient/usage/networkpolicy/ - ipBlock: cidr: 169.254.7.127/32 ``` ## Route request based on the version label of the app[​](#route-request-based-on-the-version-label-of-the-app "Direct link to Route request based on the version label of the app") 1. ### Set up cluster contexts[​](#set-up-cluster-contexts "Direct link to Set up cluster contexts") Setting up the control plane and tenant cluster contexts makes it easier to switch between them. You can adjust the values below according to your setup. Here, the `HOST_CONTEXT` is the context of the control plane cluster, while the `VCLUSTER_CONTEXT` is the context to access the created vCluster. The `VCLUSTER_HOST_NAMESPACE` is the namespace where the vCluster is created on the control plane cluster. The `ISTIO_NAMESPACE` is the namespace to deploy the Istio integration. The commands in the following steps will be automatically updated to follow your configuration. For the GatewayAPI version, you should set it according to your Istio version / GatewayAPI version matrix above. Modify the following with your specific values to replace on the whole page and generate copyable commands: HOST\_CONTEXTyour-host-context VCLUSTER\_CONTEXTvcluster-ctx VCLUSTER\_HOST\_NAMESPACEvcluster GATEWAY\_API\_VERSIONv1.2.1 ISTIO\_NAMESPACEistio tip You can find your contexts by running `kubectl config get-contexts` 2. ### Create waypoint proxy in the host[​](#create-waypoint-proxy-in-the-host "Direct link to Create waypoint proxy in the host") In this tutorial, you set Kubernetes service name as a host in the VirtualService `spec.hosts`. To make it work, you need a Waypoint proxy in the tenant cluster's control plane namespace. In many cases it is optional however. Refer to Istio documentation for more information on Waypoint proxies. Install Gateway CRD first in the host: Install Gateway CRD ``` kubectl --context="your-host-context" get crd gateways.gateway.networking.k8s.io &> /dev/null || \ kubectl --context="your-host-context" apply -f "https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml" ``` this is a Gateway for Waypoint you need: waypoint-gateway.yaml ``` apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: waypoint labels: istio.io/waypoint-for: service spec: gatewayClassName: istio-waypoint listeners: - name: mesh port: 15008 protocol: HBONE ``` create it in the control plane cluster: Create Waypoint Gateway ``` kubectl --context="your-host-context" create -f waypoint-gateway.yaml --namespace="vcluster" ``` 3. ### Create virtual namespace with ambient mode enabled[​](#create-virtual-namespace-with-ambient-mode-enabled "Direct link to Create virtual namespace with ambient mode enabled") First, you need to create a test namespace: Create a test namespace ``` kubectl --context="vcluster-ctx" create namespace istio ``` and label it with `istio.io/dataplane-mode: ambient`: Label the test namespace ``` kubectl --context="vcluster-ctx" label namespace istio istio.io/dataplane-mode=ambient ``` 4. ### Create two versions of your app[​](#create-two-versions-of-your-app "Direct link to Create two versions of your app") Next, you create 3 deployments: two of them are nginx server and the third one is to curl the other two. Create NGINX deployments that respond with different response bodies based on the contents of their respective ConfigMaps: configmap1.yaml ``` apiVersion: v1 kind: ConfigMap metadata: name: nginx-configmap-v1 namespace: istio data: index.html: | Welcome to nginx v1!

Hello from Nginx Version 1!

``` deployment1.yaml ``` apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment-v1 namespace: istio labels: app: nginx version: v1 spec: replicas: 1 selector: matchLabels: app: nginx version: v1 template: metadata: labels: app: nginx version: v1 spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 volumeMounts: - name: nginx-index-v1 mountPath: /usr/share/nginx/html/index.html subPath: index.html volumes: - name: nginx-index-v1 configMap: name: nginx-configmap-v1 ``` Create v1 config map ``` kubectl --context="vcluster-ctx" create -f configmap1.yaml --namespace istio ``` Create v1 deployment ``` kubectl --context="vcluster-ctx" create -f deployment1.yaml --namespace istio ``` make sure that this nginx app is up and running: Wait for v1 pods ``` kubectl --context="vcluster-ctx" wait --for=condition=ready pod -l app=nginx --namespace istio --timeout=300s ``` Create an additional NGINX deployment configured to serve a different response body, using a separate ConfigMap: configmap2.yaml ``` apiVersion: v1 kind: ConfigMap metadata: name: nginx-configmap-v2 namespace: istio data: index.html: | Welcome to nginx v2!

Hello from Nginx Version 2!

``` deployment2.yaml ``` apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment-v2 namespace: istio labels: app: nginx version: v2 spec: replicas: 1 selector: matchLabels: app: nginx version: v2 template: metadata: labels: app: nginx version: v2 spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 volumeMounts: - name: nginx-index-v2 mountPath: /usr/share/nginx/html/index.html subPath: index.html volumes: - name: nginx-index-v2 configMap: name: nginx-configmap-v2 ``` Create v2 config map ``` kubectl --context="vcluster-ctx" create -f configmap2.yaml --namespace istio ``` Create v1 deployment ``` kubectl --context="vcluster-ctx" create -f deployment2.yaml --namespace istio ``` To ensure your NGINX application is up and running in your Kubernetes cluster, use the following command: Wait for v2 pods ``` kubectl --context="vcluster-ctx" wait --for=condition=ready pod -l app=nginx --namespace istio --timeout=300s ``` Create a Service that targets Pods from both Deployments by using a shared label: service.yaml ``` apiVersion: v1 kind: Service metadata: name: nginx-service namespace: istio labels: app: nginx istio.io/use-waypoint: "waypoint" spec: ports: - port: 80 targetPort: 80 selector: app: nginx ``` The istio.io/use-waypoint: waypoint label directs Istio to route traffic for the labeled resource through the waypoint proxy within the same namespace. This configuration enables [Layer 7 (L7) policy](https://istio.io/latest/docs/ambient/usage/l7-features/) enforcement and observability features provided by the waypoint proxy. Applying this label to a namespace ensures that all Pods and Services within that namespace use the specified waypoint proxy. To deploy a Service defined in the `service.yaml` file within the test namespace of the Kubernetes cluster specified by the `VCLUSTER_CONTEXT` context, use the following command: Create service ``` kubectl --context="vcluster-ctx" create -f service.yaml --namespace istio ``` To test connectivity between the two NGINX deployments, deploy a temporary Pod equipped with `curl`: client\_deployment.yaml ``` apiVersion: apps/v1 kind: Deployment metadata: name: client namespace: istio labels: app: client spec: replicas: 1 selector: matchLabels: app: client template: metadata: labels: app: client spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 ``` Create client deployment ``` kubectl --context="vcluster-ctx" create -f client_deployment.yaml --namespace istio ``` 5. ### Configure your desired traffic routing using DestinationRule and VirtualService[​](#configure-your-desired-traffic-routing-using-destinationrule-and-virtualservice "Direct link to Configure your desired traffic routing using DestinationRule and VirtualService") You can create `DestinationRules` and `VirtualService` in the tenant cluster. Create a pair that routes our request based on the request path: 1. Requesting `/v2` endpoint should route our request to pods with `version=v2` label 2. All other requests are routed to `version=v1` pods. Save this `DestinationRule` and `VirtualService` definition, and apply it in the tenant cluster: destination\_rule.yaml ``` apiVersion: networking.istio.io/v1 kind: DestinationRule metadata: name: nginx-destination namespace: istio spec: host: nginx-service.istio.svc.cluster.local # vCluster translates it to the host service automatically subsets: - name: v1 labels: version: v1 - name: v2 labels: version: v2 ``` virtual\_service.yaml ``` apiVersion: networking.istio.io/v1 kind: VirtualService metadata: name: nginx-service namespace: istio spec: hosts: - nginx-service.istio.svc.cluster.local # vCluster translates it to the host service automatically http: - name: "nginx-v2" match: - uri: prefix: "/v2" rewrite: uri: "/" route: - destination: host: nginx-service.istio.svc.cluster.local # vCluster translates it to the host service automatically subset: v2 - name: "nginx-v1" route: - destination: host: nginx-service.istio.svc.cluster.local # vCluster translates it to the host service automatically subset: v1 ``` To apply a `DestinationRule` configuration to the tenant cluster specified by the `VCLUSTER_CONTEXT` context, use the following command: Create destination rule ``` kubectl --context="vcluster-ctx" create -f destination_rule.yaml ``` Create virtual service ``` kubectl --context="vcluster-ctx" create -f virtual_service.yaml ``` 6. ### Verify that DestinationRule and VirtualService is synced to the control plane cluster[​](#verify-that-destinationrule-and-virtualservice-is-synced-to-the-control-plane-cluster "Direct link to Verify that DestinationRule and VirtualService is synced to the control plane cluster") Check destination rule in the control plane cluster ``` kubectl --context="your-host-context" get destinationrules --namespace "vcluster" ``` Check virtual service in the host cluster ``` kubectl --context="your-host-context" get virtualservices --namespace "vcluster" ``` You should see an output mapping from a hash value prefixed with `v` to `HOST`: ``` $ kubectl --context="your-host-context" get destinationrules --namespace "vcluster" NAME HOST AGE v7f8a9b2cafds nginx-destination-x-istio-x-vcluster 1m $ kubectl --context="your-host-context" get virtualservices --namespace "vcluster" NAME GATEWAYS HOSTS AGE v3d4e5f6g ... ["nginx-service-x-istio-x-vcluster.vcluster.svc.cluster.local"] 2m ``` 7. ### Test traffic routing[​](#test-traffic-routing "Direct link to Test traffic routing") Execute a `curl` command from within the client Pod to verify responses from the two NGINX deployments. Depending on the request path, you should receive either "Hello from Nginx Version 1!" or "Hello from Nginx Version 2!" in the response: Query version 2 ``` kubectl --context="vcluster-ctx" exec -it -n istio deploy/client -- curl nginx-service/v2 Welcome to nginx v2!

Hello from Nginx Version 2!

``` Query version 1 ``` kubectl --context="vcluster-ctx" exec -it -n istio deploy/client -- curl nginx-service Welcome to nginx v1!

Hello from Nginx Version 1!

``` Seeing the same output means that request was intercepted by Istio and routed as we specified in the `DestinationRule` and `VirtualService`. 8. ### Summary[​](#summary "Direct link to Summary") Istio integration enables you to re-use one Istio instance from the control plane cluster for multiple tenant clusters. Tenant cluster users can define their own `Gateway`, `DestinationRule` and `VirtualService` without interfering with each other. ## Fields translated during the sync to control plane cluster[​](#fields-translated-during-the-sync-to-control-plane-cluster "Direct link to Fields translated during the sync to control plane cluster") Following fields of `Gateway` are modified by vCluster during the sync to the control plane cluster: * reference to the TLS Secret is re-written `spec.servers[*].tls.credentialName`. Secret is automatically synced to the control plane cluster. * namespace, `.` and `*` prefix, followed by `/` is stripped from `spec.servers[*].hosts[*]`, so e.g. `foo-namespace/loft.sh` becomes `loft.sh` in the control plane cluster object. * additional labels `vcluster.loft.sh/managed-by: [YOUR VIRTUAL CLUSTER NAME]` and `vcluster.loft.sh/namespace: [VIRTUAL NAMESPACE]` are automatically added to the `spec.subsets[*].labels` For additional information how Secret and Service references are translated, read [How does syncing work?](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync#how-does-syncing-work) Following fields of `DestinationRule` are modified by vCluster during the sync to the control plane cluster: * reference to the virtual Kubernetes Service is re-written for `spec.host` * reference to the TLS Secret in `spec.trafficPolicy.portLevelSettings[*].tls.credentialName` & `spec.trafficPolicy.tls.credentialName` is re-written. Secrets are automatically synced to the control plane cluster. * additional labels Following fields of `VirtualService` are modified by vCluster during the sync to the control plane cluster: * reference to the virtual Kubernetes Service is re-written for: * `spec.hosts[*]` * `spec.http[*].route[*].destination.host` * `spec.http[*].mirrors[*].destination.host` * `spec.tcp[*].route[*].destination.host` * `spec.tls[*].route[*].destination.host` * reference to the `networking.istio.io/v1` kind: `Gateway` is re-written for: * `spec.gateways[*]` * `spec.http[*].match[*].gateways[*]` * `spec.tls[*].match[*].gateways[*]` * `spec.tcp[*].match[*].gateways[*]` * reference to the `networking.istio.io/v1` kind: `VirtualService` is re-written for: * `spec.http[*].delegate` Fields not supported in `VirtualService`: * `spec.exportTo` * `spec.http[*].match[*].sourceLabels` * `spec.http[*].match[*].sourceNamespace` * `spec.tcp[*].match[*].sourceLabels` * `spec.tcp[*].match[*].sourceNamespace` * `spec.tls[*].match[*].sourceLabels` * `spec.tls[*].match[*].sourceNamespace` ## Config reference[​](#config-reference "Direct link to Config reference") ## `istio` object [​](#istio "Direct link to istio")Istio syncs DestinationRules, Gateways and VirtualServices from virtual cluster to the host. ### `enabled` boolean false [​](#istio-enabled "Direct link to istio-enabled")Enabled defines if this option should be enabled. ### `sync` object [​](#istio-sync "Direct link to istio-sync") #### `toHost` object [​](#istio-sync-toHost "Direct link to istio-sync-toHost") ##### `destinationRules` object [​](#istio-sync-toHost-destinationRules "Direct link to istio-sync-toHost-destinationRules") ##### `enabled` boolean true [​](#istio-sync-toHost-destinationRules-enabled "Direct link to istio-sync-toHost-destinationRules-enabled")Enabled defines if this option should be enabled. ##### `gateways` object [​](#istio-sync-toHost-gateways "Direct link to istio-sync-toHost-gateways") ##### `enabled` boolean true [​](#istio-sync-toHost-gateways-enabled "Direct link to istio-sync-toHost-gateways-enabled")Enabled defines if this option should be enabled. ##### `virtualServices` object [​](#istio-sync-toHost-virtualServices "Direct link to istio-sync-toHost-virtualServices") ##### `enabled` boolean true [​](#istio-sync-toHost-virtualServices-enabled "Direct link to istio-sync-toHost-virtualServices-enabled")Enabled defines if this option should be enabled. --- # KubeVirt | | | Enterprise | | | | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [KubeVirt Integration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/kubeVirt.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. vCluster offers the possibility to reuse a [KubeVirt](https://kubevirt.io/) installation of the control plane cluster within the vCluster. vCluster is able to sync [KubeVirt](https://kubevirt.io/) resources from the vCluster to the control plane cluster and allows you to use `virtctl` within the vCluster. Looking for KubeVirt as storage instead? This page covers creating KubeVirt VM resources in the tenant cluster and syncing them to the control plane cluster. If you want to provision PersistentVolumes on KubeVirt-backed private nodes instead, see [KubeVirt CSI](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/deploy.md#kubevirt-csi), a separate, unrelated feature. Tenant Pod Security doesn't see VM workloads Host-side KubeVirt creates the `virt-launcher` pod that actually runs the VM. Tenant-cluster Pod Security Standards never see the VM's tenant-side custom resource at all, since Pod Security Standards only evaluate Pods. On shared nodes, this needs two separate host-side admission policies. One evaluates the VirtualMachine or VirtualMachineInstance object for permitted networks and devices. A separate policy, or Pod Security Standards, covers the generated `virt-launcher` Pod for ordinary Pod-level security context. See [KubeVirt](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#kubevirt) in the shared-node hardening guide. ## Enable KubeVirt Integration[​](#enable-kubevirt-integration "Direct link to Enable KubeVirt Integration") To enable the integration, use the following configuration inside your `vcluster.yaml`: ``` integrations: kubeVirt: enabled: true # If you want vCluster to sync data volumes as well. #sync: # dataVolumes: # enabled: true ``` Next you can deploy and use KubeVirt resources inside the vCluster which will be synced to the control plane cluster. ## Sync DataVolumes[​](#sync-datavolumes "Direct link to Sync DataVolumes") To sync KubeVirt [DataVolumes](https://kubevirt.io/user-guide/storage/disks_and_volumes/), install the [KubeVirt Containerized Data Importer](https://kubevirt.io/user-guide/storage/containerized_data_importer/). Then enable the following setting in the `vcluster.yaml`: ``` integrations: kubeVirt: enabled: true sync: dataVolumes: enabled: true ``` After vCluster restarts, the `DataVolumes` CRD should be visible within the vCluster. ## Limitations[​](#limitations "Direct link to Limitations") vCluster currently only supports syncing and using the following resources inside the vCluster: * `VirtualMachine` * `VirtualMachineInstance` * `VirtualMachinePool` * `VirtualMachineClone` * `DataVolume` (needs to be enabled using `integrations.kubeVirt.sync.dataVolumes.enabled`) The following [virtctl](https://kubevirt.io/user-guide/user_workloads/virtctl_client_tool/) commands are supported: * `virtctl start` * `virtctl stop` * `virtctl migrate` * `virtctl ssh` * `virtctl vnc` * `virtctl pause` * `virtctl unpause` * `virtctl scp` * `virtctl restart` * `virtctl port-forward` * `virtctl console` ## How this differs from KubeVirt CSI[​](#how-this-differs-from-kubevirt-csi "Direct link to How this differs from KubeVirt CSI") This integration syncs KubeVirt VM resources (`VirtualMachine`, `VirtualMachineInstance`, and so on) from the tenant cluster to the control plane cluster. This lets you manage virtual machines from inside the tenant cluster. It doesn't provision storage. [KubeVirt CSI](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/deploy.md#kubevirt-csi) is a separate, unrelated feature. It's a CSI storage driver that provisions PersistentVolumes as CDI DataVolumes and hot-plugs them into the KubeVirt virtual machines backing the tenant cluster's private nodes. Enable it with `deploy.csi.kubeVirt.enabled`. The two features can be used together or independently. ## Config reference[​](#config-reference "Direct link to Config reference") ## `kubeVirt` object [​](#kubeVirt "Direct link to kubeVirt")KubeVirt reuses a host kubevirt and makes certain CRDs from it available inside the vCluster ### `enabled` boolean false [​](#kubeVirt-enabled "Direct link to kubeVirt-enabled")Enabled signals if the integration should be enabled ### `apiService` object [​](#kubeVirt-apiService "Direct link to kubeVirt-apiService")APIService holds information about where to find the virt-api service. Defaults to virt-api/kubevirt. #### `service` object [​](#kubeVirt-apiService-service "Direct link to kubeVirt-apiService-service")Service is a reference to the service for the API server. ##### `name` string [​](#kubeVirt-apiService-service-name "Direct link to kubeVirt-apiService-service-name")Name is the name of the host service of the apiservice. ##### `namespace` string [​](#kubeVirt-apiService-service-namespace "Direct link to kubeVirt-apiService-service-namespace")Namespace is the name of the host service of the apiservice. ##### `port` integer [​](#kubeVirt-apiService-service-port "Direct link to kubeVirt-apiService-service-port")Port is the target port on the host service to connect to. ### `webhook` object [​](#kubeVirt-webhook "Direct link to kubeVirt-webhook")Webhook holds configuration for enabling the webhook within the vCluster #### `enabled` boolean true [​](#kubeVirt-webhook-enabled "Direct link to kubeVirt-webhook-enabled")Enabled defines if this option should be enabled. ### `sync` object [​](#kubeVirt-sync "Direct link to kubeVirt-sync")Sync holds configuration on what resources to sync #### `dataVolumes` object [​](#kubeVirt-sync-dataVolumes "Direct link to kubeVirt-sync-dataVolumes")If DataVolumes should get synced ##### `enabled` boolean false [​](#kubeVirt-sync-dataVolumes-enabled "Direct link to kubeVirt-sync-dataVolumes-enabled")Enabled defines if this option should be enabled. #### `virtualMachineInstanceMigrations` object [​](#kubeVirt-sync-virtualMachineInstanceMigrations "Direct link to kubeVirt-sync-virtualMachineInstanceMigrations")If VirtualMachineInstanceMigrations should get synced ##### `enabled` boolean true [​](#kubeVirt-sync-virtualMachineInstanceMigrations-enabled "Direct link to kubeVirt-sync-virtualMachineInstanceMigrations-enabled")Enabled defines if this option should be enabled. #### `virtualMachineInstances` object [​](#kubeVirt-sync-virtualMachineInstances "Direct link to kubeVirt-sync-virtualMachineInstances")If VirtualMachineInstances should get synced ##### `enabled` boolean true [​](#kubeVirt-sync-virtualMachineInstances-enabled "Direct link to kubeVirt-sync-virtualMachineInstances-enabled")Enabled defines if this option should be enabled. #### `virtualMachines` object [​](#kubeVirt-sync-virtualMachines "Direct link to kubeVirt-sync-virtualMachines")If VirtualMachines should get synced ##### `enabled` boolean true [​](#kubeVirt-sync-virtualMachines-enabled "Direct link to kubeVirt-sync-virtualMachines-enabled")Enabled defines if this option should be enabled. #### `virtualMachineClones` object [​](#kubeVirt-sync-virtualMachineClones "Direct link to kubeVirt-sync-virtualMachineClones")If VirtualMachineClones should get synced ##### `enabled` boolean true [​](#kubeVirt-sync-virtualMachineClones-enabled "Direct link to kubeVirt-sync-virtualMachineClones-enabled")Enabled defines if this option should be enabled. #### `virtualMachinePools` object [​](#kubeVirt-sync-virtualMachinePools "Direct link to kubeVirt-sync-virtualMachinePools")If VirtualMachinePools should get synced ##### `enabled` boolean true [​](#kubeVirt-sync-virtualMachinePools-enabled "Direct link to kubeVirt-sync-virtualMachinePools-enabled")Enabled defines if this option should be enabled. --- # Metrics server Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** ### Reuse host metrics server[​](#reuse-host-metrics-server "Direct link to Reuse host metrics server") Create a `vcluster.yaml` with: ``` integrations: metricsServer: enabled: true nodes: true pods: true ``` Then create a vCluster with: ``` vcluster create host-metrics-vcluster -f vcluster.yaml ``` Then the metrics api should be available within the vCluster. This will obviously require a metrics server to be installed on the control plane cluster. If your metrics server is not using the default `kube-system/metrics-server` service, you can also configure a custom one via: ``` integrations: metricsServer: enabled: true apiService: service: name: my-custom-metrics-server namespace: my-custom-namespace port: 443 ``` If you are using [network policies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md), you need to add the metrics server pod to the allowed rules that the control plane can communicate with: ``` policies: networkPolicy: controlPlane: egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: 'kube-system' podSelector: matchLabels: k8s-app: metrics-server ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `metricsServer` object [​](#metricsServer "Direct link to metricsServer")MetricsServer reuses the metrics server from the host cluster within the vCluster. ### `enabled` boolean false [​](#metricsServer-enabled "Direct link to metricsServer-enabled")Enabled signals the metrics server integration should be enabled. ### `apiService` object [​](#metricsServer-apiService "Direct link to metricsServer-apiService")APIService holds information about where to find the metrics-server service. Defaults to metrics-server/kube-system. #### `service` object [​](#metricsServer-apiService-service "Direct link to metricsServer-apiService-service")Service is a reference to the service for the API server. ##### `name` string [​](#metricsServer-apiService-service-name "Direct link to metricsServer-apiService-service-name")Name is the name of the host service of the apiservice. ##### `namespace` string [​](#metricsServer-apiService-service-namespace "Direct link to metricsServer-apiService-service-namespace")Namespace is the name of the host service of the apiservice. ##### `port` integer [​](#metricsServer-apiService-service-port "Direct link to metricsServer-apiService-service-port")Port is the target port on the host service to connect to. ### `nodes` boolean true [​](#metricsServer-nodes "Direct link to metricsServer-nodes")Nodes defines if metrics-server nodes api should get proxied from host to virtual cluster. ### `pods` boolean true [​](#metricsServer-pods "Direct link to metricsServer-pods")Pods defines if metrics-server pods api should get proxied from host to virtual cluster. --- # Netris Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. The Netris integration enables vCluster to use [Netris](https://www.netris.io/) for network automation for the vCluster control plane and [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md). ## Features[​](#features "Direct link to Features") For more information on the features capable of integrating with Netris, see: * [Automatically configure kube-vip](#automatically-configure-kube-vip) * [BCM Node Provider with Netris](https://vcluster.com/docs/platform/administer/node-providers/bcm.md) * [KubeVirt Node Provider with Netris](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md) ## Configure Netris integration[​](#configure-netris-integration "Direct link to Configure Netris integration") 1. Create a Secret in the control plane cluster in the namespace where vCluster Platform is installed to provide authentication and endpoint information: ``` apiVersion: v1 kind: Secret metadata: name: netris-credentials namespace: vcluster-platform stringData: url: "https://netris.example.com/api" username: "admin" password: "password" ``` 2. Reference the Secret in your `vcluster.yaml`: ``` integrations: netris: enabled: true connector: netris-credentials ``` ## Automatically configure kube-vip[​](#automatically-configure-kube-vip "Direct link to Automatically configure kube-vip") info This requires [Multus](https://github.com/k8snetworkplumbingwg/multus-cni) to be installed in the control plane cluster. When the Netris integration is enabled, you can configure it to automatically configure kube-vip for your vCluster control plane. The platform looks up the server cluster in Netris and uses the information to create the necessary Multus NetworkAttachmentDefinition and configure kube-vip. Add the `kubeVip` configuration to your Netris integration: ``` integrations: netris: enabled: true connector: netris-credentials kubeVip: # Netris server cluster to use for VIP allocation serverCluster: vcluster-control-plane # Optional: IP ranges to use for allocation instead of CIDR-based allocation ipRange: "10.0.0.10-10.0.0.20" ``` ### How it works[​](#how-it-works "Direct link to How it works") When configured, vCluster Platform will: 1. Query the specified Netris server cluster to get subnet, gateway & vxlan id information 2. Allocate a VIP from the subnet (or from the specified IP range if `ipRange` is set) 3. Create a NetworkAttachmentDefinition with the specified bridge 4. Transparently configure kube-vip for the vCluster instance to announce the VIP You don't need to configure any of this yourself. The equivalent manual configuration would look like this: ``` # transparently configured controlPlane: # Set the endpoint to the allocated VIP endpoint: 10.0.0.11:8443 statefulSet: annotations: # Reference the auto-created NetworkAttachmentDefinition k8s.v1.cni.cncf.io/networks: netris-kubevip- advanced: kubeVip: # enable kube-vip enabled: true # set to Multus interface interface: net1 # set gateway and subnet prefix for policy-based routing gateway: 10.0.0.1/24 ``` For more information on manual kube-vip configuration, see [Kube-vip](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/kube-vip.md). ## Additional resources[​](#additional-resources "Direct link to Additional resources") * [Netris website](https://www.netris.io/) ## Config reference[​](#config-reference "Direct link to Config reference") ## `netris` object [​](#netris "Direct link to netris")Netris integration helps configuring netris networking for vCluster. ### `enabled` boolean [​](#netris-enabled "Direct link to netris-enabled")Enabled defines if netris integration is enabled ### `connector` string [​](#netris-connector "Direct link to netris-connector")Connector specifies the netris connector name ### `kubeVip` object [​](#netris-kubeVip "Direct link to netris-kubeVip")KubeVip holds kube-vip configuration for netris #### `serverCluster` string [​](#netris-kubeVip-serverCluster "Direct link to netris-kubeVip-serverCluster")ServerCluster specifies the server cluster name #### `bridge` string [​](#netris-kubeVip-bridge "Direct link to netris-kubeVip-bridge")Bridge specifies the bridge interface name #### `ipRange` string [​](#netris-kubeVip-ipRange "Direct link to netris-kubeVip-ipRange")IPRange specifies the IP range for kube-vip --- # Logging By default, vCluster writes logs to the control plane pod using a human-readable console format. This format is easy to scan and works well when reviewing logs manually with tools such as `kubectl logs`. You can alternatively configure the control plane to output logs in JSON format, which provides structured data compatible with external log aggregation tools (such as Elasticsearch). JSON logs simplify log collection, parsing, and analysis across multiple tenant clusters. ## Log encoding formats[​](#log-encoding-formats "Direct link to Log encoding formats") vCluster supports two log encoding formats for the vCluster control plane pod. * [Console](#console-format) (*Default*): Human-readable format, recommended for development and manual log review. * [JSON](#json-format): Structured format for log aggregation, commonly used with external monitoring and analysis tools. ### Console format[​](#console-format "Direct link to Console format") The console format provides human-readable logs with timestamps, log levels, source locations, and contextual information. This format works well for use cases where users are troubleshooting a single tenant cluster. Choose this format in environments where human readability is more important than structured log parsing. Example use cases include: * Local development and testing, where developers benefit from quick, readable output. * Single tenant cluster analysis, where users manually review logs and debug in a terminal. ``` 2025-06-16 04:43:25 INFO license loader/inject.go:52 initializing license... {"component": "vcluster"} 2025-06-16 04:43:25 INFO license loader/inject.go:59 detected license type {"component": "vcluster", "licenseType": "Online"} 2025-06-16 04:43:29 INFO online/license.go:141 Enabled features: {"component": "vcluster"} 2025-06-16 04:43:29 INFO online/license.go:146 rancher-integration {"component": "vcluster"} ``` ### JSON format[​](#json-format "Direct link to JSON format") The JSON format structures log data for programmatic processing and integration with log management systems. This format works well for use cases where external tools monitor and analyze logs for multiple tenant clusters. Log aggregation tools (such as Elasticsearch or Splunk) can import these logs to support automated analysis, alerting, and integration with monitoring platforms (for example, Prometheus and Grafana). Example use cases include: * Production environments where you monitor multiple tenant clusters and require automatic parsing and processing of logs. ``` {"level":"info","ts":1750160027.442782,"logger":"license","caller":"loader/inject.go:52","msg":"initializing license...","component":"vcluster"} {"level":"info","ts":1750160027.4497442,"logger":"license","caller":"loader/inject.go:59","msg":"detected license type","component":"vcluster","licenseType":"Online"} {"level":"info","ts":1750160029.1307533,"caller":"online/license.go:141","msg":"Enabled features:","component":"vcluster"} {"level":"info","ts":1750160029.1307998,"caller":"online/license.go:146","msg":"connected-clusters","component":"vcluster"} ``` #### JSON log fields[​](#json-log-fields "Direct link to JSON log fields") Each JSON log entry contains the following standard fields: | Field | Type | Description | | ----------- | ------ | -------------------------------------------- | | `level` | string | Log level (`info`, `warn`, `error`, `debug`) | | `ts` | number | Unix timestamp (seconds since epoch) | | `caller` | string | Source file and line number | | `msg` | string | Log message content | | `component` | string | vCluster component that generated the log | Additional fields might be displayed based on context and log content. ## Enable log encoding[​](#enable-log-encoding "Direct link to Enable log encoding") Configure logging in your `vcluster.yaml` configuration file. To enable JSON-formatted logs, set the log encoding to `json`: ``` logging: encoding: json ``` To disable JSON logging and return to the default console format, you can either remove the logging field entirely or explicitly set the format to console: ``` logging: encoding: console ``` note When changing between encoding formats, the Pod restarts and all previous logs are lost. ## Debug logging[​](#debug-logging "Direct link to Debug logging") To increase log verbosity for troubleshooting, enable debug logging on the control plane pod. Debug logging is controlled by an environment variable, not by the `logging` field in `vcluster.yaml`. See [Enable debug logging](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/enable-debug-logging.md) for setup steps. ## Config reference[​](#config-reference "Direct link to Config reference") ## `logging` object [​](#logging "Direct link to logging")Logging provides structured logging options ### `encoding` string console [​](#logging-encoding "Direct link to logging-encoding")Encoding specifies the format of vCluster logs, it can either be json or console. ### `file` object [​](#logging-file "Direct link to logging-file")File configures additive file logging for the vCluster control plane. #### `enabled` boolean false [​](#logging-file-enabled "Direct link to logging-file-enabled")Enabled controls whether vCluster logs are also written to a file. The path is fixed at /var/log/vcluster/vcluster.log, and its directory is backed by the vcluster-logs volume, which sidecarContainers can mount to read the logs. Log rotation keeps a 100Mi active file plus five uncompressed backups, so the volume never grows beyond roughly 600Mi. --- # Networking ## Ingress traffic[​](#ingress-traffic "Direct link to Ingress traffic") Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** ### Ingress to Service[​](#ingress-to-service "Direct link to Ingress to Service") The vCluster has the option to enable Ingress resources synchronization. That means that you can create an ingress in a vCluster to make a service in this vCluster available via a hostname/domain. However, instead of having to run a separate ingress controller in each vCluster, the ingress resource synchronizes to the underlying cluster (when enabled) which means that the vCluster can use a shared ingress controller that is running in the control plane cluster. This helps to share resources across different tenant clusters and is easier for users of tenant clusters because otherwise, they would need to install an ingress controller and manually configure DNS for each vCluster. ## From inside a tenant cluster[​](#from-inside-a-tenant-cluster "Direct link to From inside a tenant cluster") Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** ### Pod in the tenant cluster to Pod in the same tenant cluster[​](#pod-in-the-tenant-cluster-to-pod-in-the-same-tenant-cluster "Direct link to Pod in the tenant cluster to Pod in the same tenant cluster") Pods run inside the underlying control plane cluster. vCluster's syncer component syncs Pods between control plane cluster and tenant cluster. These synced Pods have cluster-internal IP addresses and can communicate with each other via IP-based networking. No additional configuration is required for Pod to Pod networking in the same tenant cluster. ### Pod in the tenant cluster to Service in the same tenant cluster[​](#pod-in-the-tenant-cluster-to-service-in-the-same-tenant-cluster "Direct link to Pod in the tenant cluster to Service in the same tenant cluster") To allow pods to communicate with services, vCluster also synchronizes Service objects, while stripping away unnecessary information from the resource. However, instead of using the DNS names of the Services inside the control plane cluster, vCluster has its own DNS service which allows tenant cluster pods to use much more intuitive DNS mappings, just as in a regular cluster. No additional configuration is required for Pod to Service networking in the same tenant cluster. ### Pod in the tenant cluster to Service in the control plane cluster[​](#pod-in-the-tenant-cluster-to-service-in-the-control-plane-cluster "Direct link to Pod in the tenant cluster to Service in the control plane cluster") See [Control plane cluster to tenant cluster](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking/replicate-services.md). ### Pod in the tenant cluster to Service in a different tenant cluster[​](#pod-in-the-tenant-cluster-to-service-in-a-different-tenant-cluster "Direct link to Pod in the tenant cluster to Service in a different tenant cluster") See [Mapping services across vCluster instances](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking/resolve-dns.md). ## From the control plane cluster[​](#from-the-control-plane-cluster "Direct link to From the control plane cluster") Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** ### Pod in the control plane cluster to Service in the tenant cluster[​](#pod-in-the-control-plane-cluster-to-service-in-the-tenant-cluster "Direct link to Pod in the control plane cluster to Service in the tenant cluster") See [Tenant cluster to control plane cluster](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking/replicate-services.md) ## Service CIDR and Pod CIDR[​](#service-cidr-and-pod-cidr "Direct link to Service CIDR and Pod CIDR") Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** To configure service cidr or pod cidr of the vCluster, e.g.: ``` privateNodes: enabled: true networking: podCIDR: 10.64.0.0/16 serviceCIDR: 10.128.0.0/16 ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `networking` object [​](#networking "Direct link to networking")Networking options related to the virtual cluster. ### `serviceCIDR` string [​](#networking-serviceCIDR "Direct link to networking-serviceCIDR")ServiceCIDR holds the service cidr for the virtual cluster. This should only be set if privateNodes.enabled is true or vCluster cannot detect the host service cidr. ### `podCIDR` string 10.244.0.0/16 [​](#networking-podCIDR "Direct link to networking-podCIDR")PodCIDR holds the pod cidr for the virtual cluster. This should only be set if privateNodes.enabled is true. ### `replicateServices` object [​](#networking-replicateServices "Direct link to networking-replicateServices")ReplicateServices allows replicating services from the host within the virtual cluster or the other way around. #### `toHost` object\[] [​](#networking-replicateServices-toHost "Direct link to networking-replicateServices-toHost")ToHost defines the services that should get synced from virtual cluster to the host cluster. If services are synced to a different namespace than the virtual cluster is in, additional permissions for the other namespace are required. ##### `from` string [​](#networking-replicateServices-toHost-from "Direct link to networking-replicateServices-toHost-from")From is the service that should get synced. Can be either in the form name or namespace/name. ##### `to` string [​](#networking-replicateServices-toHost-to "Direct link to networking-replicateServices-toHost-to")To is the target service that it should get synced to. Can be either in the form name or namespace/name. #### `fromHost` object\[] [​](#networking-replicateServices-fromHost "Direct link to networking-replicateServices-fromHost")FromHost defines the services that should get synced from the host to the virtual cluster. ##### `from` string [​](#networking-replicateServices-fromHost-from "Direct link to networking-replicateServices-fromHost-from")From is the service that should get synced. Can be either in the form name or namespace/name. ##### `to` string [​](#networking-replicateServices-fromHost-to "Direct link to networking-replicateServices-fromHost-to")To is the target service that it should get synced to. Can be either in the form name or namespace/name. ### `resolveDNS` object\[] [​](#networking-resolveDNS "Direct link to networking-resolveDNS")ResolveDNS allows to define extra DNS rules. This only works if embedded coredns is configured. #### `hostname` string [​](#networking-resolveDNS-hostname "Direct link to networking-resolveDNS-hostname")Hostname is the hostname within the vCluster that should be resolved from. #### `service` string [​](#networking-resolveDNS-service "Direct link to networking-resolveDNS-service")Service is the virtual cluster service that should be resolved from. #### `namespace` string [​](#networking-resolveDNS-namespace "Direct link to networking-resolveDNS-namespace")Namespace is the virtual cluster namespace that should be resolved from. #### `target` object [​](#networking-resolveDNS-target "Direct link to networking-resolveDNS-target")Target is the DNS target that should get mapped to ##### `hostname` string [​](#networking-resolveDNS-target-hostname "Direct link to networking-resolveDNS-target-hostname")Hostname to use as a DNS target ##### `ip` string [​](#networking-resolveDNS-target-ip "Direct link to networking-resolveDNS-target-ip")IP to use as a DNS target ##### `hostService` string [​](#networking-resolveDNS-target-hostService "Direct link to networking-resolveDNS-target-hostService")HostService to target, format is hostNamespace/hostService ##### `hostNamespace` string [​](#networking-resolveDNS-target-hostNamespace "Direct link to networking-resolveDNS-target-hostNamespace")HostNamespace to target ##### `vClusterService` string [​](#networking-resolveDNS-target-vClusterService "Direct link to networking-resolveDNS-target-vClusterService")VClusterService format is hostNamespace/vClusterName/vClusterNamespace/vClusterService ### `advanced` object [​](#networking-advanced "Direct link to networking-advanced")Advanced holds advanced network options. #### `clusterDomain` string cluster.local [​](#networking-advanced-clusterDomain "Direct link to networking-advanced-clusterDomain")ClusterDomain is the Kubernetes cluster domain to use within the virtual cluster. #### `fallbackHostCluster` boolean false [​](#networking-advanced-fallbackHostCluster "Direct link to networking-advanced-fallbackHostCluster")FallbackHostCluster allows to fallback dns to the host cluster. This is useful if you want to reach host services without any other modification. You will need to provide a namespace for the service, e.g. my-other-service.my-other-namespace #### `proxyKubelets` object [​](#networking-advanced-proxyKubelets "Direct link to networking-advanced-proxyKubelets")ProxyKubelets allows rewriting certain metrics and stats from the Kubelet to "fake" this for applications such as prometheus or other node exporters. ##### `byHostname` boolean true [​](#networking-advanced-proxyKubelets-byHostname "Direct link to networking-advanced-proxyKubelets-byHostname")ByHostname will add a special vCluster hostname to the nodes where the node can be reached at. This doesn't work for all applications, e.g. Prometheus requires a node IP. ##### `byIP` boolean true [​](#networking-advanced-proxyKubelets-byIP "Direct link to networking-advanced-proxyKubelets-byIP")ByIP will create a separate service in the host cluster for every node that will point to virtual cluster and will be used to route traffic. --- # Advanced networking Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Each vCluster has its own DNS service, which is CoreDNS by default. DNS allows pods in the tenant cluster to get the IP addresses of services that are also running in the tenant cluster. The [syncer](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync) ensures that the intuitive naming logic of Kubernetes DNS names for services applies, and that users can connect to these DNS names, which map to the IP address of the synchronized services that are present in the control plane cluster. However, this also means that you cannot directly access host services inside the tenant cluster using DNS. Host pods can only access tenant cluster services by their synced name. vCluster offers a feature to map services from the tenant cluster to the control plane cluster and vice versa. **Fallback to Host DNS** When you enable `fallbackHostCluster`, vCluster falls back to the control plane cluster's DNS for resolving domains. This is useful if the control plane cluster is using Istio or Dapr and the sidecar containers cannot connect to the central instance. It is also useful if you want to access the control plane cluster services from within the tenant cluster. Kubelet image pulls aren't covered by this feature `fallbackHostCluster` changes DNS resolution for pods in the tenant cluster. It doesn't affect the container runtime, which resolves image references using the underlying node's own DNS configuration on shared nodes. See [Use a control plane cluster registry from a tenant cluster](https://vcluster.com/docs/vcluster/learn-how-to/in-cluster-registry.md) if you need to pull images from a registry running on the control plane cluster. **`proxyKubelets`** The `proxyKubelets` section replaces `--fake-kubelet-ips` and `--disable-fake-kubelets`. You may need to configure `proxyKubelets` for proper resolution of Kubelet metrics when using Prometheus. Kubelet exposure on shared nodes `proxyKubelets.byHostname` and `proxyKubelets.byIP` enable a vCluster-mediated kubelet proxy, which backs kubelet metrics scraping and `kubectl top node` against synced nodes. `kubectl logs` and `exec` don't depend on this setting. They use an independent Pod subresource redirect, validated against pod and container ownership regardless of `proxyKubelets`. On vCluster v0.34.0 and later, the proxy's pod-listing and container-log paths are also scoped to the tenant's own pods. Node-level metric samples without a pod or PVC label aren't scoped on any version. Disable `proxyKubelets` unless you need the metrics workflow, both to close that remaining gap and as a safeguard on any version older than v0.34.0. See [node and kubelet disclosure](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#operational-hardening-and-lifecycle). ## Config reference[​](#config-reference "Direct link to Config reference") ## `advanced` object [​](#advanced "Direct link to advanced")Advanced holds advanced network options. ### `clusterDomain` string cluster.local [​](#advanced-clusterDomain "Direct link to advanced-clusterDomain")ClusterDomain is the Kubernetes cluster domain to use within the virtual cluster. ### `fallbackHostCluster` boolean false [​](#advanced-fallbackHostCluster "Direct link to advanced-fallbackHostCluster")FallbackHostCluster allows to fallback dns to the host cluster. This is useful if you want to reach host services without any other modification. You will need to provide a namespace for the service, e.g. my-other-service.my-other-namespace ### `proxyKubelets` object [​](#advanced-proxyKubelets "Direct link to advanced-proxyKubelets")ProxyKubelets allows rewriting certain metrics and stats from the Kubelet to "fake" this for applications such as prometheus or other node exporters. #### `byHostname` boolean true [​](#advanced-proxyKubelets-byHostname "Direct link to advanced-proxyKubelets-byHostname")ByHostname will add a special vCluster hostname to the nodes where the node can be reached at. This doesn't work for all applications, e.g. Prometheus requires a node IP. #### `byIP` boolean true [​](#advanced-proxyKubelets-byIP "Direct link to advanced-proxyKubelets-byIP")ByIP will create a separate service in the host cluster for every node that will point to virtual cluster and will be used to route traffic. --- # Replicate networking services Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** With open source vCluster, you need to replicate services between the control plane cluster and tenant cluster. ## Control plane cluster to tenant cluster[​](#control-plane-cluster-to-tenant-cluster "Direct link to Control plane cluster to tenant cluster") In this example, you map a service `my-host-service` in the namespace `my-host-namespace` to the tenant cluster service `my-virtual-service` in the tenant cluster namespace `my-virtual-namespace`. ``` networking: replicateServices: fromHost: - from: my-host-namespace/my-host-service to: my-virtual-namespace/my-virtual-service ``` vCluster replicates the service in the tenant cluster, with the tenant cluster service pointing to the service running in the control plane cluster. Pods inside the tenant cluster can access the host service using `my-virtual-service.my-virtual-namespace` syntax. For example, if you use cURL, the command is `curl http://my-virtual-service.my-virtual-namespace`. In the above example, when you remove the `my-host-namespace/my-host-service` service replication config from `networking.replicateServices.fromHost`, the tenant cluster service `my-virtual-namespace/my-virtual-service` is automatically deleted from the tenant cluster. Kubelet image pulls aren't covered by this feature `fromHost` creates a tenant cluster Service that CoreDNS resolves. A process inside a tenant pod, such as an application calling a private registry directly, can reach the replicated Service by name. It doesn't help the kubelet pull container images referenced in a pod's `image:` field. On shared nodes, the container runtime resolves image references using the underlying node's DNS configuration rather than the tenant cluster's CoreDNS. See [Use a control plane cluster registry from a tenant cluster](https://vcluster.com/docs/vcluster/learn-how-to/in-cluster-registry.md) for how to pull images from an in-cluster registry. ## Tenant cluster to control plane cluster[​](#tenant-cluster-to-control-plane-cluster "Direct link to Tenant cluster to control plane cluster") You can also map a tenant cluster service to a control plane cluster service. This is especially useful if you want to expose an application that runs inside the tenant cluster to other workloads running in the control plane cluster, which makes it easier to share services across vCluster instances. In this example, you map the virtual service `my-virtual-service` in the namespace `my-virtual-namespace` to the host namespace service `my-host-service` ``` networking: replicateServices: toHost: - from: my-virtual-namespace/my-virtual-service to: my-host-service ``` With this configuration, vCluster manages a service called `my-host-service` inside the namespace where the vCluster workloads are synced, which points to the virtual service `my-virtual-service` in namespace `my-virtual-namespace` inside the tenant cluster. Pods in the control plane cluster are able to access the virtual service by calling the host service. If you use cURL, the command based on the preceding example is `curl http://my-host-service`. In the above example, when you remove the `my-virtual-namespace/my-virtual-service` service replication config from `networking.replicateServices.toHost`, the control plane cluster service `my-host-service` is not automatically deleted from the control plane cluster, so you need to delete it manually, if you don't want to keep it in the control plane cluster. ## Network policies[​](#network-policies "Direct link to Network policies") If you are using [network policies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md), traffic to or from the replicated services must be allowed. vcluster.yaml ``` policies: networkPolicy: workload: egress: - to: # Example allowing vcluster workload traffic to all pods in the my-host-namespace namespace. # Depending on your use case, a more restrictive pod selector may be used. - namespaceSelector: matchLabels: kubernetes.io/metadata.name: my-host-namespace ingress: - from: # Example allowing vcluster workload traffic from all pods in the my-host-namespace namespace. # Depending on your use case, a more restrictive pod selector may be used. - namespaceSelector: matchLabels: kubernetes.io/metadata.name: my-host-namespace ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `replicateServices` object [​](#replicateServices "Direct link to replicateServices")ReplicateServices allows replicating services from the host within the virtual cluster or the other way around. ### `toHost` object\[] [​](#replicateServices-toHost "Direct link to replicateServices-toHost")ToHost defines the services that should get synced from virtual cluster to the host cluster. If services are synced to a different namespace than the virtual cluster is in, additional permissions for the other namespace are required. #### `from` string [​](#replicateServices-toHost-from "Direct link to replicateServices-toHost-from")From is the service that should get synced. Can be either in the form name or namespace/name. #### `to` string [​](#replicateServices-toHost-to "Direct link to replicateServices-toHost-to")To is the target service that it should get synced to. Can be either in the form name or namespace/name. ### `fromHost` object\[] [​](#replicateServices-fromHost "Direct link to replicateServices-fromHost")FromHost defines the services that should get synced from the host to the virtual cluster. #### `from` string [​](#replicateServices-fromHost-from "Direct link to replicateServices-fromHost-from")From is the service that should get synced. Can be either in the form name or namespace/name. #### `to` string [​](#replicateServices-fromHost-to "Direct link to replicateServices-fromHost-to")To is the target service that it should get synced to. Can be either in the form name or namespace/name. --- # Resolve DNS Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** This feature enables adding custom DNS rules to the tenant cluster to allow communication with services deployed in the control plane cluster and across services in separate vCluster instances. ## Examples[​](#examples "Direct link to Examples") note Enable embedded CoreDNS to ensure DNS resolution works inside the vCluster. Without this setting, DNS queries inside the vCluster, such as resolving services or external domains, fail. Ensure the following is added to `vcluster.yaml` or Helm values file: ``` controlPlane: coredns: enabled: true embedded: true ``` ### Map a hostname[​](#map-a-hostname "Direct link to Map a hostname") This is a URL-based mapping of one tenant cluster hostname to another hostname. A `wikipedia.com` DNS lookup would return a DNS response with answer as `en.wikipedia.org`. ``` controlPlane: coredns: enabled: true embedded: true networking: resolveDNS: - hostname: wikipedia.com target: hostname: en.wikipedia.org ``` ### Map a hostname wildcard[​](#map-a-hostname-wildcard "Direct link to Map a hostname wildcard") This is a URL-based mapping of one tenant cluster hostname to another hostname. A `test.svc.kubernetes` DNS lookup would return a DNS response with answer as `test.svc.cluster.local`. ``` controlPlane: coredns: enabled: true embedded: true networking: resolveDNS: - hostname: *.svc.kubernetes target: hostname: *.svc.cluster.local ``` ### Map a tenant cluster service to a control plane cluster service[​](#map-a-tenant-cluster-service-to-a-control-plane-cluster-service "Direct link to Map a tenant cluster service to a control plane cluster service") This example maps the tenant cluster's `my-namespace/my-svc` resource to the control plane cluster's `dns-test/nginx-svc` resource. The DNS response is the `nginx-svc` IP in the host's `dns-test` namespace. ``` controlPlane: coredns: enabled: true embedded: true networking: resolveDNS: - service: my-namespace/my-svc target: hostService: dns-test/nginx-svc ``` Kubelet image pulls aren't covered by this mapping This mapping is a tenant cluster CoreDNS record, so it works for a process inside a pod that resolves the hostname itself, such as an application calling a private registry directly. It doesn't help the kubelet pull container images referenced in a pod's `image:` field. On shared nodes, the container runtime uses the underlying node's DNS, not the tenant cluster's CoreDNS. See [Use a control plane cluster registry from a tenant cluster](https://vcluster.com/docs/vcluster/learn-how-to/in-cluster-registry.md). ### Map services across vCluster instances[​](#map-services-across-vcluster-instances "Direct link to Map services across vCluster instances") This example maps a tenant cluster Service to another Service in a separate tenant cluster. `my-ns-in-vcluster/my-svc-vcluster` maps to `dns-test-in-vcluster-ns/test-in-vcluster-service` in a vCluster instance named `test-cluster` deployed in the host namespace `test-vcluster-ns`. ``` controlPlane: coredns: enabled: true embedded: true networking: resolveDNS: - service: my-ns-in-vcluster/my-svc-vcluster target: vClusterService: test-vcluster-ns/test-vcluster/dns-test-in-vcluster-ns/test-in-vcluster-service ``` ### Map namespaces[​](#map-namespaces "Direct link to Map namespaces") Map all services under a tenant cluster namespace to a host namespace. This host namespace could also contain another vCluster instance, thereby mapping all vCluster services to another vCluster instance. ``` controlPlane: coredns: enabled: true embedded: true networking: resolveDNS: - namespace: test-in-vcluster-ns target: hostNamespace: external-vc-ns ``` All services in the `test-vcluster` tenant cluster's target namespace `test-in-vcluster-ns` map to namespace `external-vc-ns`. ``` ┌──────────────────────┐┌──────────────────────────────┐ │nginx.external-vc-ns ││nginx.test-in-vcluster-ns │ │svcA.external-vc-ns ││svcA.test-in-vcluster-ns │ │svcB.external-vc-ns ││svcB.test-in-vcluster-ns │ │... ││... │ └──────────────────────┘└──────────────────────────────┘ ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `resolveDNS` object\[] [​](#resolveDNS "Direct link to resolveDNS")ResolveDNS allows to define extra DNS rules. This only works if embedded coredns is configured. ### `hostname` string [​](#resolveDNS-hostname "Direct link to resolveDNS-hostname")Hostname is the hostname within the vCluster that should be resolved from. ### `service` string [​](#resolveDNS-service "Direct link to resolveDNS-service")Service is the virtual cluster service that should be resolved from. ### `namespace` string [​](#resolveDNS-namespace "Direct link to resolveDNS-namespace")Namespace is the virtual cluster namespace that should be resolved from. ### `target` object [​](#resolveDNS-target "Direct link to resolveDNS-target")Target is the DNS target that should get mapped to #### `hostname` string [​](#resolveDNS-target-hostname "Direct link to resolveDNS-target-hostname")Hostname to use as a DNS target #### `ip` string [​](#resolveDNS-target-ip "Direct link to resolveDNS-target-ip")IP to use as a DNS target #### `hostService` string [​](#resolveDNS-target-hostService "Direct link to resolveDNS-target-hostService")HostService to target, format is hostNamespace/hostService #### `hostNamespace` string [​](#resolveDNS-target-hostNamespace "Direct link to resolveDNS-target-hostNamespace")HostNamespace to target #### `vClusterService` string [​](#resolveDNS-target-vClusterService "Direct link to resolveDNS-target-vClusterService")VClusterService format is hostNamespace/vClusterName/vClusterNamespace/vClusterService --- # Platform You can connect to [vCluster Platform](https://vcluster.com/docs/platform/.md) through the `vcluster.yaml` and configure different vCluster Platform settings that work with the tenant cluster. ## Config reference[​](#config-reference "Direct link to Config reference") ## `platform` object [​](#platform "Direct link to platform")Platform holds vCluster Platform specific configuration. ### `apiKey` object [​](#platform-apiKey "Direct link to platform-apiKey")APIKey defines where to find the platform access key and host. By default, vCluster will search in the following locations in this precedence:- environment variable called LICENSE - secret specified under platform.apiKey.secretName - secret called "vcluster-platform-api-key" in the vCluster namespace #### `secretName` string [​](#platform-apiKey-secretName "Direct link to platform-apiKey-secretName")SecretName is the name of the secret where the platform access key is stored. This defaults to vcluster-platform-api-key if undefined. #### `namespace` string [​](#platform-apiKey-namespace "Direct link to platform-apiKey-namespace")Namespace defines the namespace where the access key secret should be retrieved from. If this is not equal to the namespace where the vCluster instance is deployed, you need to make sure vCluster has access to this other namespace. #### `createRBAC` boolean [​](#platform-apiKey-createRBAC "Direct link to platform-apiKey-createRBAC")CreateRBAC will automatically create the necessary RBAC roles and role bindings to allow vCluster to read the secret specified in the above namespace, if specified. This defaults to true. ### `project` string [​](#platform-project "Direct link to platform-project")Project specifies which platform project the vcluster should be imported to --- # Plugins Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Don't install untrusted plugins A plugin runs with the same permissions as vCluster itself does in the Kubernetes cluster. Plugins can also define additional permissions, so make sure you only install plugins you trust. Plugins overview Plugins extend the capabilities of vCluster. They let you to add custom capability, such as: 1. Syncing specific resources from or to the tenant clusters, including cluster scoped resources like cluster roles 2. Syncing custom resources from or to the tenant cluster 3. Deploying resources on tenant cluster startup, such as CRDs, applications, etc. 4. Manage resources and applications inside the control plane or tenant cluster 5. Enforcing certain restrictions on synced resources or extending the existing syncers of vCluster 6. Any other operator use case that could benefit from having access to the tenant cluster and the control plane cluster simultaneously. A plugin in its purest form is a [Kubernetes operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) that has access to both the tenant cluster and the control plane cluster simultaneously. This is the main difference between a vCluster plugin and a regular Kubernetes operator that you would install inside the tenant cluster itself. Given this dual access, the plugin is able to translate resources between both clusters, which is the basic building block of how vCluster works. Recommended Reads In order to better understand how vCluster plugins work, read about Kubernetes [operators](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) and [controllers](https://kubernetes.io/docs/concepts/architecture/controller/). ## Architecture[​](#architecture "Direct link to Architecture") Each plugin runs as a binary inside the vCluster syncer container. It is by default copied over from an init container. This is done to allow easier communication between vCluster and the plugins as well as provide certain capabilities such as high-availability out of the box. The plugin itself uses [go-plugin](https://github.com/hashicorp/go-plugin), which is used by many popular open source projects to extend their capability. The vCluster syncer calls the plugin binary during startup and passes relevant information to the plugin. The plugin controllers are then started with these credentials, similar to how vCluster itself starts its resource syncers. ### Plugin controllers[​](#plugin-controllers "Direct link to Plugin controllers") Resource syncing enables the tenant cluster to behave like an actual Kubernetes cluster. A [Kubernetes controller](https://kubernetes.io/docs/concepts/architecture/controller/) that is responsible for resource syncing in vCluster is called a syncer. This controller reacts on changes to objects within the tenant cluster and on changes to objects within the control plane cluster. The syncer tries to map each virtual object to a physical object in the control plane cluster and then compares those. After it discovers a change, the syncer ensures that the tenant cluster object and the physical cluster object are aligned in the desired state, and if not, the syncer changes either one of those objects to reflect the desired state. Each plugin can define several of those resource syncers that would work exactly like the built-in syncers of vCluster. However, you don't need to sync every Kubernetes resource to the control plane cluster, as some can stay purely virtual. Only resources that influence the workloads need to be synced, for example, pods, services, and endpoints, while others such as deployments, replicasets, namespaces etc. are only relevant to the Kubernetes control plane and hence are not needed in the control plane cluster. There are sometimes also cases where you want to manage specific core resources yourself without interfering with what vCluster is syncing, for example special secrets or configmaps that were created from the control plane cluster or a different resource inside the control plane cluster. For this use case you can label resources vCluster should ignore either on the physical or virtual side with a label `vcluster.loft.sh/controlled-by` and a custom value of your choosing. This tells vCluster to ignore the resource in its syncers. ### Plugin hooks[​](#plugin-hooks "Direct link to Plugin hooks") Plugin hooks are a great feature to adjust current syncing behavior of vCluster without the need to override an already existing syncer in vCluster completely. They allow you to change outgoing objects of vCluster similar to a mutating admission controller in Kubernetes. Requirement for a hook to work correctly is that vCluster itself would sync the resource, so hooks only work for the core resources that are synced by vCluster such as pods, services, secrets etc. If a plugin registers a hook to a specific resource, vCluster forwards all requests that match the plugin's defined hooks to the plugin and the plugin can then adjust or even deny the request completely. This opens up a wide variety of adjustment possibilities for plugins, where you for example only want to add a custom label or annotation. ### Custom plugins[​](#custom-plugins "Direct link to Custom plugins") You can also develop custom plugins. For more information, see the [Custom Plugins guide](https://vcluster.com/docs/vcluster/integrations/custom-plugins.md). Most of the settings are for the init container from which vCluster copies the plugin. Adjust `controlPlane.statefulSet.resources` [configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/deployment/statefulset.md) to ensure the vCluster deployment has enough CPU and memory to also run the plugin. note * `plugins[*].resources` does not change the syncer's resource requests. * `plugins[*].config` content ends up in an environment variable `PLUGIN_CONFIG` that is read by the plugin. This configuration has no specific format. ## Config reference[​](#config-reference "Direct link to Config reference") ## `plugins` {key: object} [​](#plugins "Direct link to plugins")Define which vCluster plugins to load. ### `name` string [​](#plugins-name "Direct link to plugins-name")Name is the name of the init-container and NOT the plugin name ### `image` string [​](#plugins-image "Direct link to plugins-image")Image is the container image that should be used for the plugin ### `imagePullPolicy` string [​](#plugins-imagePullPolicy "Direct link to plugins-imagePullPolicy")ImagePullPolicy is the pull policy to use for the container image ### `config` object [​](#plugins-config "Direct link to plugins-config")Config is the plugin config to use. This can be arbitrary config used for the plugin. ### `rbac` object [​](#plugins-rbac "Direct link to plugins-rbac")RBAC holds additional rbac configuration for the plugin #### `role` object [​](#plugins-rbac-role "Direct link to plugins-rbac-role")Role holds extra virtual cluster role permissions for the plugin ##### `extraRules` object\[] [​](#plugins-rbac-role-extraRules "Direct link to plugins-rbac-role-extraRules")ExtraRules are extra rbac permissions roles that will be added to role or cluster role ##### `verbs` string\[] [​](#plugins-rbac-role-extraRules-verbs "Direct link to plugins-rbac-role-extraRules-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds contained in this rule. '\*' represents all verbs. ##### `apiGroups` string\[] [​](#plugins-rbac-role-extraRules-apiGroups "Direct link to plugins-rbac-role-extraRules-apiGroups")APIGroups is the name of the APIGroup that contains the resources. If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed. "" represents the core API group and "\*" represents all API groups. ##### `resources` string\[] [​](#plugins-rbac-role-extraRules-resources "Direct link to plugins-rbac-role-extraRules-resources")Resources is a list of resources this rule applies to. '\*' represents all resources. ##### `resourceNames` string\[] [​](#plugins-rbac-role-extraRules-resourceNames "Direct link to plugins-rbac-role-extraRules-resourceNames")ResourceNames is an optional white list of names that the rule applies to. An empty set means that everything is allowed. ##### `nonResourceURLs` string\[] [​](#plugins-rbac-role-extraRules-nonResourceURLs "Direct link to plugins-rbac-role-extraRules-nonResourceURLs")NonResourceURLs is a set of partial urls that a user should have access to. \*s are allowed, but only as the full, final step in the path Since non-resource URLs are not namespaced, this field is only applicable for ClusterRoles referenced from a ClusterRoleBinding. Rules can either apply to API resources (such as "pods" or "secrets") or non-resource URL paths (such as "/api"), but not both. #### `clusterRole` object [​](#plugins-rbac-clusterRole "Direct link to plugins-rbac-clusterRole")ClusterRole holds extra virtual cluster cluster role permissions required for the plugin ##### `extraRules` object\[] [​](#plugins-rbac-clusterRole-extraRules "Direct link to plugins-rbac-clusterRole-extraRules")ExtraRules are extra rbac permissions roles that will be added to role or cluster role ##### `verbs` string\[] [​](#plugins-rbac-clusterRole-extraRules-verbs "Direct link to plugins-rbac-clusterRole-extraRules-verbs")Verbs is a list of Verbs that apply to ALL the ResourceKinds contained in this rule. '\*' represents all verbs. ##### `apiGroups` string\[] [​](#plugins-rbac-clusterRole-extraRules-apiGroups "Direct link to plugins-rbac-clusterRole-extraRules-apiGroups")APIGroups is the name of the APIGroup that contains the resources. If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed. "" represents the core API group and "\*" represents all API groups. ##### `resources` string\[] [​](#plugins-rbac-clusterRole-extraRules-resources "Direct link to plugins-rbac-clusterRole-extraRules-resources")Resources is a list of resources this rule applies to. '\*' represents all resources. ##### `resourceNames` string\[] [​](#plugins-rbac-clusterRole-extraRules-resourceNames "Direct link to plugins-rbac-clusterRole-extraRules-resourceNames")ResourceNames is an optional white list of names that the rule applies to. An empty set means that everything is allowed. ##### `nonResourceURLs` string\[] [​](#plugins-rbac-clusterRole-extraRules-nonResourceURLs "Direct link to plugins-rbac-clusterRole-extraRules-nonResourceURLs")NonResourceURLs is a set of partial urls that a user should have access to. \*s are allowed, but only as the full, final step in the path Since non-resource URLs are not namespaced, this field is only applicable for ClusterRoles referenced from a ClusterRoleBinding. Rules can either apply to API resources (such as "pods" or "secrets") or non-resource URL paths (such as "/api"), but not both. ### `command` string\[] [​](#plugins-command "Direct link to plugins-command")Command is the command that should be used for the init container ### `args` string\[] [​](#plugins-args "Direct link to plugins-args")Args are the arguments that should be used for the init container ### `securityContext` object [​](#plugins-securityContext "Direct link to plugins-securityContext")SecurityContext is the container security context used for the init container ### `resources` object [​](#plugins-resources "Direct link to plugins-resources")Resources are the container resources used for the init container ### `volumeMounts` object\[] [​](#plugins-volumeMounts "Direct link to plugins-volumeMounts")VolumeMounts are extra volume mounts for the init container --- # Policies Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** info Network policies in the tenant cluster rely on the support for this feature in the control plane cluster. Make sure that your control plane cluster satisfies the [network policy prerequisites](https://kubernetes.io/docs/concepts/services-networking/network-policies/#prerequisites). Policies cover several different topics: * [Limit ranges](https://kubernetes.io/docs/concepts/policy/limit-range/) control the storage / CPU / memory that each pod may request. * [Resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/). * [Pod security standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/) * [Network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) like network isolation. * [Admission webhooks](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks) * [Central Admission Control](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/admission-control.md) * The admission controller is typically running on the control plane cluster, where the policies enforced by the webhook cannot be changed by the tenant cluster. * Users could deploy their own admission webhooks to the tenant cluster, but there's little value in doing so, and this configuration is not concerned with that use case. * Some examples of admission controller projects: * [OPA](https://www.openpolicyagent.org/docs/latest/) * [Kyverno](https://kyverno.io/) * [jsPolicy](https://www.jspolicy.com/) * It's common for organizations to develop an in-house collection of policies that can enforce naming standards and so on. You can use these settings separately for specific cases, or together, as in the case of Isolated Mode. ## Config reference[​](#config-reference "Direct link to Config reference") ## `policies` object [​](#policies "Direct link to policies")Policies to enforce for the virtual cluster deployment as well as within the virtual cluster. ### `networkPolicy` object [​](#policies-networkPolicy "Direct link to policies-networkPolicy")NetworkPolicy specifies network policy options. #### `enabled` boolean false [​](#policies-networkPolicy-enabled "Direct link to policies-networkPolicy-enabled")Enabled defines if the network policy should be deployed by vCluster. #### `annotations` object {} [​](#policies-networkPolicy-annotations "Direct link to policies-networkPolicy-annotations")Annotations are extra annotations for this resource. #### `labels` object {} [​](#policies-networkPolicy-labels "Direct link to policies-networkPolicy-labels")Labels are extra labels for this resource. #### `fallbackDns` string 8.8.8.8 [​](#policies-networkPolicy-fallbackDns "Direct link to policies-networkPolicy-fallbackDns")FallbackDNS is the fallback DNS server to use if the virtual cluster does not have a DNS server. #### `controlPlane` object [​](#policies-networkPolicy-controlPlane "Direct link to policies-networkPolicy-controlPlane")ControlPlane network policy rules ##### `ingress` object\[] [​](#policies-networkPolicy-controlPlane-ingress "Direct link to policies-networkPolicy-controlPlane-ingress")Ingress rules for the vCluster control plane. ##### `ports` object\[] [​](#policies-networkPolicy-controlPlane-ingress-ports "Direct link to policies-networkPolicy-controlPlane-ingress-ports")ports is a list of ports which should be made accessible on the pods selected for this rule. Each item in this list is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list. ##### `protocol` string [​](#policies-networkPolicy-controlPlane-ingress-ports-protocol "Direct link to policies-networkPolicy-controlPlane-ingress-ports-protocol")protocol represents the protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this field defaults to TCP. ##### `port` object [​](#policies-networkPolicy-controlPlane-ingress-ports-port "Direct link to policies-networkPolicy-controlPlane-ingress-ports-port")port represents the port on the given protocol. This can either be a numerical or named port on a pod. If this field is not provided, this matches all port names and numbers. If present, only traffic on the specified protocol AND port will be matched. ##### `endPort` integer [​](#policies-networkPolicy-controlPlane-ingress-ports-endPort "Direct link to policies-networkPolicy-controlPlane-ingress-ports-endPort")endPort indicates that the range of ports from port to endPort if set, inclusive, should be allowed by the policy. This field cannot be defined if the port field is not defined or if the port field is defined as a named (string) port. The endPort must be equal or greater than port. ##### `from` object\[] [​](#policies-networkPolicy-controlPlane-ingress-from "Direct link to policies-networkPolicy-controlPlane-ingress-from")from is a list of sources which should be able to access the pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all sources (traffic not restricted by source). If this field is present and contains at least one item, this rule allows traffic only if the traffic matches at least one item in the from list. ##### `podSelector` object [​](#policies-networkPolicy-controlPlane-ingress-from-podSelector "Direct link to policies-networkPolicy-controlPlane-ingress-from-podSelector")podSelector is a label selector which selects pods. This field follows standard label selector semantics; if present but empty, it selects all pods.If namespaceSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the Namespaces selected by NamespaceSelector. Otherwise it selects the pods matching podSelector in the policy's own namespace. ##### `matchLabels` object [​](#policies-networkPolicy-controlPlane-ingress-from-podSelector-matchLabels "Direct link to policies-networkPolicy-controlPlane-ingress-from-podSelector-matchLabels") ##### `matchExpressions` object\[] [​](#policies-networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions "Direct link to policies-networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions") ##### `key` string [​](#policies-networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions-key "Direct link to policies-networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions-key") ##### `operator` string [​](#policies-networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions-operator "Direct link to policies-networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions-operator") ##### `values` string\[] [​](#policies-networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions-values "Direct link to policies-networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions-values") ##### `namespaceSelector` object [​](#policies-networkPolicy-controlPlane-ingress-from-namespaceSelector "Direct link to policies-networkPolicy-controlPlane-ingress-from-namespaceSelector")namespaceSelector selects namespaces using cluster-scoped labels. This field follows standard label selector semantics; if present but empty, it selects all namespaces.If podSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the namespaces selected by namespaceSelector. Otherwise it selects all pods in the namespaces selected by namespaceSelector. ##### `matchLabels` object [​](#policies-networkPolicy-controlPlane-ingress-from-namespaceSelector-matchLabels "Direct link to policies-networkPolicy-controlPlane-ingress-from-namespaceSelector-matchLabels") ##### `matchExpressions` object\[] [​](#policies-networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions "Direct link to policies-networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions") ##### `key` string [​](#policies-networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions-key "Direct link to policies-networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions-key") ##### `operator` string [​](#policies-networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions-operator "Direct link to policies-networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions-operator") ##### `values` string\[] [​](#policies-networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions-values "Direct link to policies-networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions-values") ##### `ipBlock` object [​](#policies-networkPolicy-controlPlane-ingress-from-ipBlock "Direct link to policies-networkPolicy-controlPlane-ingress-from-ipBlock")ipBlock defines policy on a particular IPBlock. If this field is set then neither of the other fields can be. ##### `cidr` string [​](#policies-networkPolicy-controlPlane-ingress-from-ipBlock-cidr "Direct link to policies-networkPolicy-controlPlane-ingress-from-ipBlock-cidr")CIDR defines the allowed workload public egress destination. Valid examples are "0.0.0.0/0", "192.168.1.0/24" or "2001:db8::/64" ##### `except` string\[] [​](#policies-networkPolicy-controlPlane-ingress-from-ipBlock-except "Direct link to policies-networkPolicy-controlPlane-ingress-from-ipBlock-except")Except is a slice of CIDRs that should not be included. Items outside the cidr range will be rejected. Valid examples are "192.168.1.0/24" or "2001:db8::/64". ##### `egress` object\[] [​](#policies-networkPolicy-controlPlane-egress "Direct link to policies-networkPolicy-controlPlane-egress")Egress rules for the vCluster control plane. ##### `ports` object\[] [​](#policies-networkPolicy-controlPlane-egress-ports "Direct link to policies-networkPolicy-controlPlane-egress-ports")ports is a list of destination ports for outgoing traffic. Each item in this list is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list. ##### `protocol` string [​](#policies-networkPolicy-controlPlane-egress-ports-protocol "Direct link to policies-networkPolicy-controlPlane-egress-ports-protocol")protocol represents the protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this field defaults to TCP. ##### `port` object [​](#policies-networkPolicy-controlPlane-egress-ports-port "Direct link to policies-networkPolicy-controlPlane-egress-ports-port")port represents the port on the given protocol. This can either be a numerical or named port on a pod. If this field is not provided, this matches all port names and numbers. If present, only traffic on the specified protocol AND port will be matched. ##### `endPort` integer [​](#policies-networkPolicy-controlPlane-egress-ports-endPort "Direct link to policies-networkPolicy-controlPlane-egress-ports-endPort")endPort indicates that the range of ports from port to endPort if set, inclusive, should be allowed by the policy. This field cannot be defined if the port field is not defined or if the port field is defined as a named (string) port. The endPort must be equal or greater than port. ##### `to` object\[] [​](#policies-networkPolicy-controlPlane-egress-to "Direct link to policies-networkPolicy-controlPlane-egress-to")to is a list of destinations for outgoing traffic of pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all destinations (traffic not restricted by destination). If this field is present and contains at least one item, this rule allows traffic only if the traffic matches at least one item in the to list. ##### `podSelector` object [​](#policies-networkPolicy-controlPlane-egress-to-podSelector "Direct link to policies-networkPolicy-controlPlane-egress-to-podSelector")podSelector is a label selector which selects pods. This field follows standard label selector semantics; if present but empty, it selects all pods.If namespaceSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the Namespaces selected by NamespaceSelector. Otherwise it selects the pods matching podSelector in the policy's own namespace. ##### `matchLabels` object [​](#policies-networkPolicy-controlPlane-egress-to-podSelector-matchLabels "Direct link to policies-networkPolicy-controlPlane-egress-to-podSelector-matchLabels") ##### `matchExpressions` object\[] [​](#policies-networkPolicy-controlPlane-egress-to-podSelector-matchExpressions "Direct link to policies-networkPolicy-controlPlane-egress-to-podSelector-matchExpressions") ##### `key` string [​](#policies-networkPolicy-controlPlane-egress-to-podSelector-matchExpressions-key "Direct link to policies-networkPolicy-controlPlane-egress-to-podSelector-matchExpressions-key") ##### `operator` string [​](#policies-networkPolicy-controlPlane-egress-to-podSelector-matchExpressions-operator "Direct link to policies-networkPolicy-controlPlane-egress-to-podSelector-matchExpressions-operator") ##### `values` string\[] [​](#policies-networkPolicy-controlPlane-egress-to-podSelector-matchExpressions-values "Direct link to policies-networkPolicy-controlPlane-egress-to-podSelector-matchExpressions-values") ##### `namespaceSelector` object [​](#policies-networkPolicy-controlPlane-egress-to-namespaceSelector "Direct link to policies-networkPolicy-controlPlane-egress-to-namespaceSelector")namespaceSelector selects namespaces using cluster-scoped labels. This field follows standard label selector semantics; if present but empty, it selects all namespaces.If podSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the namespaces selected by namespaceSelector. Otherwise it selects all pods in the namespaces selected by namespaceSelector. ##### `matchLabels` object [​](#policies-networkPolicy-controlPlane-egress-to-namespaceSelector-matchLabels "Direct link to policies-networkPolicy-controlPlane-egress-to-namespaceSelector-matchLabels") ##### `matchExpressions` object\[] [​](#policies-networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions "Direct link to policies-networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions") ##### `key` string [​](#policies-networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions-key "Direct link to policies-networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions-key") ##### `operator` string [​](#policies-networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions-operator "Direct link to policies-networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions-operator") ##### `values` string\[] [​](#policies-networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions-values "Direct link to policies-networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions-values") ##### `ipBlock` object [​](#policies-networkPolicy-controlPlane-egress-to-ipBlock "Direct link to policies-networkPolicy-controlPlane-egress-to-ipBlock")ipBlock defines policy on a particular IPBlock. If this field is set then neither of the other fields can be. ##### `cidr` string [​](#policies-networkPolicy-controlPlane-egress-to-ipBlock-cidr "Direct link to policies-networkPolicy-controlPlane-egress-to-ipBlock-cidr")CIDR defines the allowed workload public egress destination. Valid examples are "0.0.0.0/0", "192.168.1.0/24" or "2001:db8::/64" ##### `except` string\[] [​](#policies-networkPolicy-controlPlane-egress-to-ipBlock-except "Direct link to policies-networkPolicy-controlPlane-egress-to-ipBlock-except")Except is a slice of CIDRs that should not be included. Items outside the cidr range will be rejected. Valid examples are "192.168.1.0/24" or "2001:db8::/64". #### `workload` object [​](#policies-networkPolicy-workload "Direct link to policies-networkPolicy-workload")Workload network policy rules ##### `publicEgress` object [​](#policies-networkPolicy-workload-publicEgress "Direct link to policies-networkPolicy-workload-publicEgress")PublicEgress holds the public outgoing connections options for the vCluster workloads. ##### `enabled` boolean true [​](#policies-networkPolicy-workload-publicEgress-enabled "Direct link to policies-networkPolicy-workload-publicEgress-enabled")Enabled defines if the workload public egress should be enabled or disabled. ##### `cidr` string 0.0.0.0/0 [​](#policies-networkPolicy-workload-publicEgress-cidr "Direct link to policies-networkPolicy-workload-publicEgress-cidr")CIDR defines the allowed workload public egress destination. Valid examples are "0.0.0.0/0", "192.168.1.0/24" or "2001:db8::/64" ##### `except` string\[] \[100.64.0.0/10 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16] [​](#policies-networkPolicy-workload-publicEgress-except "Direct link to policies-networkPolicy-workload-publicEgress-except")Except is a slice of CIDRs that should not be included. Items outside the cidr range will be rejected. Valid examples are "192.168.1.0/24" or "2001:db8::/64". ##### `ingress` object\[] [​](#policies-networkPolicy-workload-ingress "Direct link to policies-networkPolicy-workload-ingress")Ingress rules for the vCluster workloads. ##### `ports` object\[] [​](#policies-networkPolicy-workload-ingress-ports "Direct link to policies-networkPolicy-workload-ingress-ports")ports is a list of ports which should be made accessible on the pods selected for this rule. Each item in this list is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list. ##### `protocol` string [​](#policies-networkPolicy-workload-ingress-ports-protocol "Direct link to policies-networkPolicy-workload-ingress-ports-protocol")protocol represents the protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this field defaults to TCP. ##### `port` object [​](#policies-networkPolicy-workload-ingress-ports-port "Direct link to policies-networkPolicy-workload-ingress-ports-port")port represents the port on the given protocol. This can either be a numerical or named port on a pod. If this field is not provided, this matches all port names and numbers. If present, only traffic on the specified protocol AND port will be matched. ##### `endPort` integer [​](#policies-networkPolicy-workload-ingress-ports-endPort "Direct link to policies-networkPolicy-workload-ingress-ports-endPort")endPort indicates that the range of ports from port to endPort if set, inclusive, should be allowed by the policy. This field cannot be defined if the port field is not defined or if the port field is defined as a named (string) port. The endPort must be equal or greater than port. ##### `from` object\[] [​](#policies-networkPolicy-workload-ingress-from "Direct link to policies-networkPolicy-workload-ingress-from")from is a list of sources which should be able to access the pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all sources (traffic not restricted by source). If this field is present and contains at least one item, this rule allows traffic only if the traffic matches at least one item in the from list. ##### `podSelector` object [​](#policies-networkPolicy-workload-ingress-from-podSelector "Direct link to policies-networkPolicy-workload-ingress-from-podSelector")podSelector is a label selector which selects pods. This field follows standard label selector semantics; if present but empty, it selects all pods.If namespaceSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the Namespaces selected by NamespaceSelector. Otherwise it selects the pods matching podSelector in the policy's own namespace. ##### `matchLabels` object [​](#policies-networkPolicy-workload-ingress-from-podSelector-matchLabels "Direct link to policies-networkPolicy-workload-ingress-from-podSelector-matchLabels") ##### `matchExpressions` object\[] [​](#policies-networkPolicy-workload-ingress-from-podSelector-matchExpressions "Direct link to policies-networkPolicy-workload-ingress-from-podSelector-matchExpressions") ##### `key` string [​](#policies-networkPolicy-workload-ingress-from-podSelector-matchExpressions-key "Direct link to policies-networkPolicy-workload-ingress-from-podSelector-matchExpressions-key") ##### `operator` string [​](#policies-networkPolicy-workload-ingress-from-podSelector-matchExpressions-operator "Direct link to policies-networkPolicy-workload-ingress-from-podSelector-matchExpressions-operator") ##### `values` string\[] [​](#policies-networkPolicy-workload-ingress-from-podSelector-matchExpressions-values "Direct link to policies-networkPolicy-workload-ingress-from-podSelector-matchExpressions-values") ##### `namespaceSelector` object [​](#policies-networkPolicy-workload-ingress-from-namespaceSelector "Direct link to policies-networkPolicy-workload-ingress-from-namespaceSelector")namespaceSelector selects namespaces using cluster-scoped labels. This field follows standard label selector semantics; if present but empty, it selects all namespaces.If podSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the namespaces selected by namespaceSelector. Otherwise it selects all pods in the namespaces selected by namespaceSelector. ##### `matchLabels` object [​](#policies-networkPolicy-workload-ingress-from-namespaceSelector-matchLabels "Direct link to policies-networkPolicy-workload-ingress-from-namespaceSelector-matchLabels") ##### `matchExpressions` object\[] [​](#policies-networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions "Direct link to policies-networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions") ##### `key` string [​](#policies-networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions-key "Direct link to policies-networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions-key") ##### `operator` string [​](#policies-networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions-operator "Direct link to policies-networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions-operator") ##### `values` string\[] [​](#policies-networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions-values "Direct link to policies-networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions-values") ##### `ipBlock` object [​](#policies-networkPolicy-workload-ingress-from-ipBlock "Direct link to policies-networkPolicy-workload-ingress-from-ipBlock")ipBlock defines policy on a particular IPBlock. If this field is set then neither of the other fields can be. ##### `cidr` string [​](#policies-networkPolicy-workload-ingress-from-ipBlock-cidr "Direct link to policies-networkPolicy-workload-ingress-from-ipBlock-cidr")CIDR defines the allowed workload public egress destination. Valid examples are "0.0.0.0/0", "192.168.1.0/24" or "2001:db8::/64" ##### `except` string\[] [​](#policies-networkPolicy-workload-ingress-from-ipBlock-except "Direct link to policies-networkPolicy-workload-ingress-from-ipBlock-except")Except is a slice of CIDRs that should not be included. Items outside the cidr range will be rejected. Valid examples are "192.168.1.0/24" or "2001:db8::/64". ##### `egress` object\[] [​](#policies-networkPolicy-workload-egress "Direct link to policies-networkPolicy-workload-egress")Egress rules for the vCluster workloads. ##### `ports` object\[] [​](#policies-networkPolicy-workload-egress-ports "Direct link to policies-networkPolicy-workload-egress-ports")ports is a list of destination ports for outgoing traffic. Each item in this list is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list. ##### `protocol` string [​](#policies-networkPolicy-workload-egress-ports-protocol "Direct link to policies-networkPolicy-workload-egress-ports-protocol")protocol represents the protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this field defaults to TCP. ##### `port` object [​](#policies-networkPolicy-workload-egress-ports-port "Direct link to policies-networkPolicy-workload-egress-ports-port")port represents the port on the given protocol. This can either be a numerical or named port on a pod. If this field is not provided, this matches all port names and numbers. If present, only traffic on the specified protocol AND port will be matched. ##### `endPort` integer [​](#policies-networkPolicy-workload-egress-ports-endPort "Direct link to policies-networkPolicy-workload-egress-ports-endPort")endPort indicates that the range of ports from port to endPort if set, inclusive, should be allowed by the policy. This field cannot be defined if the port field is not defined or if the port field is defined as a named (string) port. The endPort must be equal or greater than port. ##### `to` object\[] [​](#policies-networkPolicy-workload-egress-to "Direct link to policies-networkPolicy-workload-egress-to")to is a list of destinations for outgoing traffic of pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all destinations (traffic not restricted by destination). If this field is present and contains at least one item, this rule allows traffic only if the traffic matches at least one item in the to list. ##### `podSelector` object [​](#policies-networkPolicy-workload-egress-to-podSelector "Direct link to policies-networkPolicy-workload-egress-to-podSelector")podSelector is a label selector which selects pods. This field follows standard label selector semantics; if present but empty, it selects all pods.If namespaceSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the Namespaces selected by NamespaceSelector. Otherwise it selects the pods matching podSelector in the policy's own namespace. ##### `matchLabels` object [​](#policies-networkPolicy-workload-egress-to-podSelector-matchLabels "Direct link to policies-networkPolicy-workload-egress-to-podSelector-matchLabels") ##### `matchExpressions` object\[] [​](#policies-networkPolicy-workload-egress-to-podSelector-matchExpressions "Direct link to policies-networkPolicy-workload-egress-to-podSelector-matchExpressions") ##### `key` string [​](#policies-networkPolicy-workload-egress-to-podSelector-matchExpressions-key "Direct link to policies-networkPolicy-workload-egress-to-podSelector-matchExpressions-key") ##### `operator` string [​](#policies-networkPolicy-workload-egress-to-podSelector-matchExpressions-operator "Direct link to policies-networkPolicy-workload-egress-to-podSelector-matchExpressions-operator") ##### `values` string\[] [​](#policies-networkPolicy-workload-egress-to-podSelector-matchExpressions-values "Direct link to policies-networkPolicy-workload-egress-to-podSelector-matchExpressions-values") ##### `namespaceSelector` object [​](#policies-networkPolicy-workload-egress-to-namespaceSelector "Direct link to policies-networkPolicy-workload-egress-to-namespaceSelector")namespaceSelector selects namespaces using cluster-scoped labels. This field follows standard label selector semantics; if present but empty, it selects all namespaces.If podSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the namespaces selected by namespaceSelector. Otherwise it selects all pods in the namespaces selected by namespaceSelector. ##### `matchLabels` object [​](#policies-networkPolicy-workload-egress-to-namespaceSelector-matchLabels "Direct link to policies-networkPolicy-workload-egress-to-namespaceSelector-matchLabels") ##### `matchExpressions` object\[] [​](#policies-networkPolicy-workload-egress-to-namespaceSelector-matchExpressions "Direct link to policies-networkPolicy-workload-egress-to-namespaceSelector-matchExpressions") ##### `key` string [​](#policies-networkPolicy-workload-egress-to-namespaceSelector-matchExpressions-key "Direct link to policies-networkPolicy-workload-egress-to-namespaceSelector-matchExpressions-key") ##### `operator` string [​](#policies-networkPolicy-workload-egress-to-namespaceSelector-matchExpressions-operator "Direct link to policies-networkPolicy-workload-egress-to-namespaceSelector-matchExpressions-operator") ##### `values` string\[] [​](#policies-networkPolicy-workload-egress-to-namespaceSelector-matchExpressions-values "Direct link to policies-networkPolicy-workload-egress-to-namespaceSelector-matchExpressions-values") ##### `ipBlock` object [​](#policies-networkPolicy-workload-egress-to-ipBlock "Direct link to policies-networkPolicy-workload-egress-to-ipBlock")ipBlock defines policy on a particular IPBlock. If this field is set then neither of the other fields can be. ##### `cidr` string [​](#policies-networkPolicy-workload-egress-to-ipBlock-cidr "Direct link to policies-networkPolicy-workload-egress-to-ipBlock-cidr")CIDR defines the allowed workload public egress destination. Valid examples are "0.0.0.0/0", "192.168.1.0/24" or "2001:db8::/64" ##### `except` string\[] [​](#policies-networkPolicy-workload-egress-to-ipBlock-except "Direct link to policies-networkPolicy-workload-egress-to-ipBlock-except")Except is a slice of CIDRs that should not be included. Items outside the cidr range will be rejected. Valid examples are "192.168.1.0/24" or "2001:db8::/64". ### `podSecurityStandard` string [​](#policies-podSecurityStandard "Direct link to policies-podSecurityStandard")PodSecurityStandard that can be enforced can be one of: empty (""), baseline, restricted or privileged ### `resourceQuota` object [​](#policies-resourceQuota "Direct link to policies-resourceQuota")ResourceQuota specifies resource quota options. #### `enabled` string|boolean auto [​](#policies-resourceQuota-enabled "Direct link to policies-resourceQuota-enabled")Enabled defines if the resource quota should be enabled. "auto" means that if limitRange is enabled, the resourceQuota will be enabled as well. #### `quota` object map\[count/configmaps:100 count/endpoints:40 count/persistentvolumeclaims:20 count/pods:20 count/secrets:100 count/services:20 limits.cpu:20 limits.ephemeral-storage:160Gi limits.memory:40Gi requests.cpu:10 requests.ephemeral-storage:60Gi requests.memory:20Gi requests.storage:100Gi services.loadbalancers:1 services.nodeports:0] [​](#policies-resourceQuota-quota "Direct link to policies-resourceQuota-quota")Quota are the quota options #### `scopeSelector` object map\[matchExpressions:\[]] [​](#policies-resourceQuota-scopeSelector "Direct link to policies-resourceQuota-scopeSelector")ScopeSelector is the resource quota scope selector #### `scopes` string\[] \[] [​](#policies-resourceQuota-scopes "Direct link to policies-resourceQuota-scopes")Scopes are the resource quota scopes #### `annotations` object {} [​](#policies-resourceQuota-annotations "Direct link to policies-resourceQuota-annotations")Annotations are extra annotations for this resource. #### `labels` object {} [​](#policies-resourceQuota-labels "Direct link to policies-resourceQuota-labels")Labels are extra labels for this resource. ### `limitRange` object [​](#policies-limitRange "Direct link to policies-limitRange")LimitRange specifies limit range options. #### `enabled` string|boolean auto [​](#policies-limitRange-enabled "Direct link to policies-limitRange-enabled")Enabled defines if the limit range should be deployed by vCluster. "auto" means that if resourceQuota is enabled, the limitRange will be enabled as well. #### `default` object map\[cpu:1 ephemeral-storage:8Gi memory:512Mi] [​](#policies-limitRange-default "Direct link to policies-limitRange-default")Default are the default limits for the limit range #### `defaultRequest` object map\[cpu:100m ephemeral-storage:3Gi memory:128Mi] [​](#policies-limitRange-defaultRequest "Direct link to policies-limitRange-defaultRequest")DefaultRequest are the default request options for the limit range #### `max` object {} [​](#policies-limitRange-max "Direct link to policies-limitRange-max")Max are the max limits for the limit range #### `min` object {} [​](#policies-limitRange-min "Direct link to policies-limitRange-min")Min are the min limits for the limit range #### `annotations` object {} [​](#policies-limitRange-annotations "Direct link to policies-limitRange-annotations")Annotations are extra annotations for this resource. #### `labels` object {} [​](#policies-limitRange-labels "Direct link to policies-limitRange-labels")Labels are extra labels for this resource. ### `centralAdmission` object [​](#policies-centralAdmission "Direct link to policies-centralAdmission")CentralAdmission defines what validating or mutating webhooks should be enforced within the virtual cluster. #### `validatingWebhooks` object\[] [​](#policies-centralAdmission-validatingWebhooks "Direct link to policies-centralAdmission-validatingWebhooks")ValidatingWebhooks are validating webhooks that should be enforced in the virtual cluster ##### `kind` string [​](#policies-centralAdmission-validatingWebhooks-kind "Direct link to policies-centralAdmission-validatingWebhooks-kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. ##### `apiVersion` string [​](#policies-centralAdmission-validatingWebhooks-apiVersion "Direct link to policies-centralAdmission-validatingWebhooks-apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. ##### `metadata` object [​](#policies-centralAdmission-validatingWebhooks-metadata "Direct link to policies-centralAdmission-validatingWebhooks-metadata")Standard object metadata; More info: . ##### `name` string [​](#policies-centralAdmission-validatingWebhooks-metadata-name "Direct link to policies-centralAdmission-validatingWebhooks-metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. ##### `labels` object [​](#policies-centralAdmission-validatingWebhooks-metadata-labels "Direct link to policies-centralAdmission-validatingWebhooks-metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. ##### `annotations` object [​](#policies-centralAdmission-validatingWebhooks-metadata-annotations "Direct link to policies-centralAdmission-validatingWebhooks-metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. ##### `webhooks` object\[] [​](#policies-centralAdmission-validatingWebhooks-webhooks "Direct link to policies-centralAdmission-validatingWebhooks-webhooks")Webhooks is a list of webhooks and the affected resources and operations. ##### `name` string [​](#policies-centralAdmission-validatingWebhooks-webhooks-name "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-name")The name of the admission webhook. Name should be fully qualified, e.g., imagepolicy.kubernetes.io, where "imagepolicy" is the name of the webhook, and kubernetes.io is the name of the organization. ##### `clientConfig` object [​](#policies-centralAdmission-validatingWebhooks-webhooks-clientConfig "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-clientConfig")ClientConfig defines how to communicate with the hook. ##### `url` string [​](#policies-centralAdmission-validatingWebhooks-webhooks-clientConfig-url "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-clientConfig-url")URL gives the location of the webhook, in standard URL form (`scheme://host:port/path`). Exactly one of `url` or `service` must be specified. ##### `service` object [​](#policies-centralAdmission-validatingWebhooks-webhooks-clientConfig-service "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-clientConfig-service")Service is a reference to the service for this webhook. Either `service` or `url` must be specified.If the webhook is running within the cluster, then you should use `service`. ##### `namespace` string [​](#policies-centralAdmission-validatingWebhooks-webhooks-clientConfig-service-namespace "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-clientConfig-service-namespace")Namespace is the namespace of the service. ##### `name` string [​](#policies-centralAdmission-validatingWebhooks-webhooks-clientConfig-service-name "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-clientConfig-service-name")Name is the name of the service. ##### `path` string [​](#policies-centralAdmission-validatingWebhooks-webhooks-clientConfig-service-path "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-clientConfig-service-path")Path is an optional URL path which will be sent in any request to this service. ##### `port` integer [​](#policies-centralAdmission-validatingWebhooks-webhooks-clientConfig-service-port "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-clientConfig-service-port")If specified, the port on the service that hosting webhook. Default to 443 for backward compatibility. `port` should be a valid port number (1-65535, inclusive). ##### `caBundle` string [​](#policies-centralAdmission-validatingWebhooks-webhooks-clientConfig-caBundle "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-clientConfig-caBundle")CABundle is a PEM encoded CA bundle which will be used to validate the webhook's server certificate. If unspecified, system trust roots on the apiserver are used. ##### `rules` object\[] [​](#policies-centralAdmission-validatingWebhooks-webhooks-rules "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-rules")Rules describes what operations on what resources/subresources the webhook cares about. The webhook cares about an operation if it matches *any* Rule. ##### `failurePolicy` string [​](#policies-centralAdmission-validatingWebhooks-webhooks-failurePolicy "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-failurePolicy")FailurePolicy defines how unrecognized errors from the admission endpoint are handled - allowed values are Ignore or Fail. Defaults to Fail. ##### `matchPolicy` string [​](#policies-centralAdmission-validatingWebhooks-webhooks-matchPolicy "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-matchPolicy")matchPolicy defines how the "rules" list is used to match incoming requests. Allowed values are "Exact" or "Equivalent". ##### `namespaceSelector` object [​](#policies-centralAdmission-validatingWebhooks-webhooks-namespaceSelector "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-namespaceSelector")NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the webhook. ##### `objectSelector` object [​](#policies-centralAdmission-validatingWebhooks-webhooks-objectSelector "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-objectSelector")ObjectSelector decides whether to run the webhook based on if the object has matching labels. objectSelector is evaluated against both the oldObject and newObject that would be sent to the webhook, and is considered to match if either object matches the selector. ##### `sideEffects` string [​](#policies-centralAdmission-validatingWebhooks-webhooks-sideEffects "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-sideEffects")SideEffects states whether this webhook has side effects. ##### `timeoutSeconds` integer [​](#policies-centralAdmission-validatingWebhooks-webhooks-timeoutSeconds "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-timeoutSeconds")TimeoutSeconds specifies the timeout for this webhook. ##### `admissionReviewVersions` string\[] [​](#policies-centralAdmission-validatingWebhooks-webhooks-admissionReviewVersions "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-admissionReviewVersions")AdmissionReviewVersions is an ordered list of preferred `AdmissionReview` versions the Webhook expects. ##### `matchConditions` object\[] [​](#policies-centralAdmission-validatingWebhooks-webhooks-matchConditions "Direct link to policies-centralAdmission-validatingWebhooks-webhooks-matchConditions")MatchConditions is a list of conditions that must be met for a request to be sent to this webhook. Match conditions filter requests that have already been matched by the rules, namespaceSelector, and objectSelector. An empty list of matchConditions matches all requests. There are a maximum of 64 match conditions allowed. #### `mutatingWebhooks` object\[] [​](#policies-centralAdmission-mutatingWebhooks "Direct link to policies-centralAdmission-mutatingWebhooks")MutatingWebhooks are mutating webhooks that should be enforced in the virtual cluster ##### `kind` string [​](#policies-centralAdmission-mutatingWebhooks-kind "Direct link to policies-centralAdmission-mutatingWebhooks-kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. ##### `apiVersion` string [​](#policies-centralAdmission-mutatingWebhooks-apiVersion "Direct link to policies-centralAdmission-mutatingWebhooks-apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. ##### `metadata` object [​](#policies-centralAdmission-mutatingWebhooks-metadata "Direct link to policies-centralAdmission-mutatingWebhooks-metadata")Standard object metadata; More info: . ##### `name` string [​](#policies-centralAdmission-mutatingWebhooks-metadata-name "Direct link to policies-centralAdmission-mutatingWebhooks-metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. ##### `labels` object [​](#policies-centralAdmission-mutatingWebhooks-metadata-labels "Direct link to policies-centralAdmission-mutatingWebhooks-metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. ##### `annotations` object [​](#policies-centralAdmission-mutatingWebhooks-metadata-annotations "Direct link to policies-centralAdmission-mutatingWebhooks-metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. ##### `webhooks` object\[] [​](#policies-centralAdmission-mutatingWebhooks-webhooks "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks")Webhooks is a list of webhooks and the affected resources and operations. ##### `reinvocationPolicy` string [​](#policies-centralAdmission-mutatingWebhooks-webhooks-reinvocationPolicy "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-reinvocationPolicy")reinvocationPolicy indicates whether this webhook should be called multiple times as part of a single admission evaluation. Allowed values are "Never" and "IfNeeded". ##### `name` string [​](#policies-centralAdmission-mutatingWebhooks-webhooks-name "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-name")The name of the admission webhook. Name should be fully qualified, e.g., imagepolicy.kubernetes.io, where "imagepolicy" is the name of the webhook, and kubernetes.io is the name of the organization. ##### `clientConfig` object [​](#policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig")ClientConfig defines how to communicate with the hook. ##### `url` string [​](#policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig-url "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig-url")URL gives the location of the webhook, in standard URL form (`scheme://host:port/path`). Exactly one of `url` or `service` must be specified. ##### `service` object [​](#policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig-service "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig-service")Service is a reference to the service for this webhook. Either `service` or `url` must be specified.If the webhook is running within the cluster, then you should use `service`. ##### `namespace` string [​](#policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-namespace "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-namespace")Namespace is the namespace of the service. ##### `name` string [​](#policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-name "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-name")Name is the name of the service. ##### `path` string [​](#policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-path "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-path")Path is an optional URL path which will be sent in any request to this service. ##### `port` integer [​](#policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-port "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-port")If specified, the port on the service that hosting webhook. Default to 443 for backward compatibility. `port` should be a valid port number (1-65535, inclusive). ##### `caBundle` string [​](#policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig-caBundle "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-clientConfig-caBundle")CABundle is a PEM encoded CA bundle which will be used to validate the webhook's server certificate. If unspecified, system trust roots on the apiserver are used. ##### `rules` object\[] [​](#policies-centralAdmission-mutatingWebhooks-webhooks-rules "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-rules")Rules describes what operations on what resources/subresources the webhook cares about. The webhook cares about an operation if it matches *any* Rule. ##### `failurePolicy` string [​](#policies-centralAdmission-mutatingWebhooks-webhooks-failurePolicy "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-failurePolicy")FailurePolicy defines how unrecognized errors from the admission endpoint are handled - allowed values are Ignore or Fail. Defaults to Fail. ##### `matchPolicy` string [​](#policies-centralAdmission-mutatingWebhooks-webhooks-matchPolicy "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-matchPolicy")matchPolicy defines how the "rules" list is used to match incoming requests. Allowed values are "Exact" or "Equivalent". ##### `namespaceSelector` object [​](#policies-centralAdmission-mutatingWebhooks-webhooks-namespaceSelector "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-namespaceSelector")NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the webhook. ##### `objectSelector` object [​](#policies-centralAdmission-mutatingWebhooks-webhooks-objectSelector "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-objectSelector")ObjectSelector decides whether to run the webhook based on if the object has matching labels. objectSelector is evaluated against both the oldObject and newObject that would be sent to the webhook, and is considered to match if either object matches the selector. ##### `sideEffects` string [​](#policies-centralAdmission-mutatingWebhooks-webhooks-sideEffects "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-sideEffects")SideEffects states whether this webhook has side effects. ##### `timeoutSeconds` integer [​](#policies-centralAdmission-mutatingWebhooks-webhooks-timeoutSeconds "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-timeoutSeconds")TimeoutSeconds specifies the timeout for this webhook. ##### `admissionReviewVersions` string\[] [​](#policies-centralAdmission-mutatingWebhooks-webhooks-admissionReviewVersions "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-admissionReviewVersions")AdmissionReviewVersions is an ordered list of preferred `AdmissionReview` versions the Webhook expects. ##### `matchConditions` object\[] [​](#policies-centralAdmission-mutatingWebhooks-webhooks-matchConditions "Direct link to policies-centralAdmission-mutatingWebhooks-webhooks-matchConditions")MatchConditions is a list of conditions that must be met for a request to be sent to this webhook. Match conditions filter requests that have already been matched by the rules, namespaceSelector, and objectSelector. An empty list of matchConditions matches all requests. There are a maximum of 64 match conditions allowed. --- # Central admission control | | | Enterprise | | | | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Centralized Admission Control](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/admission-control.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. Centralized admission control is an advanced feature for cluster admins that have custom rules that they need to apply to the tenant cluster. Cluster admins can enforce [Kubernetes admission webhooks](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) that reference the control plane cluster or external policy services from within the tenant cluster. Examples of validating and mutating webhook based policy engines include OPA, Kyverno, and jsPolicy. This feature requires the `MutatingAdmissionWebhook` and `ValidatingAdmissionWebhook` admission controllers to be enabled. For more information, see the Kubernetes [admission controllers reference](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#is-there-a-recommended-set-of-admission-controllers-to-use). Configuring this is not the same as enforcing anything This setting wires up the webhook plumbing only. It provides no protection on its own. Protection requires a reachable policy engine on the other end, with active policies whose rules actually cover the resource kinds and operations you care about. Verify the engine has rules loaded, not just that the webhook configuration exists. Tenant-facing admission also never sees translated, synced objects, or workloads a host-side controller generates from a tenant custom resource, such as a KubeVirt `virt-launcher` pod. Those need control plane cluster-side admission instead. See [admission and policy coverage](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#admission-and-policy-coverage). warning * Central admission control is read-only after tenant cluster creation. Admins cannot bypass or alter hooks after vCluster deployment. * If you need to modify or delete webhook configurations, do not use central admission control. Instead, manually map the host policy service into the tenant cluster. Central admission webhooks are different from the other policies: * LimitRange and ResourceQuota resources are created and enforced on the control plane cluster. They do not appear as resources in the tenant cluster. * A user could define LimitRange and ResourceQuota resources inside the tenant cluster, but they have full control over them and can delete them when needed. * Webhooks are different because they need to be configured inside the tenant cluster in order for them to be called by the vCluster's API server. * vCluster rewrites these definitions to point to a proxy for a control plane cluster service that handles webhook requests. The host might also have webhook configurations that use this service. * A user can still install a webhook service or webhook configuration into the tenant cluster outside of this config, but it would run inside the tenant cluster like any other workload. ## Access tenant identifiers in webhooks[​](#access-tenant-identifiers-in-webhooks "Direct link to Access tenant identifiers in webhooks") vCluster rewrites the namespace of a namespace-scoped object to the host namespace where vCluster runs. This is not visible to the user, but the admission controller can use mechanisms like the namespace selector as usual. Some admission controllers such as Gatekeeper rely on the namespace always being present in the underlying cluster and wouldn't work otherwise. To access the tenant namespace in a webhook, read the `request.options.vCluster.namespace.metadata.name` field in the [admission request object](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#request), which holds the full tenant namespace object. In some cases a webhook running inside the tenant cluster needs the translated name of a pod. This is the name used when the pod is created on the control plane cluster. vCluster puts the translated pod name under the `request.options.vCluster.name` field in the [admission request object](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#request). The final name is only available during the validation phase, so this only works with `validatingWebhooks` that have rules configured for pods. ## Certificate configuration[​](#certificate-configuration "Direct link to Certificate configuration") For webhooks using self-signed or private certificates, add the `caBundle` field: ``` # Get base64-encoded CA certificate from secret kubectl get secret webhook-tls-secret -n webhook-namespace -o jsonpath='{.data.ca\.crt}' ``` warning Missing `caBundle` with `failurePolicy: Fail` can make vCluster unrecoverable. Test with `failurePolicy: Ignore` first. ## Kyverno example[​](#kyverno-example "Direct link to Kyverno example") This example ensures that all new ConfigMap resources are sent to a mutating webhook on the control plane cluster. Apply Enforce-mode policies after the tenant is ready A Kyverno `ClusterPolicy` running on the control plane cluster in `Enforce` mode validates the resources that vCluster syncs from the tenant. This includes vCluster's own system pods, which don't carry an `app.kubernetes.io/name` label. A label-requiring policy applied before the tenant is provisioned rejects those pods, and the VirtualClusterInstance ends in `Failed`. To avoid this, do one of the following: * Apply Enforce-mode policies only after the tenant cluster reaches `Ready`. * Start the policy in `Audit` mode. * Scope the policy to exclude vCluster system namespaces and service accounts. ``` centralAdmission: mutatingWebhooks: - apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: name: kyverno-webhook webhooks: - admissionReviewVersions: - v1 clientConfig: # Add caBundle for self-signed certificates caBundle: LS0tLS1CRUdJTi... # Base64-encoded PEM certificate service: name: kyverno-svc namespace: kyverno path: /mutate/fail port: 443 failurePolicy: Fail matchPolicy: Equivalent name: mutate.kyverno.svc-fail objectSelector: {} reinvocationPolicy: IfNeeded rules: - apiGroups: - "" apiVersions: - v1 operations: - CREATE resources: - configmaps scope: '*' sideEffects: NoneOnDryRun timeoutSeconds: 10 ``` Any time a ConfigMap is created, the vCluster API server forwards the admission request to a proxy, which in turn calls the actual admission hook running in the control plane cluster in the `kyverno` namespace. The service providing the webhooks should not rely on the real names of the objects if they are synced on the control plane cluster (for example, pods). The requests that reach the host admission service hold the objects from the tenant cluster, so your policies need to be able to handle those objects (except for the namespace which is rewritten as outlined above). Policies that key off name or namespace The admission webhook fires against the translated name (`-x--x-`) and the host namespace, not the tenant's view. Policies that key off `metadata.name` or the namespace produce wrong results: * A uniqueness or collision check silently passes resources that actually collide. * A naming or length check rejects tenant resources that are valid in the tenant's own view. * Namespace-scoped match and exclude rules apply to the wrong set of objects. To read the original tenant identifiers, see [Access tenant identifiers in webhooks](#access-tenant-identifiers-in-webhooks). The hook services do not require authentication as the proxy does not have access to the `AdmissionConfiguration` object or the files it references. Most policy engines (such as jsPolicy, Kyverno, or Gatekeeper) do not set authentication by default, so it should work with most installations. If a user inside the tenant cluster (admin or not) tries to delete one of your admission hooks, that user is prompted with the following error message: ``` kubectl delete kyverno-webhook error: the resource kyverno-webhook is protected ``` ## Gatekeeper (OPA) example[​](#gatekeeper-opa-example "Direct link to Gatekeeper (OPA) example") Ensure you [install Gatekeeper](https://open-policy-agent.github.io/gatekeeper/website/docs/install/) to your Kubernetes cluster. Create the following constraint template that denies pods without specified labels: ``` apiVersion: config.gatekeeper.sh/v1alpha1 kind: Config metadata: name: config namespace: "gatekeeper-system" spec: match: - excludedNamespaces: ["kube-*"] processes: ["*"] sync: syncOnly: - group: "" version: "v1" kind: "Namespace" - group: "" version: "v1" kind: "Pod" --- apiVersion: templates.gatekeeper.sh/v1 kind: ConstraintTemplate metadata: name: k8srequiredlabels spec: crd: spec: names: kind: K8sRequiredLabels validation: # Schema for the `parameters` field openAPIV3Schema: type: object properties: labels: type: array items: type: string targets: - target: admission.k8s.gatekeeper.sh rego: | package k8srequiredlabels violation[{"msg": msg, "details": {"missing_labels": missing}}] { provided := {label | input.review.object.metadata.labels[label]} required := {label | label := input.parameters.labels[_]} missing := required - provided count(missing) > 0 msg := sprintf("you must provide labels: %v", [missing]) } ``` Create the constraint: ``` apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sRequiredLabels metadata: name: pod-required-labels spec: match: kinds: - apiGroups: [""] kinds: ["Pod"] labelSelector: matchLabels: test: gatekeeper parameters: labels: ["gatekeeper"] ``` Then start the vCluster with the following configuration: ``` policies: centralAdmission: validatingWebhooks: - apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: name: gatekeeper-webhook webhooks: - admissionReviewVersions: - v1 clientConfig: # Add caBundle for self-signed certificates caBundle: LS0tLS1CRUdJTi... # Base64-encoded PEM certificate service: name: gatekeeper-webhook-service namespace: gatekeeper-system path: /v1/admit port: 443 failurePolicy: Fail matchPolicy: Exact name: validation.gatekeeper.sh objectSelector: {} rules: - apiGroups: - '*' apiVersions: - '*' operations: - CREATE - UPDATE resources: - pods scope: Namespaced sideEffects: None timeoutSeconds: 10 ``` Within the vCluster try to create the following pod which should get denied: ``` $ echo "apiVersion: v1 kind: Pod metadata: name: nginx labels: test: gatekeeper spec: containers: - image: nginx name: nginx" | kubectl create -f - admission webhook "validation.gatekeeper.sh" denied the request: you must provide labels: [gatekeeper] ``` ## Network policies[​](#network-policies "Direct link to Network policies") If you are using [network policies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md), admission webhooks traffic from the vCluster control plane must be allowed. vcluster.yaml ``` policies: networkPolicy: controlPlane: egress: - to: # Allow vcluster control plane traffic to the gatekeeper webhook. - namespaceSelector: matchLabels: kubernetes.io/metadata.name: gatekeeper-system podSelector: matchLabels: gatekeeper.sh/operation: webhook ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `centralAdmission` object [​](#centralAdmission "Direct link to centralAdmission")CentralAdmission defines what validating or mutating webhooks should be enforced within the virtual cluster. ### `validatingWebhooks` object\[] [​](#centralAdmission-validatingWebhooks "Direct link to centralAdmission-validatingWebhooks")ValidatingWebhooks are validating webhooks that should be enforced in the virtual cluster #### `kind` string [​](#centralAdmission-validatingWebhooks-kind "Direct link to centralAdmission-validatingWebhooks-kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. #### `apiVersion` string [​](#centralAdmission-validatingWebhooks-apiVersion "Direct link to centralAdmission-validatingWebhooks-apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. #### `metadata` object [​](#centralAdmission-validatingWebhooks-metadata "Direct link to centralAdmission-validatingWebhooks-metadata")Standard object metadata; More info: . ##### `name` string [​](#centralAdmission-validatingWebhooks-metadata-name "Direct link to centralAdmission-validatingWebhooks-metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. ##### `labels` object [​](#centralAdmission-validatingWebhooks-metadata-labels "Direct link to centralAdmission-validatingWebhooks-metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. ##### `annotations` object [​](#centralAdmission-validatingWebhooks-metadata-annotations "Direct link to centralAdmission-validatingWebhooks-metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. #### `webhooks` object\[] [​](#centralAdmission-validatingWebhooks-webhooks "Direct link to centralAdmission-validatingWebhooks-webhooks")Webhooks is a list of webhooks and the affected resources and operations. ##### `name` string [​](#centralAdmission-validatingWebhooks-webhooks-name "Direct link to centralAdmission-validatingWebhooks-webhooks-name")The name of the admission webhook. Name should be fully qualified, e.g., imagepolicy.kubernetes.io, where "imagepolicy" is the name of the webhook, and kubernetes.io is the name of the organization. ##### `clientConfig` object [​](#centralAdmission-validatingWebhooks-webhooks-clientConfig "Direct link to centralAdmission-validatingWebhooks-webhooks-clientConfig")ClientConfig defines how to communicate with the hook. ##### `url` string [​](#centralAdmission-validatingWebhooks-webhooks-clientConfig-url "Direct link to centralAdmission-validatingWebhooks-webhooks-clientConfig-url")URL gives the location of the webhook, in standard URL form (`scheme://host:port/path`). Exactly one of `url` or `service` must be specified. ##### `service` object [​](#centralAdmission-validatingWebhooks-webhooks-clientConfig-service "Direct link to centralAdmission-validatingWebhooks-webhooks-clientConfig-service")Service is a reference to the service for this webhook. Either `service` or `url` must be specified.If the webhook is running within the cluster, then you should use `service`. ##### `namespace` string [​](#centralAdmission-validatingWebhooks-webhooks-clientConfig-service-namespace "Direct link to centralAdmission-validatingWebhooks-webhooks-clientConfig-service-namespace")Namespace is the namespace of the service. ##### `name` string [​](#centralAdmission-validatingWebhooks-webhooks-clientConfig-service-name "Direct link to centralAdmission-validatingWebhooks-webhooks-clientConfig-service-name")Name is the name of the service. ##### `path` string [​](#centralAdmission-validatingWebhooks-webhooks-clientConfig-service-path "Direct link to centralAdmission-validatingWebhooks-webhooks-clientConfig-service-path")Path is an optional URL path which will be sent in any request to this service. ##### `port` integer [​](#centralAdmission-validatingWebhooks-webhooks-clientConfig-service-port "Direct link to centralAdmission-validatingWebhooks-webhooks-clientConfig-service-port")If specified, the port on the service that hosting webhook. Default to 443 for backward compatibility. `port` should be a valid port number (1-65535, inclusive). ##### `caBundle` string [​](#centralAdmission-validatingWebhooks-webhooks-clientConfig-caBundle "Direct link to centralAdmission-validatingWebhooks-webhooks-clientConfig-caBundle")CABundle is a PEM encoded CA bundle which will be used to validate the webhook's server certificate. If unspecified, system trust roots on the apiserver are used. ##### `rules` object\[] [​](#centralAdmission-validatingWebhooks-webhooks-rules "Direct link to centralAdmission-validatingWebhooks-webhooks-rules")Rules describes what operations on what resources/subresources the webhook cares about. The webhook cares about an operation if it matches *any* Rule. ##### `failurePolicy` string [​](#centralAdmission-validatingWebhooks-webhooks-failurePolicy "Direct link to centralAdmission-validatingWebhooks-webhooks-failurePolicy")FailurePolicy defines how unrecognized errors from the admission endpoint are handled - allowed values are Ignore or Fail. Defaults to Fail. ##### `matchPolicy` string [​](#centralAdmission-validatingWebhooks-webhooks-matchPolicy "Direct link to centralAdmission-validatingWebhooks-webhooks-matchPolicy")matchPolicy defines how the "rules" list is used to match incoming requests. Allowed values are "Exact" or "Equivalent". ##### `namespaceSelector` object [​](#centralAdmission-validatingWebhooks-webhooks-namespaceSelector "Direct link to centralAdmission-validatingWebhooks-webhooks-namespaceSelector")NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the webhook. ##### `objectSelector` object [​](#centralAdmission-validatingWebhooks-webhooks-objectSelector "Direct link to centralAdmission-validatingWebhooks-webhooks-objectSelector")ObjectSelector decides whether to run the webhook based on if the object has matching labels. objectSelector is evaluated against both the oldObject and newObject that would be sent to the webhook, and is considered to match if either object matches the selector. ##### `sideEffects` string [​](#centralAdmission-validatingWebhooks-webhooks-sideEffects "Direct link to centralAdmission-validatingWebhooks-webhooks-sideEffects")SideEffects states whether this webhook has side effects. ##### `timeoutSeconds` integer [​](#centralAdmission-validatingWebhooks-webhooks-timeoutSeconds "Direct link to centralAdmission-validatingWebhooks-webhooks-timeoutSeconds")TimeoutSeconds specifies the timeout for this webhook. ##### `admissionReviewVersions` string\[] [​](#centralAdmission-validatingWebhooks-webhooks-admissionReviewVersions "Direct link to centralAdmission-validatingWebhooks-webhooks-admissionReviewVersions")AdmissionReviewVersions is an ordered list of preferred `AdmissionReview` versions the Webhook expects. ##### `matchConditions` object\[] [​](#centralAdmission-validatingWebhooks-webhooks-matchConditions "Direct link to centralAdmission-validatingWebhooks-webhooks-matchConditions")MatchConditions is a list of conditions that must be met for a request to be sent to this webhook. Match conditions filter requests that have already been matched by the rules, namespaceSelector, and objectSelector. An empty list of matchConditions matches all requests. There are a maximum of 64 match conditions allowed. ### `mutatingWebhooks` object\[] [​](#centralAdmission-mutatingWebhooks "Direct link to centralAdmission-mutatingWebhooks")MutatingWebhooks are mutating webhooks that should be enforced in the virtual cluster #### `kind` string [​](#centralAdmission-mutatingWebhooks-kind "Direct link to centralAdmission-mutatingWebhooks-kind")Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. #### `apiVersion` string [​](#centralAdmission-mutatingWebhooks-apiVersion "Direct link to centralAdmission-mutatingWebhooks-apiVersion")APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. #### `metadata` object [​](#centralAdmission-mutatingWebhooks-metadata "Direct link to centralAdmission-mutatingWebhooks-metadata")Standard object metadata; More info: . ##### `name` string [​](#centralAdmission-mutatingWebhooks-metadata-name "Direct link to centralAdmission-mutatingWebhooks-metadata-name")Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. ##### `labels` object [​](#centralAdmission-mutatingWebhooks-metadata-labels "Direct link to centralAdmission-mutatingWebhooks-metadata-labels")Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. ##### `annotations` object [​](#centralAdmission-mutatingWebhooks-metadata-annotations "Direct link to centralAdmission-mutatingWebhooks-metadata-annotations")Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. #### `webhooks` object\[] [​](#centralAdmission-mutatingWebhooks-webhooks "Direct link to centralAdmission-mutatingWebhooks-webhooks")Webhooks is a list of webhooks and the affected resources and operations. ##### `reinvocationPolicy` string [​](#centralAdmission-mutatingWebhooks-webhooks-reinvocationPolicy "Direct link to centralAdmission-mutatingWebhooks-webhooks-reinvocationPolicy")reinvocationPolicy indicates whether this webhook should be called multiple times as part of a single admission evaluation. Allowed values are "Never" and "IfNeeded". ##### `name` string [​](#centralAdmission-mutatingWebhooks-webhooks-name "Direct link to centralAdmission-mutatingWebhooks-webhooks-name")The name of the admission webhook. Name should be fully qualified, e.g., imagepolicy.kubernetes.io, where "imagepolicy" is the name of the webhook, and kubernetes.io is the name of the organization. ##### `clientConfig` object [​](#centralAdmission-mutatingWebhooks-webhooks-clientConfig "Direct link to centralAdmission-mutatingWebhooks-webhooks-clientConfig")ClientConfig defines how to communicate with the hook. ##### `url` string [​](#centralAdmission-mutatingWebhooks-webhooks-clientConfig-url "Direct link to centralAdmission-mutatingWebhooks-webhooks-clientConfig-url")URL gives the location of the webhook, in standard URL form (`scheme://host:port/path`). Exactly one of `url` or `service` must be specified. ##### `service` object [​](#centralAdmission-mutatingWebhooks-webhooks-clientConfig-service "Direct link to centralAdmission-mutatingWebhooks-webhooks-clientConfig-service")Service is a reference to the service for this webhook. Either `service` or `url` must be specified.If the webhook is running within the cluster, then you should use `service`. ##### `namespace` string [​](#centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-namespace "Direct link to centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-namespace")Namespace is the namespace of the service. ##### `name` string [​](#centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-name "Direct link to centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-name")Name is the name of the service. ##### `path` string [​](#centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-path "Direct link to centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-path")Path is an optional URL path which will be sent in any request to this service. ##### `port` integer [​](#centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-port "Direct link to centralAdmission-mutatingWebhooks-webhooks-clientConfig-service-port")If specified, the port on the service that hosting webhook. Default to 443 for backward compatibility. `port` should be a valid port number (1-65535, inclusive). ##### `caBundle` string [​](#centralAdmission-mutatingWebhooks-webhooks-clientConfig-caBundle "Direct link to centralAdmission-mutatingWebhooks-webhooks-clientConfig-caBundle")CABundle is a PEM encoded CA bundle which will be used to validate the webhook's server certificate. If unspecified, system trust roots on the apiserver are used. ##### `rules` object\[] [​](#centralAdmission-mutatingWebhooks-webhooks-rules "Direct link to centralAdmission-mutatingWebhooks-webhooks-rules")Rules describes what operations on what resources/subresources the webhook cares about. The webhook cares about an operation if it matches *any* Rule. ##### `failurePolicy` string [​](#centralAdmission-mutatingWebhooks-webhooks-failurePolicy "Direct link to centralAdmission-mutatingWebhooks-webhooks-failurePolicy")FailurePolicy defines how unrecognized errors from the admission endpoint are handled - allowed values are Ignore or Fail. Defaults to Fail. ##### `matchPolicy` string [​](#centralAdmission-mutatingWebhooks-webhooks-matchPolicy "Direct link to centralAdmission-mutatingWebhooks-webhooks-matchPolicy")matchPolicy defines how the "rules" list is used to match incoming requests. Allowed values are "Exact" or "Equivalent". ##### `namespaceSelector` object [​](#centralAdmission-mutatingWebhooks-webhooks-namespaceSelector "Direct link to centralAdmission-mutatingWebhooks-webhooks-namespaceSelector")NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the webhook. ##### `objectSelector` object [​](#centralAdmission-mutatingWebhooks-webhooks-objectSelector "Direct link to centralAdmission-mutatingWebhooks-webhooks-objectSelector")ObjectSelector decides whether to run the webhook based on if the object has matching labels. objectSelector is evaluated against both the oldObject and newObject that would be sent to the webhook, and is considered to match if either object matches the selector. ##### `sideEffects` string [​](#centralAdmission-mutatingWebhooks-webhooks-sideEffects "Direct link to centralAdmission-mutatingWebhooks-webhooks-sideEffects")SideEffects states whether this webhook has side effects. ##### `timeoutSeconds` integer [​](#centralAdmission-mutatingWebhooks-webhooks-timeoutSeconds "Direct link to centralAdmission-mutatingWebhooks-webhooks-timeoutSeconds")TimeoutSeconds specifies the timeout for this webhook. ##### `admissionReviewVersions` string\[] [​](#centralAdmission-mutatingWebhooks-webhooks-admissionReviewVersions "Direct link to centralAdmission-mutatingWebhooks-webhooks-admissionReviewVersions")AdmissionReviewVersions is an ordered list of preferred `AdmissionReview` versions the Webhook expects. ##### `matchConditions` object\[] [​](#centralAdmission-mutatingWebhooks-webhooks-matchConditions "Direct link to centralAdmission-mutatingWebhooks-webhooks-matchConditions")MatchConditions is a list of conditions that must be met for a request to be sent to this webhook. Match conditions filter requests that have already been matched by the rules, namespaceSelector, and objectSelector. An empty list of matchConditions matches all requests. There are a maximum of 64 match conditions allowed. --- # Limit range Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** You can create a [LimitRange](https://kubernetes.io/docs/concepts/policy/limit-range/) policy to constrain tenant cluster resource allocation. vCluster creates the LimitRange on the host in the same namespace as vCluster itself. LimitRange only applies to synced resources, such as pods. ## Example[​](#example "Direct link to Example") If you would normally deploy the following LimitRange: ``` apiVersion: v1 kind: LimitRange metadata: name: vcluster-limit-range spec: limits: - default: memory: 512Mi cpu: "1" defaultRequest: memory: 128Mi cpu: 100m type: Container ``` Instead configure the LimitRange in your `vcluster.yaml` config file like this: ``` policies: limitRange: enabled: true default: memory: 512Mi cpu: "1" defaultRequest: memory: 128Mi cpu: 100m ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `limitRange` object [​](#limitRange "Direct link to limitRange")LimitRange specifies limit range options. ### `enabled` string|boolean auto [​](#limitRange-enabled "Direct link to limitRange-enabled")Enabled defines if the limit range should be deployed by vCluster. "auto" means that if resourceQuota is enabled, the limitRange will be enabled as well. ### `default` object map\[cpu:1 ephemeral-storage:8Gi memory:512Mi] [​](#limitRange-default "Direct link to limitRange-default")Default are the default limits for the limit range ### `defaultRequest` object map\[cpu:100m ephemeral-storage:3Gi memory:128Mi] [​](#limitRange-defaultRequest "Direct link to limitRange-defaultRequest")DefaultRequest are the default request options for the limit range ### `max` object {} [​](#limitRange-max "Direct link to limitRange-max")Max are the max limits for the limit range ### `min` object {} [​](#limitRange-min "Direct link to limitRange-min")Min are the min limits for the limit range ### `annotations` object {} [​](#limitRange-annotations "Direct link to limitRange-annotations")Annotations are extra annotations for this resource. ### `labels` object {} [​](#limitRange-labels "Direct link to limitRange-labels")Labels are extra labels for this resource. --- # Network policy Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** note This feature is disabled by default. By default, workloads created by vCluster are able to communicate with other workloads in the control plane cluster through their cluster IPs. Configure network policies when you want to isolate namespaces and do not want the pods running inside the tenant cluster to have access to other workloads in the control plane cluster. Enabling this creates Kubernetes [NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) resources in the host namespace that control how vCluster pods (both control plane and workloads) communicate with each other and with other pods on the control plane cluster. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") [Network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) are implemented by the [network plugin (CNI)](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/). To use network policies, you must be using a networking solution which supports NetworkPolicy. Creating a NetworkPolicy resource without a controller that implements it will have no effect. Some CNIs accept NetworkPolicy resources without enforcing them, so confirm enforcement directly rather than assuming it from CNI support alone. Coverage is limited to the primary pod interface NetworkPolicy governs the primary pod network interface, including a VM's primary interface through its `virt-launcher` Pod. It doesn't cover Multus secondary interfaces or direct underlay network attachments. Host-network Pod traffic is undefined in the Kubernetes NetworkPolicy spec and CNI-dependent. Most CNIs ignore it, but confirm your CNI's actual behavior rather than assuming either way. See [network and endpoint isolation](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#network-and-endpoint-isolation) for the controls those paths need. ## Enable network isolation[​](#enable-isolation "Direct link to Enable network isolation") Set `policies.networkPolicy.enabled` to create NetworkPolicies that isolate the tenant cluster: vcluster.yaml ``` policies: networkPolicy: enabled: true ``` This creates Kubernetes NetworkPolicies resources in the host namespace that: * Allow traffic between pods within the tenant cluster * Block traffic from other namespaces * Permit DNS (port 1053) and API server communication note The Kubernetes NetworkPolicies resources are managed by vCluster. Manual changes to these resources will be overwritten. Example of NetworkPolicies resources created in the host namespace ``` apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: vc-work-{name} namespace: vcluster-{name} labels: app: vcluster chart: vcluster-0.31.0 heritage: Helm release: {name} spec: # Pod selector matching virtual cluster workloads pods. podSelector: matchLabels: vcluster.loft.sh/managed-by: {name} policyTypes: - Egress - Ingress egress: # Allow egress to vcluster DNS and control plane. - ports: - port: 1053 protocol: UDP - port: 1053 protocol: TCP - port: 8443 protocol: TCP to: - podSelector: matchLabels: release: {name} # Allow egress to other vcluster workloads, including coredns when not embedded. - to: - podSelector: matchLabels: vcluster.loft.sh/managed-by: {name} # Allow public egress. - to: - ipBlock: cidr: 0.0.0.0/0 except: - 100.64.0.0/10 - 127.0.0.0/8 - 10.0.0.0/8 - 172.16.0.0/12 - 192.168.0.0/16 ingress: # Allow ingress from vcluster control plane. - from: - podSelector: matchLabels: release: {name} # Allow ingress from other vcluster workloads. - from: - podSelector: matchLabels: vcluster.loft.sh/managed-by: {name} --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: vc-cp-{name} namespace: vcluster-{name} labels: app: vcluster chart: vcluster-0.31.0 heritage: Helm release: {name} spec: # Pod selector matching virtual cluster control plane pods. podSelector: matchLabels: release: {name} policyTypes: - Egress - Ingress egress: # Allow egress to host kube-dns. - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: 'kube-system' podSelector: matchLabels: k8s-app: kube-dns # Allow egress to host control plane. - ports: - port: 443 protocol: TCP - port: 8443 protocol: TCP - port: 6443 protocol: TCP # Allow egress to vcluster control plane peers, including etcd peers, when using etcd as the backend in HA mode. - to: - podSelector: matchLabels: release: {name} # Allow egress connections to vcluster workloads. - to: - podSelector: matchLabels: vcluster.loft.sh/managed-by: {name} # Allow egress to vcluster platform. - to: - podSelector: matchLabels: app: loft namespaceSelector: {} ingress: # Allow ingress from vcluster control plane peers, including etcd peers, when using etcd as the backend in HA mode. - from: - podSelector: matchLabels: release: {name} # Allow ingress for vcluster workloads. - ports: - port: 1053 protocol: UDP - port: 1053 protocol: TCP - port: 8443 protocol: TCP from: - podSelector: matchLabels: vcluster.loft.sh/managed-by: {name} # Allow ingress from vcluster snapshot. - from: - podSelector: matchLabels: app: vcluster-snapshot # Allow ingress from vcluster platform. - from: - podSelector: matchLabels: app: loft namespaceSelector: {} ``` ### Example configurations[​](#examples "Direct link to Example configurations") #### Custom ingress and egress rules[​](#custom-rules "Direct link to Custom ingress and egress rules") DNS port in tenant clusters Tenant cluster DNS runs on port 1053, not the standard port 53. Custom egress rules that allow DNS traffic must target port 1053 or DNS resolution silently fails. UDP packets are dropped with no error in pod logs. ``` ports: - port: 1053 protocol: UDP - port: 1053 protocol: TCP ``` Control inbound and outbound traffic with specific ports and IP addresses for vCluster control plane and workloads: vcluster.yaml ``` policies: networkPolicy: enabled: true workload: ingress: # Allow ingress from anywhere to specific ports - ports: - port: 6060 - port: 444 egress: # Allow egress to a specific address and port - to: - ipBlock: cidr: 172.19.10.23/32 ports: - port: 7777 protocol: TCP publicEgress: # Disable convenience common public egress rule. enabled: false controlPlane: ingress: # Allow ingress traffic from anywhere to the virtual cluster control plane api - ports: - port: 8443 egress: # Allow egress traffic to a specific address - to: - ipBlock: cidr: 172.19.10.23/32 ``` note `ingress` and `egress` config sections follow the same schema as [PodNetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/#podnetworkpolicy-resource) #### Add custom labels[​](#custom-labels "Direct link to Add custom labels") Apply labels to generated NetworkPolicies for easier management: vcluster.yaml ``` policies: networkPolicy: enabled: true labels: environment: production team: platform annotations: description: "Network isolation for production vCluster" ``` ## Project-scoped isolation with Platform[​](#project-scoped-isolation "Direct link to Project-scoped isolation with Platform") For Platform users needing project-level network boundaries, combine `policies.networkPolicy` with [VirtualClusterTemplates](https://vcluster.com/docs/platform/administer/templates/create-templates.md): project-isolated-template.yaml ``` apiVersion: management.loft.sh/v1 kind: VirtualClusterTemplate metadata: name: project-isolated spec: template: helmRelease: chart: version: 0.26.0 values: | policies: networkPolicy: enabled: true labels: vcluster.io/project: "{{ .Values.loft.project }}" ``` This automatically: * Isolates tenant clusters by project * Allows communication within the same project * Enforces network boundaries for CI/CD pipelines ## Migration from v0.30 config[​](#migration "Direct link to Migration from v0.30 config") `workload` and `controlPlane` configuration sections are introduced to allow defining additional ingress/egress rules for the specific components. vcluster.yaml (v0.30 and earlier) ``` policies: networkPolicy: enabled: true extraControlPlaneRules: - ports: - port: 8443 extraWorkloadRules: - ports: - port: 6060 outgoingConnections: ipBlock: cidr: 172.19.10.23/32 ``` vcluster.yaml (v0.31) ``` policies: networkPolicy: enabled: true controlPlane: egress: - ports: - port: 8443 workload: egress: - ports: - port: 6060 publicEgress: cidr: 172.19.10.23/32 ``` ## Config reference[​](#config-reference "Direct link to Config reference") | Deprecated Field | New Field | | ----------------------------- | ----------------------- | | `extraControlPlaneRules` | `controlPlane.egress` | | `extraWorkloadRules` | `workload.egress` | | `outgoingConnections.ipBlock` | `workload.publicEgress` | ## `networkPolicy` object [​](#networkPolicy "Direct link to networkPolicy")NetworkPolicy specifies network policy options. ### `enabled` boolean false [​](#networkPolicy-enabled "Direct link to networkPolicy-enabled")Enabled defines if the network policy should be deployed by vCluster. ### `annotations` object {} [​](#networkPolicy-annotations "Direct link to networkPolicy-annotations")Annotations are extra annotations for this resource. ### `labels` object {} [​](#networkPolicy-labels "Direct link to networkPolicy-labels")Labels are extra labels for this resource. ### `fallbackDns` string 8.8.8.8 [​](#networkPolicy-fallbackDns "Direct link to networkPolicy-fallbackDns")FallbackDNS is the fallback DNS server to use if the virtual cluster does not have a DNS server. ### `controlPlane` object [​](#networkPolicy-controlPlane "Direct link to networkPolicy-controlPlane")ControlPlane network policy rules #### `ingress` object\[] [​](#networkPolicy-controlPlane-ingress "Direct link to networkPolicy-controlPlane-ingress")Ingress rules for the vCluster control plane. ##### `ports` object\[] [​](#networkPolicy-controlPlane-ingress-ports "Direct link to networkPolicy-controlPlane-ingress-ports")ports is a list of ports which should be made accessible on the pods selected for this rule. Each item in this list is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list. ##### `protocol` string [​](#networkPolicy-controlPlane-ingress-ports-protocol "Direct link to networkPolicy-controlPlane-ingress-ports-protocol")protocol represents the protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this field defaults to TCP. ##### `port` object [​](#networkPolicy-controlPlane-ingress-ports-port "Direct link to networkPolicy-controlPlane-ingress-ports-port")port represents the port on the given protocol. This can either be a numerical or named port on a pod. If this field is not provided, this matches all port names and numbers. If present, only traffic on the specified protocol AND port will be matched. ##### `endPort` integer [​](#networkPolicy-controlPlane-ingress-ports-endPort "Direct link to networkPolicy-controlPlane-ingress-ports-endPort")endPort indicates that the range of ports from port to endPort if set, inclusive, should be allowed by the policy. This field cannot be defined if the port field is not defined or if the port field is defined as a named (string) port. The endPort must be equal or greater than port. ##### `from` object\[] [​](#networkPolicy-controlPlane-ingress-from "Direct link to networkPolicy-controlPlane-ingress-from")from is a list of sources which should be able to access the pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all sources (traffic not restricted by source). If this field is present and contains at least one item, this rule allows traffic only if the traffic matches at least one item in the from list. ##### `podSelector` object [​](#networkPolicy-controlPlane-ingress-from-podSelector "Direct link to networkPolicy-controlPlane-ingress-from-podSelector")podSelector is a label selector which selects pods. This field follows standard label selector semantics; if present but empty, it selects all pods.If namespaceSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the Namespaces selected by NamespaceSelector. Otherwise it selects the pods matching podSelector in the policy's own namespace. ##### `matchLabels` object [​](#networkPolicy-controlPlane-ingress-from-podSelector-matchLabels "Direct link to networkPolicy-controlPlane-ingress-from-podSelector-matchLabels") ##### `matchExpressions` object\[] [​](#networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions "Direct link to networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions") ##### `key` string [​](#networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions-key "Direct link to networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions-key") ##### `operator` string [​](#networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions-operator "Direct link to networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions-operator") ##### `values` string\[] [​](#networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions-values "Direct link to networkPolicy-controlPlane-ingress-from-podSelector-matchExpressions-values") ##### `namespaceSelector` object [​](#networkPolicy-controlPlane-ingress-from-namespaceSelector "Direct link to networkPolicy-controlPlane-ingress-from-namespaceSelector")namespaceSelector selects namespaces using cluster-scoped labels. This field follows standard label selector semantics; if present but empty, it selects all namespaces.If podSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the namespaces selected by namespaceSelector. Otherwise it selects all pods in the namespaces selected by namespaceSelector. ##### `matchLabels` object [​](#networkPolicy-controlPlane-ingress-from-namespaceSelector-matchLabels "Direct link to networkPolicy-controlPlane-ingress-from-namespaceSelector-matchLabels") ##### `matchExpressions` object\[] [​](#networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions "Direct link to networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions") ##### `key` string [​](#networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions-key "Direct link to networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions-key") ##### `operator` string [​](#networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions-operator "Direct link to networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions-operator") ##### `values` string\[] [​](#networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions-values "Direct link to networkPolicy-controlPlane-ingress-from-namespaceSelector-matchExpressions-values") ##### `ipBlock` object [​](#networkPolicy-controlPlane-ingress-from-ipBlock "Direct link to networkPolicy-controlPlane-ingress-from-ipBlock")ipBlock defines policy on a particular IPBlock. If this field is set then neither of the other fields can be. ##### `cidr` string [​](#networkPolicy-controlPlane-ingress-from-ipBlock-cidr "Direct link to networkPolicy-controlPlane-ingress-from-ipBlock-cidr")CIDR defines the allowed workload public egress destination. Valid examples are "0.0.0.0/0", "192.168.1.0/24" or "2001:db8::/64" ##### `except` string\[] [​](#networkPolicy-controlPlane-ingress-from-ipBlock-except "Direct link to networkPolicy-controlPlane-ingress-from-ipBlock-except")Except is a slice of CIDRs that should not be included. Items outside the cidr range will be rejected. Valid examples are "192.168.1.0/24" or "2001:db8::/64". #### `egress` object\[] [​](#networkPolicy-controlPlane-egress "Direct link to networkPolicy-controlPlane-egress")Egress rules for the vCluster control plane. ##### `ports` object\[] [​](#networkPolicy-controlPlane-egress-ports "Direct link to networkPolicy-controlPlane-egress-ports")ports is a list of destination ports for outgoing traffic. Each item in this list is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list. ##### `protocol` string [​](#networkPolicy-controlPlane-egress-ports-protocol "Direct link to networkPolicy-controlPlane-egress-ports-protocol")protocol represents the protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this field defaults to TCP. ##### `port` object [​](#networkPolicy-controlPlane-egress-ports-port "Direct link to networkPolicy-controlPlane-egress-ports-port")port represents the port on the given protocol. This can either be a numerical or named port on a pod. If this field is not provided, this matches all port names and numbers. If present, only traffic on the specified protocol AND port will be matched. ##### `endPort` integer [​](#networkPolicy-controlPlane-egress-ports-endPort "Direct link to networkPolicy-controlPlane-egress-ports-endPort")endPort indicates that the range of ports from port to endPort if set, inclusive, should be allowed by the policy. This field cannot be defined if the port field is not defined or if the port field is defined as a named (string) port. The endPort must be equal or greater than port. ##### `to` object\[] [​](#networkPolicy-controlPlane-egress-to "Direct link to networkPolicy-controlPlane-egress-to")to is a list of destinations for outgoing traffic of pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all destinations (traffic not restricted by destination). If this field is present and contains at least one item, this rule allows traffic only if the traffic matches at least one item in the to list. ##### `podSelector` object [​](#networkPolicy-controlPlane-egress-to-podSelector "Direct link to networkPolicy-controlPlane-egress-to-podSelector")podSelector is a label selector which selects pods. This field follows standard label selector semantics; if present but empty, it selects all pods.If namespaceSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the Namespaces selected by NamespaceSelector. Otherwise it selects the pods matching podSelector in the policy's own namespace. ##### `matchLabels` object [​](#networkPolicy-controlPlane-egress-to-podSelector-matchLabels "Direct link to networkPolicy-controlPlane-egress-to-podSelector-matchLabels") ##### `matchExpressions` object\[] [​](#networkPolicy-controlPlane-egress-to-podSelector-matchExpressions "Direct link to networkPolicy-controlPlane-egress-to-podSelector-matchExpressions") ##### `key` string [​](#networkPolicy-controlPlane-egress-to-podSelector-matchExpressions-key "Direct link to networkPolicy-controlPlane-egress-to-podSelector-matchExpressions-key") ##### `operator` string [​](#networkPolicy-controlPlane-egress-to-podSelector-matchExpressions-operator "Direct link to networkPolicy-controlPlane-egress-to-podSelector-matchExpressions-operator") ##### `values` string\[] [​](#networkPolicy-controlPlane-egress-to-podSelector-matchExpressions-values "Direct link to networkPolicy-controlPlane-egress-to-podSelector-matchExpressions-values") ##### `namespaceSelector` object [​](#networkPolicy-controlPlane-egress-to-namespaceSelector "Direct link to networkPolicy-controlPlane-egress-to-namespaceSelector")namespaceSelector selects namespaces using cluster-scoped labels. This field follows standard label selector semantics; if present but empty, it selects all namespaces.If podSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the namespaces selected by namespaceSelector. Otherwise it selects all pods in the namespaces selected by namespaceSelector. ##### `matchLabels` object [​](#networkPolicy-controlPlane-egress-to-namespaceSelector-matchLabels "Direct link to networkPolicy-controlPlane-egress-to-namespaceSelector-matchLabels") ##### `matchExpressions` object\[] [​](#networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions "Direct link to networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions") ##### `key` string [​](#networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions-key "Direct link to networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions-key") ##### `operator` string [​](#networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions-operator "Direct link to networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions-operator") ##### `values` string\[] [​](#networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions-values "Direct link to networkPolicy-controlPlane-egress-to-namespaceSelector-matchExpressions-values") ##### `ipBlock` object [​](#networkPolicy-controlPlane-egress-to-ipBlock "Direct link to networkPolicy-controlPlane-egress-to-ipBlock")ipBlock defines policy on a particular IPBlock. If this field is set then neither of the other fields can be. ##### `cidr` string [​](#networkPolicy-controlPlane-egress-to-ipBlock-cidr "Direct link to networkPolicy-controlPlane-egress-to-ipBlock-cidr")CIDR defines the allowed workload public egress destination. Valid examples are "0.0.0.0/0", "192.168.1.0/24" or "2001:db8::/64" ##### `except` string\[] [​](#networkPolicy-controlPlane-egress-to-ipBlock-except "Direct link to networkPolicy-controlPlane-egress-to-ipBlock-except")Except is a slice of CIDRs that should not be included. Items outside the cidr range will be rejected. Valid examples are "192.168.1.0/24" or "2001:db8::/64". ### `workload` object [​](#networkPolicy-workload "Direct link to networkPolicy-workload")Workload network policy rules #### `publicEgress` object [​](#networkPolicy-workload-publicEgress "Direct link to networkPolicy-workload-publicEgress")PublicEgress holds the public outgoing connections options for the vCluster workloads. ##### `enabled` boolean true [​](#networkPolicy-workload-publicEgress-enabled "Direct link to networkPolicy-workload-publicEgress-enabled")Enabled defines if the workload public egress should be enabled or disabled. ##### `cidr` string 0.0.0.0/0 [​](#networkPolicy-workload-publicEgress-cidr "Direct link to networkPolicy-workload-publicEgress-cidr")CIDR defines the allowed workload public egress destination. Valid examples are "0.0.0.0/0", "192.168.1.0/24" or "2001:db8::/64" ##### `except` string\[] \[100.64.0.0/10 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16] [​](#networkPolicy-workload-publicEgress-except "Direct link to networkPolicy-workload-publicEgress-except")Except is a slice of CIDRs that should not be included. Items outside the cidr range will be rejected. Valid examples are "192.168.1.0/24" or "2001:db8::/64". #### `ingress` object\[] [​](#networkPolicy-workload-ingress "Direct link to networkPolicy-workload-ingress")Ingress rules for the vCluster workloads. ##### `ports` object\[] [​](#networkPolicy-workload-ingress-ports "Direct link to networkPolicy-workload-ingress-ports")ports is a list of ports which should be made accessible on the pods selected for this rule. Each item in this list is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list. ##### `protocol` string [​](#networkPolicy-workload-ingress-ports-protocol "Direct link to networkPolicy-workload-ingress-ports-protocol")protocol represents the protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this field defaults to TCP. ##### `port` object [​](#networkPolicy-workload-ingress-ports-port "Direct link to networkPolicy-workload-ingress-ports-port")port represents the port on the given protocol. This can either be a numerical or named port on a pod. If this field is not provided, this matches all port names and numbers. If present, only traffic on the specified protocol AND port will be matched. ##### `endPort` integer [​](#networkPolicy-workload-ingress-ports-endPort "Direct link to networkPolicy-workload-ingress-ports-endPort")endPort indicates that the range of ports from port to endPort if set, inclusive, should be allowed by the policy. This field cannot be defined if the port field is not defined or if the port field is defined as a named (string) port. The endPort must be equal or greater than port. ##### `from` object\[] [​](#networkPolicy-workload-ingress-from "Direct link to networkPolicy-workload-ingress-from")from is a list of sources which should be able to access the pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all sources (traffic not restricted by source). If this field is present and contains at least one item, this rule allows traffic only if the traffic matches at least one item in the from list. ##### `podSelector` object [​](#networkPolicy-workload-ingress-from-podSelector "Direct link to networkPolicy-workload-ingress-from-podSelector")podSelector is a label selector which selects pods. This field follows standard label selector semantics; if present but empty, it selects all pods.If namespaceSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the Namespaces selected by NamespaceSelector. Otherwise it selects the pods matching podSelector in the policy's own namespace. ##### `matchLabels` object [​](#networkPolicy-workload-ingress-from-podSelector-matchLabels "Direct link to networkPolicy-workload-ingress-from-podSelector-matchLabels") ##### `matchExpressions` object\[] [​](#networkPolicy-workload-ingress-from-podSelector-matchExpressions "Direct link to networkPolicy-workload-ingress-from-podSelector-matchExpressions") ##### `key` string [​](#networkPolicy-workload-ingress-from-podSelector-matchExpressions-key "Direct link to networkPolicy-workload-ingress-from-podSelector-matchExpressions-key") ##### `operator` string [​](#networkPolicy-workload-ingress-from-podSelector-matchExpressions-operator "Direct link to networkPolicy-workload-ingress-from-podSelector-matchExpressions-operator") ##### `values` string\[] [​](#networkPolicy-workload-ingress-from-podSelector-matchExpressions-values "Direct link to networkPolicy-workload-ingress-from-podSelector-matchExpressions-values") ##### `namespaceSelector` object [​](#networkPolicy-workload-ingress-from-namespaceSelector "Direct link to networkPolicy-workload-ingress-from-namespaceSelector")namespaceSelector selects namespaces using cluster-scoped labels. This field follows standard label selector semantics; if present but empty, it selects all namespaces.If podSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the namespaces selected by namespaceSelector. Otherwise it selects all pods in the namespaces selected by namespaceSelector. ##### `matchLabels` object [​](#networkPolicy-workload-ingress-from-namespaceSelector-matchLabels "Direct link to networkPolicy-workload-ingress-from-namespaceSelector-matchLabels") ##### `matchExpressions` object\[] [​](#networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions "Direct link to networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions") ##### `key` string [​](#networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions-key "Direct link to networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions-key") ##### `operator` string [​](#networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions-operator "Direct link to networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions-operator") ##### `values` string\[] [​](#networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions-values "Direct link to networkPolicy-workload-ingress-from-namespaceSelector-matchExpressions-values") ##### `ipBlock` object [​](#networkPolicy-workload-ingress-from-ipBlock "Direct link to networkPolicy-workload-ingress-from-ipBlock")ipBlock defines policy on a particular IPBlock. If this field is set then neither of the other fields can be. ##### `cidr` string [​](#networkPolicy-workload-ingress-from-ipBlock-cidr "Direct link to networkPolicy-workload-ingress-from-ipBlock-cidr")CIDR defines the allowed workload public egress destination. Valid examples are "0.0.0.0/0", "192.168.1.0/24" or "2001:db8::/64" ##### `except` string\[] [​](#networkPolicy-workload-ingress-from-ipBlock-except "Direct link to networkPolicy-workload-ingress-from-ipBlock-except")Except is a slice of CIDRs that should not be included. Items outside the cidr range will be rejected. Valid examples are "192.168.1.0/24" or "2001:db8::/64". #### `egress` object\[] [​](#networkPolicy-workload-egress "Direct link to networkPolicy-workload-egress")Egress rules for the vCluster workloads. ##### `ports` object\[] [​](#networkPolicy-workload-egress-ports "Direct link to networkPolicy-workload-egress-ports")ports is a list of destination ports for outgoing traffic. Each item in this list is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list. ##### `protocol` string [​](#networkPolicy-workload-egress-ports-protocol "Direct link to networkPolicy-workload-egress-ports-protocol")protocol represents the protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this field defaults to TCP. ##### `port` object [​](#networkPolicy-workload-egress-ports-port "Direct link to networkPolicy-workload-egress-ports-port")port represents the port on the given protocol. This can either be a numerical or named port on a pod. If this field is not provided, this matches all port names and numbers. If present, only traffic on the specified protocol AND port will be matched. ##### `endPort` integer [​](#networkPolicy-workload-egress-ports-endPort "Direct link to networkPolicy-workload-egress-ports-endPort")endPort indicates that the range of ports from port to endPort if set, inclusive, should be allowed by the policy. This field cannot be defined if the port field is not defined or if the port field is defined as a named (string) port. The endPort must be equal or greater than port. ##### `to` object\[] [​](#networkPolicy-workload-egress-to "Direct link to networkPolicy-workload-egress-to")to is a list of destinations for outgoing traffic of pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all destinations (traffic not restricted by destination). If this field is present and contains at least one item, this rule allows traffic only if the traffic matches at least one item in the to list. ##### `podSelector` object [​](#networkPolicy-workload-egress-to-podSelector "Direct link to networkPolicy-workload-egress-to-podSelector")podSelector is a label selector which selects pods. This field follows standard label selector semantics; if present but empty, it selects all pods.If namespaceSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the Namespaces selected by NamespaceSelector. Otherwise it selects the pods matching podSelector in the policy's own namespace. ##### `matchLabels` object [​](#networkPolicy-workload-egress-to-podSelector-matchLabels "Direct link to networkPolicy-workload-egress-to-podSelector-matchLabels") ##### `matchExpressions` object\[] [​](#networkPolicy-workload-egress-to-podSelector-matchExpressions "Direct link to networkPolicy-workload-egress-to-podSelector-matchExpressions") ##### `key` string [​](#networkPolicy-workload-egress-to-podSelector-matchExpressions-key "Direct link to networkPolicy-workload-egress-to-podSelector-matchExpressions-key") ##### `operator` string [​](#networkPolicy-workload-egress-to-podSelector-matchExpressions-operator "Direct link to networkPolicy-workload-egress-to-podSelector-matchExpressions-operator") ##### `values` string\[] [​](#networkPolicy-workload-egress-to-podSelector-matchExpressions-values "Direct link to networkPolicy-workload-egress-to-podSelector-matchExpressions-values") ##### `namespaceSelector` object [​](#networkPolicy-workload-egress-to-namespaceSelector "Direct link to networkPolicy-workload-egress-to-namespaceSelector")namespaceSelector selects namespaces using cluster-scoped labels. This field follows standard label selector semantics; if present but empty, it selects all namespaces.If podSelector is also set, then the NetworkPolicyPeer as a whole selects the pods matching podSelector in the namespaces selected by namespaceSelector. Otherwise it selects all pods in the namespaces selected by namespaceSelector. ##### `matchLabels` object [​](#networkPolicy-workload-egress-to-namespaceSelector-matchLabels "Direct link to networkPolicy-workload-egress-to-namespaceSelector-matchLabels") ##### `matchExpressions` object\[] [​](#networkPolicy-workload-egress-to-namespaceSelector-matchExpressions "Direct link to networkPolicy-workload-egress-to-namespaceSelector-matchExpressions") ##### `key` string [​](#networkPolicy-workload-egress-to-namespaceSelector-matchExpressions-key "Direct link to networkPolicy-workload-egress-to-namespaceSelector-matchExpressions-key") ##### `operator` string [​](#networkPolicy-workload-egress-to-namespaceSelector-matchExpressions-operator "Direct link to networkPolicy-workload-egress-to-namespaceSelector-matchExpressions-operator") ##### `values` string\[] [​](#networkPolicy-workload-egress-to-namespaceSelector-matchExpressions-values "Direct link to networkPolicy-workload-egress-to-namespaceSelector-matchExpressions-values") ##### `ipBlock` object [​](#networkPolicy-workload-egress-to-ipBlock "Direct link to networkPolicy-workload-egress-to-ipBlock")ipBlock defines policy on a particular IPBlock. If this field is set then neither of the other fields can be. ##### `cidr` string [​](#networkPolicy-workload-egress-to-ipBlock-cidr "Direct link to networkPolicy-workload-egress-to-ipBlock-cidr")CIDR defines the allowed workload public egress destination. Valid examples are "0.0.0.0/0", "192.168.1.0/24" or "2001:db8::/64" ##### `except` string\[] [​](#networkPolicy-workload-egress-to-ipBlock-except "Direct link to networkPolicy-workload-egress-to-ipBlock-except")Except is a slice of CIDRs that should not be included. Items outside the cidr range will be rejected. Valid examples are "192.168.1.0/24" or "2001:db8::/64". --- # Pod security standard Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** note This feature is disabled by default. [Pod security standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/) prevent Pods from starting if they request permissions beyond what's allowed. These standards check settings like `spec.securityContext`, host ports, volume types, and AppArmor annotations. Enable this feature to block privileged Pods from escaping the tenant cluster. ``` policies: podSecurityStandard: ``` * Replace `` with `privileged`, `baseline`, or `restricted`. See the [Kubernetes Pod Security profile details](https://kubernetes.io/docs/concepts/security/pod-security-standards/#profile-details) for more information. What this validates Pod Security Standards evaluate the Pod spec: fields like `spec.securityContext`, host ports, volume types, and AppArmor annotations. They don't inspect the backing source of a PersistentVolumeClaim's volume. They also don't validate a host-side Pod that a controller such as KubeVirt generates from a tenant custom resource. On shared nodes, prefer `restricted` over `baseline`, and pair this with the additional controls in the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md). ## Config reference[​](#config-reference "Direct link to Config reference") ## `podSecurityStandard` string [​](#podSecurityStandard "Direct link to podSecurityStandard")PodSecurityStandard that can be enforced can be one of: empty (""), baseline, restricted or privileged --- # Resource quota Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** You can control vCluster resource consumption by configuring a ResourceQuota. vCluster creates the ResourceQuota in the same namespace as vCluster itself. Resource quotas apply to all resources synced to the control plane cluster. ## Example[​](#example "Direct link to Example") If you would normally deploy the following ResourceQuota: ``` apiVersion: v1 kind: ResourceQuota metadata: name: pods-medium spec: hard: cpu: "10" memory: 20Gi pods: "10" scopeSelector: matchExpressions: - operator : In scopeName: PriorityClass values: ["medium"] ``` Instead configure the ResourceQuota in your `vcluster.yaml` config file like this: ``` policies: resourceQuota: enabled: true quota: cpu: "10" memory: 20Gi pods: "10" scopeSelector: matchExpressions: - operator : In scopeName: PriorityClass values: ["medium"] ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `resourceQuota` object [​](#resourceQuota "Direct link to resourceQuota")ResourceQuota specifies resource quota options. ### `enabled` string|boolean auto [​](#resourceQuota-enabled "Direct link to resourceQuota-enabled")Enabled defines if the resource quota should be enabled. "auto" means that if limitRange is enabled, the resourceQuota will be enabled as well. ### `quota` object map\[count/configmaps:100 count/endpoints:40 count/persistentvolumeclaims:20 count/pods:20 count/secrets:100 count/services:20 limits.cpu:20 limits.ephemeral-storage:160Gi limits.memory:40Gi requests.cpu:10 requests.ephemeral-storage:60Gi requests.memory:20Gi requests.storage:100Gi services.loadbalancers:1 services.nodeports:0] [​](#resourceQuota-quota "Direct link to resourceQuota-quota")Quota are the quota options ### `scopeSelector` object map\[matchExpressions:\[]] [​](#resourceQuota-scopeSelector "Direct link to resourceQuota-scopeSelector")ScopeSelector is the resource quota scope selector ### `scopes` string\[] \[] [​](#resourceQuota-scopes "Direct link to resourceQuota-scopes")Scopes are the resource quota scopes ### `annotations` object {} [​](#resourceQuota-annotations "Direct link to resourceQuota-annotations")Annotations are extra annotations for this resource. ### `labels` object {} [​](#resourceQuota-labels "Direct link to resourceQuota-labels")Labels are extra labels for this resource. --- # Private Nodes | | | Enterprise | | | | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Private Nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes.md) | | | | | | [Private Nodes VPN](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. Using private nodes is a tenancy model for vCluster where, instead of sharing the control plane cluster's worker nodes, individual worker nodes are joined to a vCluster. These private nodes act as the vCluster's worker nodes and are treated as worker nodes for the vCluster. Because these nodes are real Kubernetes nodes, vCluster does not sync any resources to the control plane cluster as no control plane cluster worker nodes are used. All workloads run directly on the attached nodes as if they were native to the tenant cluster. This approach can be understood as follows: the control plane creates a virtual Kubernetes context, while the workloads can run on separate physical nodes that exist entirely within that virtual context. One key benefit of using private nodes is that vCluster can automatically manage the lifecycle of the worker nodes. To allow nodes to join the tenant cluster, the cluster must be exposed and accessible. One option is to expose a `LoadBalancer` service and use its endpoint for the worker nodes. vCluster does not need direct access to the nodes; instead, it uses [Konnectivity](https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/) to tunnel requests—such as `kubectl logs`—from the control plane to the nodes.
![Private nodes architecture showing dedicated worker nodes per vCluster](/docs/media/diagrams/private-nodes.png) ## How private nodes can be provisioned[​](#how-private-nodes-can-be-provisioned "Direct link to How private nodes can be provisioned") Private nodes can be provisioned in two different ways: * **[Manually provisioned](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md)** - Nodes that were provisioned outside of vCluster. These nodes are joined to vCluster using a vCluster CLI command. * **[Automatically provisioned](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/auto-nodes.md)** - Nodes that are provisioned on-demand based on the vCluster configuration and resource requirements. vCluster is connected to vCluster Platform and references a node provider defined in vCluster Platform. ## vcluster.yaml configuration[​](#vclusteryaml-configuration "Direct link to vcluster.yaml configuration") To create a vCluster with private nodes, you'll need to enable the feature as well as set some other configuration options in preparation for worker nodes to join. Enable private nodes ``` # Enable private nodes privateNodes: enabled: true # vCluster control plane options controlPlane: service: spec: type: LoadBalancer # If you want to expose vCluster using LoadBalancer (Recommended option) # Networking configuration networking: # Specify the pod CIDR podCIDR: 10.64.0.0/16 # Specify the service CIDR serviceCIDR: 10.128.0.0/16 ``` ### Scheduling enabled by default[​](#scheduling-enabled-by-default "Direct link to Scheduling enabled by default") When private nodes are enabled, scheduling is handled by the tenant cluster and the virtual scheduler is automatically enabled. You do not need to manually configure this in your `vcluster.yaml`. Virtual scheduler automatically enabled with private nodes ``` # Enabling private nodes enables the virtual scheduler # Disabling the virtual scheduler is not possible controlPlane: distro: k8s: scheduler: enabled: true ``` ### Other vCluster feature limitations[​](#other-vcluster-feature-limitations "Direct link to Other vCluster feature limitations") Certain vCluster features are automatically disabled or unavailable. If you include these options in your `vcluster.yaml`, they are ignored or might cause configuration errors. The following features are not available: * `sync.*` - No resource syncing between tenant and control plane clusters * `integrations.*` - Integrations depend on syncing functionality * `networking.replicateServices` - Services are not replicated to host * `controlPlane.coredns.embedded: true` - Embedded CoreDNS conflicts with custom CNI * `controlPlane.distro.k8s.scheduler.enabled: false` - Virtual scheduler cannot be disabled * `sleep.*` - No ability to use auto sleep for workloads or control plane Unsupported vcluster.yaml options with private nodes ``` # These configurations are NOT supported with private nodes # Resource syncing between tenant and control plane clusters is disabled sync: toHost: services: enabled: false # Services cannot be synced to the control plane cluster secrets: enabled: false # Secrets cannot be synced to the control plane cluster # All other sync options (pods, configmaps, etc.) are also disabled # Platform integrations require syncing functionality integrations: metricsServer: enabled: false # Metrics server integration not supported # All other integrations are disabled due to sync dependency controlPlane: # DNS configuration limitations coredns: embedded: false # Embedded CoreDNS conflicts with custom CNI options distro: k8s: # Virtual scheduler is required for workload placement scheduler: enabled: true # Always enabled (cannot be disabled) # Host Path Mapper is not supported hostPathMapper: enabled: false # Service replication (networking.replicateServices) and sleep mode # (sleep.*) are also unavailable; see the list above. ``` ### Network policies[​](#network-policies "Direct link to Network policies") If you are using [network policies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md), private nodes traffic into the virtual cluster control plane must be allowed. vcluster.yaml ``` privateNodes: enabled: true controlPlane: service: spec: type: LoadBalancer policies: networkPolicy: enabled: true controlPlane: ingress: - from: # Allow incoming traffic from the load balancer internal IP address. # This example is allowing incoming traffic from any address. Load balancer internal CIDR should be used. - ipBlock: cidr: 0.0.0.0/0 ``` ## Features available with private nodes[​](#features-available-with-private-nodes "Direct link to Features available with private nodes") The following vCluster features are available when using private nodes as your tenancy model: ## Deploy vCluster by manually joining private nodes[​](#deploy-vcluster-by-manually-joining-private-nodes "Direct link to Deploy vCluster by manually joining private nodes") For step-by-step instructions on deploying vCluster with private nodes and joining worker nodes, see the [Deploy private nodes guide](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md). ## Default node profile[​](#default-node-profile "Direct link to Default node profile") Set `privateNodes.defaultProfile` to a [node profile](https://vcluster.com/docs/platform/administer/node-profiles/overview.md) name to apply it to any node that joins the tenant cluster without an explicit profile: Set a default node profile ``` privateNodes: enabled: true defaultProfile: general-compute ``` This takes precedence over the project's default node profile, if the project also has one configured. See [Assignment and precedence](https://vcluster.com/docs/platform/administer/node-profiles/overview.md#assignment-and-precedence) for the full resolution order. ## Config reference[​](#config-reference "Direct link to Config reference") ## `privateNodes` object [​](#privateNodes "Direct link to privateNodes")PrivateNodes holds configuration for vCluster private nodes mode. ### `enabled` boolean false [​](#privateNodes-enabled "Direct link to privateNodes-enabled")Enabled defines if dedicated nodes should be enabled. ### `kubelet` object [​](#privateNodes-kubelet "Direct link to privateNodes-kubelet")Kubelet holds kubelet configuration that is used for all nodes. #### `config` object {} [​](#privateNodes-kubelet-config "Direct link to privateNodes-kubelet-config")Config is the config for the kubelet that will be merged into the default kubelet config. More information can be found here: ### `autoUpgrade` object [​](#privateNodes-autoUpgrade "Direct link to privateNodes-autoUpgrade")AutoUpgrade holds configuration for auto upgrade. #### `enabled` boolean true [​](#privateNodes-autoUpgrade-enabled "Direct link to privateNodes-autoUpgrade-enabled")Enabled defines if auto upgrade should be enabled. #### `image` string [​](#privateNodes-autoUpgrade-image "Direct link to privateNodes-autoUpgrade-image")Image is the image for the auto upgrade pod started by vCluster. If empty defaults to the controlPlane.statefulSet.image. #### `imagePullPolicy` string [​](#privateNodes-autoUpgrade-imagePullPolicy "Direct link to privateNodes-autoUpgrade-imagePullPolicy")ImagePullPolicy is the policy how to pull the image. #### `nodeSelector` object [​](#privateNodes-autoUpgrade-nodeSelector "Direct link to privateNodes-autoUpgrade-nodeSelector")NodeSelector is the node selector for the auto upgrade. If empty will select all worker nodes. #### `binariesPath` string [​](#privateNodes-autoUpgrade-binariesPath "Direct link to privateNodes-autoUpgrade-binariesPath")BinariesPath is the base path for the kubeadm binaries. Defaults to /usr/local/bin #### `cniBinariesPath` string [​](#privateNodes-autoUpgrade-cniBinariesPath "Direct link to privateNodes-autoUpgrade-cniBinariesPath")CNIBinariesPath is the base path for the CNI binaries. Defaults to /opt/cni/bin #### `concurrency` integer 1 [​](#privateNodes-autoUpgrade-concurrency "Direct link to privateNodes-autoUpgrade-concurrency")Concurrency is the number of nodes that can be upgraded at the same time. #### `podSecurityContext` object {} [​](#privateNodes-autoUpgrade-podSecurityContext "Direct link to privateNodes-autoUpgrade-podSecurityContext")PodSecurityContext specifies security context options on the pod level for the upgrade pod. #### `containerSecurityContext` object {} [​](#privateNodes-autoUpgrade-containerSecurityContext "Direct link to privateNodes-autoUpgrade-containerSecurityContext")ContainerSecurityContext specifies security context options on the container level for the upgrade container. ### `joinNode` object [​](#privateNodes-joinNode "Direct link to privateNodes-joinNode")JoinNode holds configuration specifically used during joining the node (see "kubeadm join"). #### `preInstallCommands` string\[] [​](#privateNodes-joinNode-preInstallCommands "Direct link to privateNodes-joinNode-preInstallCommands")PreInstallCommands are commands that will be executed before containerd, kubelet etc. is installed. #### `preJoinCommands` string\[] [​](#privateNodes-joinNode-preJoinCommands "Direct link to privateNodes-joinNode-preJoinCommands")PreJoinCommands are commands that will be executed before kubeadm join is executed. #### `postJoinCommands` string\[] [​](#privateNodes-joinNode-postJoinCommands "Direct link to privateNodes-joinNode-postJoinCommands")PostJoinCommands are commands that will be executed after kubeadm join is executed. #### `containerd` object [​](#privateNodes-joinNode-containerd "Direct link to privateNodes-joinNode-containerd")Containerd holds configuration for the containerd join process. ##### `enabled` boolean true [​](#privateNodes-joinNode-containerd-enabled "Direct link to privateNodes-joinNode-containerd-enabled")Enabled defines if containerd should be installed and configured by vCluster. ##### `registry` object [​](#privateNodes-joinNode-containerd-registry "Direct link to privateNodes-joinNode-containerd-registry")Registry holds configuration for how containerd should be configured to use a registries. ##### `configPath` string [​](#privateNodes-joinNode-containerd-registry-configPath "Direct link to privateNodes-joinNode-containerd-registry-configPath")ConfigPath is the path to the containerd registry config. ##### `mirrors` {key: object} [​](#privateNodes-joinNode-containerd-registry-mirrors "Direct link to privateNodes-joinNode-containerd-registry-mirrors")Mirrors holds configuration for the containerd registry mirrors. E.g. myregistry.io:5000 or docker.io. See for more details. ##### `server` string [​](#privateNodes-joinNode-containerd-registry-mirrors-server "Direct link to privateNodes-joinNode-containerd-registry-mirrors-server")Server is the fallback server to use for the containerd registry mirror. E.g. . See for more details. ##### `caCert` string\[] [​](#privateNodes-joinNode-containerd-registry-mirrors-caCert "Direct link to privateNodes-joinNode-containerd-registry-mirrors-caCert")CACert are paths to CA certificates to use for the containerd registry mirror. ##### `skipVerify` boolean [​](#privateNodes-joinNode-containerd-registry-mirrors-skipVerify "Direct link to privateNodes-joinNode-containerd-registry-mirrors-skipVerify")SkipVerify is a boolean to skip the certificate verification for the containerd registry mirror and allows http connections. ##### `capabilities` string\[] [​](#privateNodes-joinNode-containerd-registry-mirrors-capabilities "Direct link to privateNodes-joinNode-containerd-registry-mirrors-capabilities")Capabilities is a list of capabilities to enable for the containerd registry mirror. If empty, will use pull and resolve capabilities. ##### `overridePath` boolean [​](#privateNodes-joinNode-containerd-registry-mirrors-overridePath "Direct link to privateNodes-joinNode-containerd-registry-mirrors-overridePath")OverridePath is a boolean to override the path for the containerd registry mirror. ##### `hosts` object\[] [​](#privateNodes-joinNode-containerd-registry-mirrors-hosts "Direct link to privateNodes-joinNode-containerd-registry-mirrors-hosts")Hosts holds configuration for the containerd registry mirror hosts. See for more details. ##### `server` string [​](#privateNodes-joinNode-containerd-registry-mirrors-hosts-server "Direct link to privateNodes-joinNode-containerd-registry-mirrors-hosts-server")Server is the server to use for the containerd registry mirror host. E.g. . ##### `caCert` string\[] [​](#privateNodes-joinNode-containerd-registry-mirrors-hosts-caCert "Direct link to privateNodes-joinNode-containerd-registry-mirrors-hosts-caCert")CACert are paths to CA certificates to use for the containerd registry mirror host. ##### `skipVerify` boolean [​](#privateNodes-joinNode-containerd-registry-mirrors-hosts-skipVerify "Direct link to privateNodes-joinNode-containerd-registry-mirrors-hosts-skipVerify")SkipVerify is a boolean to skip the certificate verification for the containerd registry mirror and allows http connections. ##### `capabilities` string\[] [​](#privateNodes-joinNode-containerd-registry-mirrors-hosts-capabilities "Direct link to privateNodes-joinNode-containerd-registry-mirrors-hosts-capabilities")Capabilities is a list of capabilities to enable for the containerd registry mirror. If empty, will use pull and resolve capabilities. ##### `overridePath` boolean [​](#privateNodes-joinNode-containerd-registry-mirrors-hosts-overridePath "Direct link to privateNodes-joinNode-containerd-registry-mirrors-hosts-overridePath")OverridePath is a boolean to override the path for the containerd registry mirror. ##### `auth` {key: object} [​](#privateNodes-joinNode-containerd-registry-auth "Direct link to privateNodes-joinNode-containerd-registry-auth")Auth holds configuration for the containerd registry auth. See for more details. ##### `username` string [​](#privateNodes-joinNode-containerd-registry-auth-username "Direct link to privateNodes-joinNode-containerd-registry-auth-username")Username is the username for the containerd registry. ##### `password` string [​](#privateNodes-joinNode-containerd-registry-auth-password "Direct link to privateNodes-joinNode-containerd-registry-auth-password")Password is the password for the containerd registry. ##### `identityToken` string [​](#privateNodes-joinNode-containerd-registry-auth-identityToken "Direct link to privateNodes-joinNode-containerd-registry-auth-identityToken")IdentityToken is the token for the containerd registry. ##### `auth` string [​](#privateNodes-joinNode-containerd-registry-auth-auth "Direct link to privateNodes-joinNode-containerd-registry-auth-auth")Auth is the auth config for the containerd registry. ##### `pauseImage` string [​](#privateNodes-joinNode-containerd-pauseImage "Direct link to privateNodes-joinNode-containerd-pauseImage")PauseImage is the image for the pause container. #### `caCertPath` string [​](#privateNodes-joinNode-caCertPath "Direct link to privateNodes-joinNode-caCertPath")CACertPath is the path to the SSL certificate authority used to secure communications between node and control-plane. Defaults to "/etc/kubernetes/pki/ca.crt". #### `skipPhases` string\[] [​](#privateNodes-joinNode-skipPhases "Direct link to privateNodes-joinNode-skipPhases")SkipPhases is a list of phases to skip during command execution. The list of phases can be obtained with the "kubeadm join --help" command. #### `nodeRegistration` object [​](#privateNodes-joinNode-nodeRegistration "Direct link to privateNodes-joinNode-nodeRegistration")NodeRegistration holds configuration for the node registration similar to the kubeadm node registration. ##### `criSocket` string [​](#privateNodes-joinNode-nodeRegistration-criSocket "Direct link to privateNodes-joinNode-nodeRegistration-criSocket")CRI socket is the socket for the CRI. ##### `kubeletExtraArgs` object\[] [​](#privateNodes-joinNode-nodeRegistration-kubeletExtraArgs "Direct link to privateNodes-joinNode-nodeRegistration-kubeletExtraArgs")KubeletExtraArgs passes through extra arguments to the kubelet. The arguments here are passed to the kubelet command line via the environment file kubeadm writes at runtime for the kubelet to source. This overrides the generic base-level configuration in the kubelet-config ConfigMap Flags have higher priority when parsing. These values are local and specific to the node kubeadm is executing on. An argument name in this list is the flag name as it appears on the command line except without leading dash(es). Extra arguments will override existing default arguments. Duplicate extra arguments are allowed. ##### `name` string [​](#privateNodes-joinNode-nodeRegistration-kubeletExtraArgs-name "Direct link to privateNodes-joinNode-nodeRegistration-kubeletExtraArgs-name")Name is the name of the argument. ##### `value` string [​](#privateNodes-joinNode-nodeRegistration-kubeletExtraArgs-value "Direct link to privateNodes-joinNode-nodeRegistration-kubeletExtraArgs-value")Value is the value of the argument. ##### `taints` object\[] [​](#privateNodes-joinNode-nodeRegistration-taints "Direct link to privateNodes-joinNode-nodeRegistration-taints")Taints are additional taints to set for the kubelet. ##### `key` string [​](#privateNodes-joinNode-nodeRegistration-taints-key "Direct link to privateNodes-joinNode-nodeRegistration-taints-key")Required. The taint key to be applied to a node. ##### `value` string [​](#privateNodes-joinNode-nodeRegistration-taints-value "Direct link to privateNodes-joinNode-nodeRegistration-taints-value")The taint value corresponding to the taint key. ##### `effect` string [​](#privateNodes-joinNode-nodeRegistration-taints-effect "Direct link to privateNodes-joinNode-nodeRegistration-taints-effect")Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. ##### `ignorePreflightErrors` string\[] [​](#privateNodes-joinNode-nodeRegistration-ignorePreflightErrors "Direct link to privateNodes-joinNode-nodeRegistration-ignorePreflightErrors")IgnorePreflightErrors provides a slice of pre-flight errors to be ignored when the current node is registered, e.g. 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. ##### `imagePullPolicy` string [​](#privateNodes-joinNode-nodeRegistration-imagePullPolicy "Direct link to privateNodes-joinNode-nodeRegistration-imagePullPolicy")ImagePullPolicy specifies the policy for image pulling during kubeadm "init" and "join" operations. The value of this field must be one of "Always", "IfNotPresent" or "Never". If this field is unset kubeadm will default it to "IfNotPresent", or pull the required images if not present on the host. ### `autoNodes` object\[] [​](#privateNodes-autoNodes "Direct link to privateNodes-autoNodes")AutoNodes stores auto nodes configuration. #### `provider` required string [​](#privateNodes-autoNodes-provider "Direct link to privateNodes-autoNodes-provider")Provider is the node provider of the nodes in this pool. #### `properties` object [​](#privateNodes-autoNodes-properties "Direct link to privateNodes-autoNodes-properties")Properties are the node provider properties. This is a simple key value map and can contain things like region, subscription, etc. that is then used by the node provider to create the nodes and node environment. #### `static` object\[] [​](#privateNodes-autoNodes-static "Direct link to privateNodes-autoNodes-static")Static defines static node pools. Static node pools have a fixed size and are not scaled automatically. ##### `name` required string [​](#privateNodes-autoNodes-static-name "Direct link to privateNodes-autoNodes-static-name")Name is the name of this static nodePool ##### `properties` object [​](#privateNodes-autoNodes-static-properties "Direct link to privateNodes-autoNodes-static-properties")Properties are extra properties that will be set directly on the node claim. ##### `nodeTypeSelector` object\[] [​](#privateNodes-autoNodes-static-nodeTypeSelector "Direct link to privateNodes-autoNodes-static-nodeTypeSelector")NodeTypeSelector filters the types of nodes that can be provisioned by this pool. All requirements must be met for a node type to be eligible. ##### `property` required string [​](#privateNodes-autoNodes-static-nodeTypeSelector-property "Direct link to privateNodes-autoNodes-static-nodeTypeSelector-property")Property is the property on the node type to select. ##### `operator` string [​](#privateNodes-autoNodes-static-nodeTypeSelector-operator "Direct link to privateNodes-autoNodes-static-nodeTypeSelector-operator")Operator is the comparison operator, such as "In", "NotIn", "Exists". If empty, defaults to "In". ##### `values` string\[] [​](#privateNodes-autoNodes-static-nodeTypeSelector-values "Direct link to privateNodes-autoNodes-static-nodeTypeSelector-values")Values is the list of values to use for comparison. This is mutually exclusive with value. ##### `value` string [​](#privateNodes-autoNodes-static-nodeTypeSelector-value "Direct link to privateNodes-autoNodes-static-nodeTypeSelector-value")Value is the value to use for comparison. This is mutually exclusive with values. ##### `taints` object\[] [​](#privateNodes-autoNodes-static-taints "Direct link to privateNodes-autoNodes-static-taints")Taints are the taints to apply to the nodes in this pool. ##### `key` string [​](#privateNodes-autoNodes-static-taints-key "Direct link to privateNodes-autoNodes-static-taints-key")Required. The taint key to be applied to a node. ##### `value` string [​](#privateNodes-autoNodes-static-taints-value "Direct link to privateNodes-autoNodes-static-taints-value")The taint value corresponding to the taint key. ##### `effect` string [​](#privateNodes-autoNodes-static-taints-effect "Direct link to privateNodes-autoNodes-static-taints-effect")Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. ##### `nodeLabels` object [​](#privateNodes-autoNodes-static-nodeLabels "Direct link to privateNodes-autoNodes-static-nodeLabels")NodeLabels are the labels to apply to the nodes in this pool. ##### `terminationGracePeriod` string [​](#privateNodes-autoNodes-static-terminationGracePeriod "Direct link to privateNodes-autoNodes-static-terminationGracePeriod")TerminationGracePeriod is the maximum duration the controller will wait before forcefully deleting the pods on a node, measured from when deletion is first initiated.Warning: this feature takes precedence over a Pod's terminationGracePeriodSeconds value, and bypasses any blocked PDBs or the karpenter.sh/do-not-disrupt annotation.This field is intended to be used by cluster administrators to enforce that nodes can be cycled within a given time period. When set, drifted nodes will begin draining even if there are pods blocking eviction. Draining will respect PDBs and the do-not-disrupt annotation until the TGP is reached.Karpenter will preemptively delete pods so their terminationGracePeriodSeconds align with the node's terminationGracePeriod. If a pod would be terminated without being granted its full terminationGracePeriodSeconds prior to the node timeout, that pod will be deleted at T = node timeout - pod terminationGracePeriodSeconds.The feature can also be used to allow maximum time limits for long-running jobs which can delay node termination with preStop hooks. Defaults to 30s. Set to Never to wait indefinitely for pods to be drained. ##### `quantity` required integer [​](#privateNodes-autoNodes-static-quantity "Direct link to privateNodes-autoNodes-static-quantity")Quantity is the number of desired nodes in this pool. ##### `profile` string [​](#privateNodes-autoNodes-static-profile "Direct link to privateNodes-autoNodes-static-profile")Profile references a NodeProfile applied to nodes created by this pool. #### `dynamic` object\[] [​](#privateNodes-autoNodes-dynamic "Direct link to privateNodes-autoNodes-dynamic")Dynamic defines dynamic node pools. Dynamic node pools are scaled automatically based on the requirements within the cluster. Karpenter is used under the hood to handle the scheduling of the nodes. ##### `name` required string [​](#privateNodes-autoNodes-dynamic-name "Direct link to privateNodes-autoNodes-dynamic-name")Name is the name of this NodePool ##### `properties` object [​](#privateNodes-autoNodes-dynamic-properties "Direct link to privateNodes-autoNodes-dynamic-properties")Properties are extra properties that will be set directly on the node claim. ##### `nodeTypeSelector` object\[] [​](#privateNodes-autoNodes-dynamic-nodeTypeSelector "Direct link to privateNodes-autoNodes-dynamic-nodeTypeSelector")NodeTypeSelector filters the types of nodes that can be provisioned by this pool. All requirements must be met for a node type to be eligible. ##### `property` required string [​](#privateNodes-autoNodes-dynamic-nodeTypeSelector-property "Direct link to privateNodes-autoNodes-dynamic-nodeTypeSelector-property")Property is the property on the node type to select. ##### `operator` string [​](#privateNodes-autoNodes-dynamic-nodeTypeSelector-operator "Direct link to privateNodes-autoNodes-dynamic-nodeTypeSelector-operator")Operator is the comparison operator, such as "In", "NotIn", "Exists". If empty, defaults to "In". ##### `values` string\[] [​](#privateNodes-autoNodes-dynamic-nodeTypeSelector-values "Direct link to privateNodes-autoNodes-dynamic-nodeTypeSelector-values")Values is the list of values to use for comparison. This is mutually exclusive with value. ##### `value` string [​](#privateNodes-autoNodes-dynamic-nodeTypeSelector-value "Direct link to privateNodes-autoNodes-dynamic-nodeTypeSelector-value")Value is the value to use for comparison. This is mutually exclusive with values. ##### `taints` object\[] [​](#privateNodes-autoNodes-dynamic-taints "Direct link to privateNodes-autoNodes-dynamic-taints")Taints are the taints to apply to the nodes in this pool. ##### `key` string [​](#privateNodes-autoNodes-dynamic-taints-key "Direct link to privateNodes-autoNodes-dynamic-taints-key")Required. The taint key to be applied to a node. ##### `value` string [​](#privateNodes-autoNodes-dynamic-taints-value "Direct link to privateNodes-autoNodes-dynamic-taints-value")The taint value corresponding to the taint key. ##### `effect` string [​](#privateNodes-autoNodes-dynamic-taints-effect "Direct link to privateNodes-autoNodes-dynamic-taints-effect")Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. ##### `nodeLabels` object [​](#privateNodes-autoNodes-dynamic-nodeLabels "Direct link to privateNodes-autoNodes-dynamic-nodeLabels")NodeLabels are the labels to apply to the nodes in this pool. ##### `limits` object [​](#privateNodes-autoNodes-dynamic-limits "Direct link to privateNodes-autoNodes-dynamic-limits")Limits specify the maximum resources that can be provisioned by this node pool, mapping to the 'limits' field in Karpenter's NodePool API. ##### `disruption` object [​](#privateNodes-autoNodes-dynamic-disruption "Direct link to privateNodes-autoNodes-dynamic-disruption")Disruption contains the parameters that relate to Karpenter's disruption logic ##### `consolidateAfter` string [​](#privateNodes-autoNodes-dynamic-disruption-consolidateAfter "Direct link to privateNodes-autoNodes-dynamic-disruption-consolidateAfter")ConsolidateAfter is the duration the controller will wait before attempting to terminate nodes that are underutilized. Refer to ConsolidationPolicy for how underutilization is considered. ##### `consolidationPolicy` string [​](#privateNodes-autoNodes-dynamic-disruption-consolidationPolicy "Direct link to privateNodes-autoNodes-dynamic-disruption-consolidationPolicy")ConsolidationPolicy describes which nodes Karpenter can disrupt through its consolidation algorithm. This policy defaults to "WhenEmptyOrUnderutilized" if not specified ##### `budgets` object\[] [​](#privateNodes-autoNodes-dynamic-disruption-budgets "Direct link to privateNodes-autoNodes-dynamic-disruption-budgets")Budgets is a list of Budgets. If there are multiple active budgets, Karpenter uses the most restrictive value. If left undefined, this will default to one budget with a value to 10%. ##### `nodes` string [​](#privateNodes-autoNodes-dynamic-disruption-budgets-nodes "Direct link to privateNodes-autoNodes-dynamic-disruption-budgets-nodes")Nodes dictates the maximum number of NodeClaims owned by this NodePool that can be terminating at once. This is calculated by counting nodes that have a deletion timestamp set, or are actively being deleted by Karpenter. This field is required when specifying a budget. ##### `schedule` string [​](#privateNodes-autoNodes-dynamic-disruption-budgets-schedule "Direct link to privateNodes-autoNodes-dynamic-disruption-budgets-schedule")Schedule specifies when a budget begins being active, following the upstream cronjob syntax. If omitted, the budget is always active. Timezones are not supported. ##### `duration` string [​](#privateNodes-autoNodes-dynamic-disruption-budgets-duration "Direct link to privateNodes-autoNodes-dynamic-disruption-budgets-duration")Duration determines how long a Budget is active since each Schedule hit. Only minutes and hours are accepted, as cron does not work in seconds. If omitted, the budget is always active. This is required if Schedule is set. ##### `terminationGracePeriod` string [​](#privateNodes-autoNodes-dynamic-terminationGracePeriod "Direct link to privateNodes-autoNodes-dynamic-terminationGracePeriod")TerminationGracePeriod is the maximum duration the controller will wait before forcefully deleting the pods on a node, measured from when deletion is first initiated.Warning: this feature takes precedence over a Pod's terminationGracePeriodSeconds value, and bypasses any blocked PDBs or the karpenter.sh/do-not-disrupt annotation.This field is intended to be used by cluster administrators to enforce that nodes can be cycled within a given time period. When set, drifted nodes will begin draining even if there are pods blocking eviction. Draining will respect PDBs and the do-not-disrupt annotation until the TGP is reached.Karpenter will preemptively delete pods so their terminationGracePeriodSeconds align with the node's terminationGracePeriod. If a pod would be terminated without being granted its full terminationGracePeriodSeconds prior to the node timeout, that pod will be deleted at T = node timeout - pod terminationGracePeriodSeconds.The feature can also be used to allow maximum time limits for long-running jobs which can delay node termination with preStop hooks. Defaults to 30s. Set to Never to wait indefinitely for pods to be drained. ##### `expireAfter` string [​](#privateNodes-autoNodes-dynamic-expireAfter "Direct link to privateNodes-autoNodes-dynamic-expireAfter")The amount of time a Node can live on the cluster before being removed ##### `weight` integer [​](#privateNodes-autoNodes-dynamic-weight "Direct link to privateNodes-autoNodes-dynamic-weight")Weight is the weight of this node pool. ##### `profile` string [​](#privateNodes-autoNodes-dynamic-profile "Direct link to privateNodes-autoNodes-dynamic-profile")Profile references a NodeProfile applied to nodes created by this pool. ### `vpn` object [​](#privateNodes-vpn "Direct link to privateNodes-vpn")VPN holds configuration for the private nodes vpn. This can be used to connect the private nodes to the control plane or connect the private nodes to each other if they are not running in the same network. Platform connection is required for the vpn to work. #### `enabled` boolean false [​](#privateNodes-vpn-enabled "Direct link to privateNodes-vpn-enabled")Enabled defines if the private nodes vpn should be enabled. #### `nodeToNode` object [​](#privateNodes-vpn-nodeToNode "Direct link to privateNodes-vpn-nodeToNode")NodeToNode holds configuration for the node to node vpn. This can be used to connect the private nodes to each other if they are not running in the same network. ##### `enabled` boolean false [​](#privateNodes-vpn-nodeToNode-enabled "Direct link to privateNodes-vpn-nodeToNode-enabled")Enabled defines if the node to node vpn should be enabled. ### `daemon` object [​](#privateNodes-daemon "Direct link to privateNodes-daemon")Daemon holds configuration for the private nodes daemon that is deployed on the nodes. #### `enabled` boolean false [​](#privateNodes-daemon-enabled "Direct link to privateNodes-daemon-enabled")Enabled defines if the private nodes daemon should be enabled. #### `controlPlaneLoadBalancer` object [​](#privateNodes-daemon-controlPlaneLoadBalancer "Direct link to privateNodes-daemon-controlPlaneLoadBalancer")ControlPlaneLoadBalancer holds configuration for the control plane load balancer. This is used to load balance the control plane traffic on the node to the control plane nodes. This is useful to achieve true high availability for the control plane without having to deploy a separate load balancer. ##### `enabled` boolean false [​](#privateNodes-daemon-controlPlaneLoadBalancer-enabled "Direct link to privateNodes-daemon-controlPlaneLoadBalancer-enabled")Enabled defines if the control plane load balancer should be enabled. The control plane load balancer is used to load balance the control plane traffic on the node to the control plane nodes. ##### `kubeProxy` boolean true [​](#privateNodes-daemon-controlPlaneLoadBalancer-kubeProxy "Direct link to privateNodes-daemon-controlPlaneLoadBalancer-kubeProxy")KubeProxy defines if the kube proxy should be proxied through the control plane load balancer as well. ##### `port` integer 11343 [​](#privateNodes-daemon-controlPlaneLoadBalancer-port "Direct link to privateNodes-daemon-controlPlaneLoadBalancer-port")Port defines the port for the control plane load balancer. ### `defaultProfile` string [​](#privateNodes-defaultProfile "Direct link to privateNodes-defaultProfile")DefaultProfile is the name of a NodeProfile applied to nodes joining this vCluster that do not select one explicitly. Overridden by per-pool or per-node selection. Requires a platform connection to resolve. --- # Auto Nodes | | | Enterprise | | | | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Private Nodes Auto Nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/auto-nodes.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). You can configure vCluster to automatically provision and join worker nodes based on the node and resource requirements. To use auto nodes, you need vCluster Platform installed and vCluster needs to be connected to it. The feature is based on [Karpenter](https://karpenter.sh/), which is a cluster autoscaler for Kubernetes that chooses the best node for the requested amount of pods and resources. Karpenter is built into vCluster and does not need to be installed. This allows vCluster to use Karpenter to do the node management and scheduling. The provisioning of the nodes is handled by vCluster Platform.
vCluster provides multiple [node providers](https://vcluster.com/docs/platform/administer/node-providers/overview.md) to automatically create worker nodes: * Terraform * KubeVirt * Nvidia Base Command Manager ### How does it work?[​](#how-does-it-work "Direct link to How does it work?") vCluster uses [Karpenter](https://karpenter.sh/docs/) to provision nodes. Karpenter is an open source node autoscaler built for Kubernetes which can dramatically improve the efficiency and cost of running workloads on that cluster. Karpenter works by: * **Watching** for pods that the Kubernetes scheduler has marked as unschedulable * **Evaluating** scheduling constraints (resource requests, nodeselectors, affinities, tolerations, and topology spread constraints) requested by the pods * **Provisioning** nodes that meet the requirements of the pods * **Disrupting** the nodes when the nodes are no longer needed Under the hood, vCluster creates [Karpenter NodePools](https://karpenter.sh/docs/concepts/nodepools/) which are read-only to the end-users inside the vCluster. When Karpenter finds unschedulable pods inside the vCluster or if there is a static node pool that does not have the desired quantity of nodes deployed, a new [Karpenter NodeClaim](https://karpenter.sh/docs/concepts/nodeclaims/) is created. Out of the Karpenter node claim, vCluster will create a platform node claim, which then is assigned to a node type and provisioned by the node provider (e.g. terraform). Terraform then creates a new node according to the specified terraform script and joins the node via [cloud-init](https://cloud-init.io/) into the vCluster.
For each node claim, Karpenter specifies the potentially fitting platform node types sorted by cost as well as the requested resources for the node. In addition, requirements within node pools can be defined by the user to filter node types, e.g. by region, cpu generation or custom defined properties. The platform will then do the final scheduling based on Karpenter suggestions what node type to use and then ultimately provision. The decided node type is then passed to the node provider which in turn creates the actual node based on the node type and joins it into the vCluster. #### Scheduling Example[​](#scheduling-example "Direct link to Scheduling Example") For example, when using the following `vcluster.yaml` configuration: ``` privateNodes: enabled: true autoNodes: provider: my-node-provider properties: my-custom_property: my-value dynamic: - name: my-node-pool ``` The following Karpenter NodePool will get created: ``` apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: my-node-pool spec: template: metadata: labels: vcluster.loft.sh/provider-platform: my-node-provider spec: requirements: - key: my-custom-property operator: In values: - my-value ``` When Karpenter decides to create a new node, it will create a new node claim resource that looks like this (the `j6b8n` is an example of the auto-generated ID by Kubernetes): ``` apiVersion: karpenter.sh/v1 kind: NodeClaim metadata: name: my-node-pool-j6b8n spec: requirements: - key: my-custom-property operator: In values: - my-value - key: node.kubernetes.io/instance-type operator: In values: - my-node-provider.large - my-node-provider.medium resources: requests: cpu: 120m memory: 114Mi pods: '5' ``` vCluster will then create a new platform node claim in the project the vCluster was created or connected to: ``` apiVersion: management.loft.sh/v1 kind: NodeClaim metadata: name: vcluster-62ftb namespace: p-default spec: desiredCapacity: cpu: 120m memory: 114Mi pods: '5' requirements: - key: my-custom_property operator: In values: - my-value - key: node.kubernetes.io/instance-type operator: In values: - my-node-provider.large - my-node-provider.medium providerRef: my-node-provider vClusterRef: vcluster ``` Karpenter will only list the node types that would actually fit the desired capacity, so there might be a `my-node-provider.small` which would have been filtered out by Karpenter already. The platform will then decide what node type to use based on the specified requirements and then set the `spec.typeRef` field. In this case it would use the `my-node-provider.medium` node type as that is cheaper than the large one an then create a node based with the node provider (e.g. through `tofu apply`). warning Additional workloads and pods might be introduced across different vCluster versions which would require additional resources. When auto node is enabled in older versions, the vCluster will automatically provision more nodes to meet the increased resource demands in newer versions. ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * vCluster control plane has to be running and in `Ready` state and connected to vCluster Platform * A [node provider](https://vcluster.com/docs/platform/administer/node-providers/overview.md) is configured in vCluster Platform. ### Node pools[​](#node-pools "Direct link to Node pools") There are two types of node pools, that can be figured independently or combined with each other. * **static**: Defines a fixed quantity of each node to provision. When the number of nodes is changed, the built-in [Karpenter](https://karpenter.sh/) automatically scales up/down the number of nodes. * **dynamic**: No quantity of nodes is defined. The built-in [Karpenter](https://karpenter.sh/) automatically decides how many nodes are needed. You can define a limit of nodes to provision. Deciding how to limit which node types to provision is based on the requirements defined in each node pool. Example with both static and dynamic node pools ``` privateNodes: enabled: true autoNodes: - provider: my-node-provider # Fixed size node pool of 2 static: - name: my-static-node-pool quantity: 2 # Dynamic node pool dynamic: - name: my-dynamic-node-pool limits: nodes: "3" ``` No vCluster restart required Changing fields within `privateNodes.autoNodes` will not restart the vCluster even on a `helm upgrade` It's also possible to mix different providers within the same vCluster. You can specify the provider using the `provider` field that should reference a node provider created in the platform. warning Once the node pool is created using the configured provider in vCluster, you can not edit the provider later. ### Node type selector[​](#node-type-selector "Direct link to Node type selector") Node type selector on a node pool can be used to include or exclude certain node types. These allow you to select properties on node types using [Kubernetes set-based requirements](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#set-based-requirement). Examples of how to set node type selector ``` privateNodes: enabled: true autoNodes: - provider: my-provider static: - name: my-static-pool quantity: 1 nodeTypeSelector: # Exact match - property: my-property value: my-value # One of - property: my-property operator: In values: - value-1 - value-2 - value-3 # Not in - property: my-property operator: NotIn values: - value-1 - value-2 - value-3 # Exists - property: my-property operator: Exists # Not exists - property: my-property operator: NotExists ``` The following operators are available and supported: * `In` (default): Matches one or multiple values on the node type * `NotIn`: Matches if the given values aren't part of the properties * `Exists`: Matches if the property is defined on the node type * `NotExists`: Matches if the property is not defined on the node type When an invalid property is updated to a valid value, or a valid property is changed to an invalid one, the nodes are automatically rolled out and updated to comply with the new configuration. warning Changing the node region is not supported. If the node region property is modified, the nodes will not roll out to the new region, and the nodes in the previous region will not be cleaned up automatically. #### Built-in node type properties[​](#built-in-node-type-properties "Direct link to Built-in node type properties") Each node type automatically has the following properties available. You can also add custom properties to node types. | Property | Value | Use Case | | ---------------------------------- | --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `vcluster.com/node-type` | The name of the node type to use. | Map vCluster node pools to only use a specific node type. Since node type names are globally unique, they also always map to a single node provider. | | `node.kubernetes.io/instance-type` | Same as `vcluster.com/node-type`, but just the official Kubernetes label. | Map vCluster node pools to only use a specific node type. Since node type names are globally unique, they also always map to a single node provider. | | `topology.kubernetes.io/zone` | Maps to the `spec.zone` field of the node type. If unspecified, will be `global`. | Map vCluster node pools to only use specific regions of node types. | | `karpenter.sh/capacity-type` | Fixed to `on-demand`. | Only on-demand nodes are supported. | | `kubernetes.io/os` | Fixed to `linux`. | Only Linux nodes are supported. | ### Dynamic node pools[​](#dynamic-node-pools "Direct link to Dynamic node pools") Dynamic node pools are powered by Karpenter, and for each dynamic node pool a [Karpenter node pool](https://karpenter.sh/docs/concepts/nodepools/) is created. Example of only using nodes from the a specific node provider ``` privateNodes: # Private nodes need to be enabled for this feature to work enabled: true autoNodes: - provider: my-node-provider dynamic: - name: my-dynamic-node-pool ``` #### Disruption[​](#disruption "Direct link to Disruption") Disruption configures how Karpenter should disrupt nodes and the config corresponds to the [Karpenter disruption config](https://karpenter.sh/docs/concepts/disruption/). By default, Karpenter will disrupt nodes if they are empty or underutilized after 30 seconds of inactivity. You can define more advanced ways of disruptions using schedules or budgets according to the [Karpenter config](https://karpenter.sh/docs/concepts/disruption/#nodepool-disruption-budgets). Each budget supports the `nodes`, `schedule`, and `duration` fields. Example of creating advanced disruption configuration ``` privateNodes: enabled: true autoNodes: - provider: my-provider dynamic: - name: my-dynamic-node-pool disruption: consolidationPolicy: WhenEmptyOrUnderutilized consolidateAfter: 10s budgets: - nodes: "20%" - nodes: "5" - nodes: "0" schedule: "@daily" duration: 10m ``` #### Limits[​](#limits "Direct link to Limits") Limits can be used as an upper limit for scheduling. These limits correspond to [Karpenter limits](https://karpenter.sh/docs/concepts/nodepools/#speclimits). Besides what Karpenter offers, it is also possible to specify `nodes` as a limit itself. The number of nodes specified in limits can be modified after the nodes are provisioned. If the new node limit is lower than the number of currently provisioned nodes, the Available Nodes count in the vCluster platform may appear as a negative value. Example of limiting 10 nodes or 100 cpus total ``` privateNodes: enabled: true autoNodes: - provider: my-provider dynamic: - name: my-dynamic-node-pool limits: cpu: "100" # either combined amount of cpus across all nodes in this node pool nodes: "10" # or maximum amount of nodes ``` Too small limits When limits are too low (for example, CPU is 1 and the smallest node type has CPU of 2) **nodes will not provision**. Make sure to use appropriate limits. When using `limits.nodes` consider that this might be the biggest nodes, so its usually a good idea to use a combination of `limits.cpu` and `limits.nodes`. ### Static node pools[​](#static-node-pools "Direct link to Static node pools") Static node pools are always created independent regardless of how many nodes are needed. They always require a quantity and a set of requirements to select which node types to deploy. When creating static node pools, they are also Karpenter [NodeClaims](https://karpenter.sh/docs/concepts/nodeclaims/), which allows Karpenter to take these static nodes into account when a dynamic node pool is also configured. Example of a static node pool of 2 nodes from a specific node provider ``` privateNodes: # Private nodes need to be enabled for this feature to work enabled: true autoNodes: - provider: my-provider static: - name: my-static-node-pool quantity: 2 ``` You can change the quantity of static node pools without restarting vCluster and vCluster will scale these nodes up or down based on the changing quantity. ### Taints and node labels[​](#taints-and-node-labels "Direct link to Taints and node labels") You can define taints and node labels for each node pool using the `taints` and `nodeLabels` fields which are useful to control scheduling on these nodes. Example of node pools with taints and node labels ``` privateNodes: enabled: true autoNodes: - provider: my-provider static: - name: my-static-pool quantity: 1 nodeLabels: my-label: my-value taints: - key: my-taint effect: NoSchedule dynamic: - name: my-static-pool nodeLabels: my-label: my-value taints: - key: my-taint effect: NoSchedule ``` ### Node claim properties[​](#node-claim-properties "Direct link to Node claim properties") A node provider can set a `properties` map at the provider level. Use it for details the provider needs to create nodes, such as region or subscription. Independently, each static or dynamic node pool can set its own `properties` map, passed through to the node claim the pool creates. `properties` doesn't select or filter anything, unlike `nodeTypeSelector`. It supplies extra operational details the node provider needs when creating a node from this pool, such as which region or subscription to use. If a provider and a node pool both set the same property key, the node pool's value takes precedence. Example of static and dynamic node pools setting properties, with provider-level defaults ``` privateNodes: enabled: true autoNodes: - provider: my-provider # Provider-level properties apply to every pool using this provider. properties: region: us-east-1 static: - name: my-static-pool quantity: 1 # Pool-level properties are set on this pool's node claims. # A key set here overrides the same key set at the provider level, # so this pool's nodes get region "us-west-2", not "us-east-1". properties: region: us-west-2 dynamic: - name: my-dynamic-pool properties: environment: production ``` ### Node profiles[​](#node-profiles "Direct link to Node profiles") Each static or dynamic node pool can reference a [node profile](https://vcluster.com/docs/platform/administer/node-profiles/overview.md) using the `profile` field. The platform continuously reconciles the profile's labels, annotations, and taints onto every node the pool provisions, so editing the profile updates those nodes without a rollout. Example of a node pool referencing a node profile ``` privateNodes: enabled: true autoNodes: - provider: my-provider static: - name: gpu-burst quantity: 2 profile: gpu-training dynamic: - name: gpu-dynamic profile: gpu-training ``` If a pool doesn't set `profile`, nodes it provisions fall back to the tenant cluster's `defaultProfile` or the project's default node profile, if either is configured. A node resolves to at most one profile. `profile` is independent of `taints` and `nodeLabels` set directly on the pool. Both apply, but a profile is reconciled continuously and shared across every provisioning path, while `taints` and `nodeLabels` on the pool are specific to that pool. ## Config reference[​](#config-reference "Direct link to Config reference") ## `autoNodes` object\[] [​](#autoNodes "Direct link to autoNodes")AutoNodes stores auto nodes configuration. ### `provider` required string [​](#autoNodes-provider "Direct link to autoNodes-provider")Provider is the node provider of the nodes in this pool. ### `properties` object [​](#autoNodes-properties "Direct link to autoNodes-properties")Properties are the node provider properties. This is a simple key value map and can contain things like region, subscription, etc. that is then used by the node provider to create the nodes and node environment. ### `static` object\[] [​](#autoNodes-static "Direct link to autoNodes-static")Static defines static node pools. Static node pools have a fixed size and are not scaled automatically. #### `name` required string [​](#autoNodes-static-name "Direct link to autoNodes-static-name")Name is the name of this static nodePool #### `properties` object [​](#autoNodes-static-properties "Direct link to autoNodes-static-properties")Properties are extra properties that will be set directly on the node claim. #### `nodeTypeSelector` object\[] [​](#autoNodes-static-nodeTypeSelector "Direct link to autoNodes-static-nodeTypeSelector")NodeTypeSelector filters the types of nodes that can be provisioned by this pool. All requirements must be met for a node type to be eligible. ##### `property` required string [​](#autoNodes-static-nodeTypeSelector-property "Direct link to autoNodes-static-nodeTypeSelector-property")Property is the property on the node type to select. ##### `operator` string [​](#autoNodes-static-nodeTypeSelector-operator "Direct link to autoNodes-static-nodeTypeSelector-operator")Operator is the comparison operator, such as "In", "NotIn", "Exists". If empty, defaults to "In". ##### `values` string\[] [​](#autoNodes-static-nodeTypeSelector-values "Direct link to autoNodes-static-nodeTypeSelector-values")Values is the list of values to use for comparison. This is mutually exclusive with value. ##### `value` string [​](#autoNodes-static-nodeTypeSelector-value "Direct link to autoNodes-static-nodeTypeSelector-value")Value is the value to use for comparison. This is mutually exclusive with values. #### `taints` object\[] [​](#autoNodes-static-taints "Direct link to autoNodes-static-taints")Taints are the taints to apply to the nodes in this pool. ##### `key` string [​](#autoNodes-static-taints-key "Direct link to autoNodes-static-taints-key")Required. The taint key to be applied to a node. ##### `value` string [​](#autoNodes-static-taints-value "Direct link to autoNodes-static-taints-value")The taint value corresponding to the taint key. ##### `effect` string [​](#autoNodes-static-taints-effect "Direct link to autoNodes-static-taints-effect")Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. #### `nodeLabels` object [​](#autoNodes-static-nodeLabels "Direct link to autoNodes-static-nodeLabels")NodeLabels are the labels to apply to the nodes in this pool. #### `terminationGracePeriod` string [​](#autoNodes-static-terminationGracePeriod "Direct link to autoNodes-static-terminationGracePeriod")TerminationGracePeriod is the maximum duration the controller will wait before forcefully deleting the pods on a node, measured from when deletion is first initiated.Warning: this feature takes precedence over a Pod's terminationGracePeriodSeconds value, and bypasses any blocked PDBs or the karpenter.sh/do-not-disrupt annotation.This field is intended to be used by cluster administrators to enforce that nodes can be cycled within a given time period. When set, drifted nodes will begin draining even if there are pods blocking eviction. Draining will respect PDBs and the do-not-disrupt annotation until the TGP is reached.Karpenter will preemptively delete pods so their terminationGracePeriodSeconds align with the node's terminationGracePeriod. If a pod would be terminated without being granted its full terminationGracePeriodSeconds prior to the node timeout, that pod will be deleted at T = node timeout - pod terminationGracePeriodSeconds.The feature can also be used to allow maximum time limits for long-running jobs which can delay node termination with preStop hooks. Defaults to 30s. Set to Never to wait indefinitely for pods to be drained. #### `quantity` required integer [​](#autoNodes-static-quantity "Direct link to autoNodes-static-quantity")Quantity is the number of desired nodes in this pool. #### `profile` string [​](#autoNodes-static-profile "Direct link to autoNodes-static-profile")Profile references a NodeProfile applied to nodes created by this pool. ### `dynamic` object\[] [​](#autoNodes-dynamic "Direct link to autoNodes-dynamic")Dynamic defines dynamic node pools. Dynamic node pools are scaled automatically based on the requirements within the cluster. Karpenter is used under the hood to handle the scheduling of the nodes. #### `name` required string [​](#autoNodes-dynamic-name "Direct link to autoNodes-dynamic-name")Name is the name of this NodePool #### `properties` object [​](#autoNodes-dynamic-properties "Direct link to autoNodes-dynamic-properties")Properties are extra properties that will be set directly on the node claim. #### `nodeTypeSelector` object\[] [​](#autoNodes-dynamic-nodeTypeSelector "Direct link to autoNodes-dynamic-nodeTypeSelector")NodeTypeSelector filters the types of nodes that can be provisioned by this pool. All requirements must be met for a node type to be eligible. ##### `property` required string [​](#autoNodes-dynamic-nodeTypeSelector-property "Direct link to autoNodes-dynamic-nodeTypeSelector-property")Property is the property on the node type to select. ##### `operator` string [​](#autoNodes-dynamic-nodeTypeSelector-operator "Direct link to autoNodes-dynamic-nodeTypeSelector-operator")Operator is the comparison operator, such as "In", "NotIn", "Exists". If empty, defaults to "In". ##### `values` string\[] [​](#autoNodes-dynamic-nodeTypeSelector-values "Direct link to autoNodes-dynamic-nodeTypeSelector-values")Values is the list of values to use for comparison. This is mutually exclusive with value. ##### `value` string [​](#autoNodes-dynamic-nodeTypeSelector-value "Direct link to autoNodes-dynamic-nodeTypeSelector-value")Value is the value to use for comparison. This is mutually exclusive with values. #### `taints` object\[] [​](#autoNodes-dynamic-taints "Direct link to autoNodes-dynamic-taints")Taints are the taints to apply to the nodes in this pool. ##### `key` string [​](#autoNodes-dynamic-taints-key "Direct link to autoNodes-dynamic-taints-key")Required. The taint key to be applied to a node. ##### `value` string [​](#autoNodes-dynamic-taints-value "Direct link to autoNodes-dynamic-taints-value")The taint value corresponding to the taint key. ##### `effect` string [​](#autoNodes-dynamic-taints-effect "Direct link to autoNodes-dynamic-taints-effect")Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. #### `nodeLabels` object [​](#autoNodes-dynamic-nodeLabels "Direct link to autoNodes-dynamic-nodeLabels")NodeLabels are the labels to apply to the nodes in this pool. #### `limits` object [​](#autoNodes-dynamic-limits "Direct link to autoNodes-dynamic-limits")Limits specify the maximum resources that can be provisioned by this node pool, mapping to the 'limits' field in Karpenter's NodePool API. #### `disruption` object [​](#autoNodes-dynamic-disruption "Direct link to autoNodes-dynamic-disruption")Disruption contains the parameters that relate to Karpenter's disruption logic ##### `consolidateAfter` string [​](#autoNodes-dynamic-disruption-consolidateAfter "Direct link to autoNodes-dynamic-disruption-consolidateAfter")ConsolidateAfter is the duration the controller will wait before attempting to terminate nodes that are underutilized. Refer to ConsolidationPolicy for how underutilization is considered. ##### `consolidationPolicy` string [​](#autoNodes-dynamic-disruption-consolidationPolicy "Direct link to autoNodes-dynamic-disruption-consolidationPolicy")ConsolidationPolicy describes which nodes Karpenter can disrupt through its consolidation algorithm. This policy defaults to "WhenEmptyOrUnderutilized" if not specified ##### `budgets` object\[] [​](#autoNodes-dynamic-disruption-budgets "Direct link to autoNodes-dynamic-disruption-budgets")Budgets is a list of Budgets. If there are multiple active budgets, Karpenter uses the most restrictive value. If left undefined, this will default to one budget with a value to 10%. ##### `nodes` string [​](#autoNodes-dynamic-disruption-budgets-nodes "Direct link to autoNodes-dynamic-disruption-budgets-nodes")Nodes dictates the maximum number of NodeClaims owned by this NodePool that can be terminating at once. This is calculated by counting nodes that have a deletion timestamp set, or are actively being deleted by Karpenter. This field is required when specifying a budget. ##### `schedule` string [​](#autoNodes-dynamic-disruption-budgets-schedule "Direct link to autoNodes-dynamic-disruption-budgets-schedule")Schedule specifies when a budget begins being active, following the upstream cronjob syntax. If omitted, the budget is always active. Timezones are not supported. ##### `duration` string [​](#autoNodes-dynamic-disruption-budgets-duration "Direct link to autoNodes-dynamic-disruption-budgets-duration")Duration determines how long a Budget is active since each Schedule hit. Only minutes and hours are accepted, as cron does not work in seconds. If omitted, the budget is always active. This is required if Schedule is set. #### `terminationGracePeriod` string [​](#autoNodes-dynamic-terminationGracePeriod "Direct link to autoNodes-dynamic-terminationGracePeriod")TerminationGracePeriod is the maximum duration the controller will wait before forcefully deleting the pods on a node, measured from when deletion is first initiated.Warning: this feature takes precedence over a Pod's terminationGracePeriodSeconds value, and bypasses any blocked PDBs or the karpenter.sh/do-not-disrupt annotation.This field is intended to be used by cluster administrators to enforce that nodes can be cycled within a given time period. When set, drifted nodes will begin draining even if there are pods blocking eviction. Draining will respect PDBs and the do-not-disrupt annotation until the TGP is reached.Karpenter will preemptively delete pods so their terminationGracePeriodSeconds align with the node's terminationGracePeriod. If a pod would be terminated without being granted its full terminationGracePeriodSeconds prior to the node timeout, that pod will be deleted at T = node timeout - pod terminationGracePeriodSeconds.The feature can also be used to allow maximum time limits for long-running jobs which can delay node termination with preStop hooks. Defaults to 30s. Set to Never to wait indefinitely for pods to be drained. #### `expireAfter` string [​](#autoNodes-dynamic-expireAfter "Direct link to autoNodes-dynamic-expireAfter")The amount of time a Node can live on the cluster before being removed #### `weight` integer [​](#autoNodes-dynamic-weight "Direct link to autoNodes-dynamic-weight")Weight is the weight of this node pool. #### `profile` string [​](#autoNodes-dynamic-profile "Direct link to autoNodes-dynamic-profile")Profile references a NodeProfile applied to nodes created by this pool. --- # vCluster VPN Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** vCluster VPN allows you to connect the private worker nodes to the vCluster control plane through vCluster Platform. This is useful in scenarios where the nodes cannot reach the control plane directly, for example if you cannot use `LoadBalancer` or `NodePort` type services, but nodes can reach the vCluster Platform URL. vCluster VPN builds a network across the control plane and the worker nodes powered by the same technology used by [tailscale](https://tailscale.com/). You don't need a tailscale account for VPN to work as vCluster reuses tailscale as a library under the hood and will never reach out to any tailscale services directly. The only requirement for the VPN to work is that the vCluster control plane and any worker node(s) can reach the platform URL (e.g. `https://platform.my-domain.com`). vCluster VPN figures out a direct connection between worker nodes and the control plane through [wireguard](https://tailscale.com/blog/how-tailscale-works), which can traverse even complex NAT setups and make direct connections. If wireguard fails to create a direct connection, traffic is routed through the vCluster Platform.
Enabling VPN ``` privateNodes: enabled: true vpn: enabled: true # This enables node to control plane vpn nodeToNode: enabled: true # This enables node to node vpn ``` ## When to use vCluster VPN[​](#when-to-use-vcluster-vpn "Direct link to When to use vCluster VPN") ### VPN is useful in the following scenarios:[​](#vpn-is-useful-in-the-following-scenarios "Direct link to VPN is useful in the following scenarios:") * Worker nodes cannot reach the control plane directly, because the control plane service cannot be exposed with a `NodePort` or a `LoadBalancer`. * Worker nodes cannot reach each other directly, because the nodes are in different VPCs or spread across cloud and bare-metal. * Traffic between nodes is not encrypted through the CNI and might be sniffed by other parties. ### When to *not* use VPN:[​](#when-to-not-use-vpn "Direct link to when-to-not-use-vpn") * Only the control plane cannot reach the worker nodes directly. This is already solved by using [Konnectivity](https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/) and does not require vCluster VPN or any extra configuration * Nodes cannot reach the vCluster platform URL, in this case the VPN cannot be used * You need the maximum possible network throughput. While tailscale will only add minimal overhead it still cannot be tolerated in some cases * You cannot change the networking setup on the node. vCluster VPN will create a new network interface (tailscale0) and needs permissions to create it * When the goal is to isolate node networking. While vCluster VPN will encrypt all traffic, it cannot prevent nodes reaching each other or block traffic between them. If that is the goal, use a proper VLAN or underlay setup for this. Only use vCluster VPN if you have to If there is a direct way for node-to-node or node-to-control-plane communication it's usually preferred over using vCluster VPN. ## How does it work?[​](#how-does-it-work "Direct link to How does it work?") When enabled, the join script will setup a systemd service called `vcluster-vpn` that connects the node to the platform. The service runs a modified version of [tailscaled](https://tailscale.com/kb/1278/tailscaled). As soon as the node has joined the vpn it will join the node itself into the vCluster. On each vCluster control plane an integrated tailnet service is running to connect the control plane to the vpn as well. Within the Kubernetes cluster the Kubernetes endpoint will then be the tailscale ip of the control plane. If node to node communication is wanted, the default cni flannel is then configured to use the network interface `tailscale0` on the node as well as the internal node ip will be exchanged with the tailscale one. ## Node to control plane VPN[​](#node-to-control-plane-vpn "Direct link to Node to control plane VPN") Node to control plane VPN allows your nodes to communicate to the control plane over the tailscale vpn. This is useful if you cannot expose the vCluster control plane via a `LoadBalancer` service or a public ip. To configure node to control plane tunneling, you need to set the following in your `vcluster.yaml`: ``` privateNodes: enabled: true vpn: enabled: true # This enables node to control plane vpn ``` After you have started the vCluster, you should see that the endpoints of the `kubernetes` service is a `100.64.x.x` tailscale ip address: ``` $ kubectl get endpoints NAME ENDPOINTS AGE kubernetes 100.64.0.4:443 5d19h ``` ## Node to node VPN[​](#node-to-node-vpn "Direct link to Node to node VPN") Node to node VPN allows your nodes to communicate with each other over the tailscale vpn. This is useful when the nodes have no direct connection to each other, for example if you have nodes in different clouds or you use a mix of bare-metal and cloud nodes. To configure node to node VPN, you just need to set the following in your `vcluster.yaml`: ``` privateNodes: enabled: true vpn: enabled: true # This enables node to control plane vpn nodeToNode: enabled: true # This enables node to node vpn ``` Using node to node VPN requires node to control plane VPN, so you cannot opt-out of that. After you have started the vCluster and joined a node, you should see that the node ip address is a `100.64.x.x` tailscale ip address: ``` $ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME my-node Ready 5d19h v1.36.0 100.64.0.5 Ubuntu 6.14.10 containerd://2.0.5 ``` ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") If you experience any problems with the connection, you can check the vCluster VPN logs directly or via the [tailscale CLI](https://tailscale.com/kb/1080/cli) ### Make sure cni is configured to use interface tailscale0[​](#make-sure-cni-is-configured-to-use-interface-tailscale0 "Direct link to Make sure cni is configured to use interface tailscale0") If you want to use your own custom cni (not flannel) and node-to-node communication over the vpn, make sure to configure it to use interface `tailscale0` instead of the default one. ### Check vCluster VPN logs[​](#check-vcluster-vpn-logs "Direct link to Check vCluster VPN logs") vCluster VPN is installed as a systemd service on the node that should join, you can check the logs via the following command on the node: ``` journalctl -u vcluster-vpn --no-pager ``` ### Join hangs on a previously used node[​](#join-hangs-on-a-previously-used-node "Direct link to Join hangs on a previously used node") If the node was joined to a cluster that was later deleted, `vcluster-vpn` keeps that cluster's VPN state. The join script rewrites `/etc/vcluster-vpn/config.json` but doesn't restart the service, so the join waits for a control plane it can't reach. The logs show `platform access key not found. Please login again`. Restart the service on the node: ``` systemctl restart vcluster-vpn ``` The join continues once the service reconnects. ### Check with tailscale CLI[​](#check-with-tailscale-cli "Direct link to Check with tailscale CLI") Do not install tailscaled daemon Make sure to only install the CLI and NOT tailscaled as vCluster VPN is replacing that and if you install tailscale the regular way it will override and conflict with vCluster VPN. If the logs look good, you can also install tailscale CLI on the node via: ``` # Arch either amd64 or arm64 ARCH=amd64 # Download and install tailscale curl -sSLf https://pkgs.tailscale.com/stable/tailscale_1.86.0_$ARCH.tgz > tailscale.tgz && tar -zxf tailscale.tgz && mv tailscale_1.86.0_$ARCH/tailscale /usr/local/bin ``` Then you can use tailscale CLI commands to debug: ``` # Try to ping control plane or other nodes (check 'kubectl get endpoints' for the control plane ip or 'kubectl get nodes -o wide' for node ips) tailscale ping 100.64.X.X # Check if platform url shows up tailscale debug derp-map # Check if peers correctly show up tailscale debug netmap ``` ### Changing settings after node join[​](#changing-settings-after-node-join "Direct link to Changing settings after node join") When changing the way the virtual cluster control plane is exposed, e.g. switching from LoadBalancer to VPN, you will need to re-join the nodes to pick up the new settings. ### Configure firewall to enable Tailscale to establish peer-to-peer connections[​](#configure-firewall-to-enable-tailscale-to-establish-peer-to-peer-connections "Direct link to Configure firewall to enable Tailscale to establish peer-to-peer connections") While it is possible to utilize the vCluster Platform as a DERP relay server, there are cases where the additional latency is not acceptable. You may check whether your nodes are using a relay using the `tailscale status` command: ``` 1 2 3 4 5 100.1.2.3 device-a alice@ linux active; direct , tx 1116 rx 1124 100.4.5.6 device-b bob@ macOS active; relay , tx 1351 rx 4262 100.7.8.9 device-c charlie@ windows idle; tx 1214 rx 50 100.0.1.2 device-d diane@ iOS — ``` If you notice that two devices are connected over a relay server, you can try opening firewall ports to enable them to connect peer-to-peer (direct): * UDP connections from `:41641` to `*:*`. * UDP connections to `*:3478` * TCP connections to `*:80` * TCP connections to `*:443` In some cases it may be enough to enable connections to UDP port 41641. ## Config reference[​](#config-reference "Direct link to Config reference") ## `vpn` object [​](#vpn "Direct link to vpn")VPN holds configuration for the private nodes vpn. This can be used to connect the private nodes to the control plane or connect the private nodes to each other if they are not running in the same network. Platform connection is required for the vpn to work. ### `enabled` boolean false [​](#vpn-enabled "Direct link to vpn-enabled")Enabled defines if the private nodes vpn should be enabled. ### `nodeToNode` object [​](#vpn-nodeToNode "Direct link to vpn-nodeToNode")NodeToNode holds configuration for the node to node vpn. This can be used to connect the private nodes to each other if they are not running in the same network. #### `enabled` boolean false [​](#vpn-nodeToNode-enabled "Direct link to vpn-nodeToNode-enabled")Enabled defines if the node to node vpn should be enabled. --- # RBAC Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** vCluster automatically generates the RBAC roles and bindings it needs based on the features you enable and the resources you choose to [sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync). note Some configuration options modify or add to the default RBAC rules. If you're using any of the following, vCluster includes the necessary permissions. When enabled, the following configuration fields modify the default behavior: * [`controlPlane.distro.k8s.scheduler`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/virtual-scheduler.md): Adds required read permissions for the virtual scheduler. * [`networking.replicateServices.fromHost`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking/replicate-services.md): Adds required permissions to manage endpoints and services. * [`integrations.metrics.proxy.nodes`](https://vcluster.com/docs/vcluster/integrations/metrics-server.md): Adds required read permissions for node resources. * [`plugins`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/plugins.md): Adds `roles` and `clusterRoles` defined by each plugin. * [`sync.toHost.namespaces`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/namespaces.md): Adds required permissions for vCluster to manage namespaces and service accounts. * [`sync.toHost.volumeSnapshots`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/volume-snapshots.md): Adds required permissions for volume snapshot syncing. `rbac.enableVolumeSnapshotRules` defaults to `auto`, which enables these rules automatically. ## Disable automatic RBAC generation[​](#disable-automatic-rbac-generation "Direct link to Disable automatic RBAC generation") In environments with strict access controls, users may not have permission to create Kubernetes RBAC resources such as `Role`, `ClusterRole`, or `ServiceAccount`. In these cases, you can configure vCluster to use a service account that has already been created by an administrator with the necessary permissions. To do this, disable vCluster’s automatic RBAC generation and reference the managed service account in `vcluster.yaml`: Use a managed service account and disable default RBAC ``` controlPlane: advanced: serviceAccount: name: custom-vc-sa rbac: role: enabled: false clusterRole: enabled: false ``` Alternatively, you can disable `ClusterRole` creation—this is often necessary when users don’t have permission to create `ClusterRole` resources: Disable only ClusterRole creation ``` rbac: clusterRole: enabled: false ``` ## Add custom RBAC rules[​](#add-custom-rbac-rules "Direct link to Add custom RBAC rules") Use `extraRules` to configure additional RBAC rules when the automatically generated or [plugin](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/plugins.md) rules are not sufficient. Use this option sparingly—most configurations should not require it. The following example shows how to apply it: Add extra RBAC rules ``` rbac: role: enabled: true extraRules: - apiGroups: [""] resources: ["pods/status", "pods/ephemeralcontainers"] verbs: ["patch", "update"] ``` ## Overwrite rules[​](#overwrite-rules "Direct link to Overwrite rules") For more control over permissions, use `overwriteRules` to disable the default rules generated by the vCluster Helm chart. When `overwriteRules` is enabled, you must manually define all required rules for the vCluster and its plugins to function. This approach can simplify permission management by allowing the Helm chart to still create the `Role`, `RoleBinding`, and `ServiceAccount`, while you retain full control over the rules. The following example disables cluster-wide rules and specifies namespace-level permissions: Overwrite all default RBAC rules ``` rbac: role: enabled: true overwriteRules: - apiGroups: [""] resources: ["pods/status", "pods/ephemeralcontainers"] verbs: ["patch", "update"] clusterRole: enabled: false ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `rbac` object [​](#rbac "Direct link to rbac")RBAC options for the virtual cluster. ### `role` object [​](#rbac-role "Direct link to rbac-role")Role holds virtual cluster role configuration #### `enabled` boolean true [​](#rbac-role-enabled "Direct link to rbac-role-enabled")Enabled defines if the role should be enabled or disabled. #### `extraRules` object\[] \[] [​](#rbac-role-extraRules "Direct link to rbac-role-extraRules")ExtraRules will add rules to the role. #### `overwriteRules` object\[] \[] [​](#rbac-role-overwriteRules "Direct link to rbac-role-overwriteRules")OverwriteRules will overwrite the role rules completely. ### `clusterRole` object [​](#rbac-clusterRole "Direct link to rbac-clusterRole")ClusterRole holds virtual cluster cluster role configuration #### `enabled` string|boolean auto [​](#rbac-clusterRole-enabled "Direct link to rbac-clusterRole-enabled")Enabled defines if the cluster role should be enabled or disabled. If auto, vCluster automatically determines whether the virtual cluster requires a cluster role. #### `extraRules` object\[] \[] [​](#rbac-clusterRole-extraRules "Direct link to rbac-clusterRole-extraRules")ExtraRules will add rules to the cluster role. #### `overwriteRules` object\[] \[] [​](#rbac-clusterRole-overwriteRules "Direct link to rbac-clusterRole-overwriteRules")OverwriteRules will overwrite the cluster role rules completely. ### `enableVolumeSnapshotRules` object [​](#rbac-enableVolumeSnapshotRules "Direct link to rbac-enableVolumeSnapshotRules")EnableVolumeSnapshotRules enables all required volume snapshot rules in the Role and ClusterRole. #### `enabled` string|boolean auto [​](#rbac-enableVolumeSnapshotRules-enabled "Direct link to rbac-enableVolumeSnapshotRules-enabled")Enabled defines if this option should be enabled. --- # Configure auto sleep | | | Enterprise | | | | -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Auto Sleep](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. Auto sleep reduces the compute resources that idle tenant clusters consume. It can respond to inactivity, a cron schedule, or supported HTTP traffic. Pre-production workloads Auto sleep is intended for pre-production workloads. Sleeping interrupts workloads and deletes standalone Pods that no controller manages. ## How auto sleep is managed[​](#how-auto-sleep-is-managed "Direct link to How auto sleep is managed") The component that manages auto sleep determines which resources can stop: | Deployment | Sleep behavior | | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | Tenant cluster without a platform agent | The tenant cluster scales down its workloads. Its control plane stays running to detect activity and restore workloads. | | Tenant cluster connected to vCluster Platform | The platform agent takes over sleep management and can scale down both workloads and the tenant cluster control plane. | When an existing tenant cluster connects to a compatible platform agent, the agent reads its `sleep` configuration and takes over automatically. Don't configure a second sleep mechanism. This handoff requires vCluster 0.24.0 or later and vCluster Platform 4.3.0 or later. For vCluster 0.33.0 or later, add the `sleepmode.loft.sh/scope: "workloads-only"` annotation to the platform VirtualClusterInstance. Use it when the tenant cluster should keep managing its own workload sleep and its control plane must stay running. With this scope, the platform agent doesn't take over auto sleep. Set the scope before the platform agent takes over sleep management. If the tenant cluster is already connected, set the scope and then remove the `vcluster.loft.sh/agent-installed` annotation from its `vc-config-` Secret. The Secret is in the namespace where the tenant cluster control plane runs on the control plane cluster. The platform agent leaves the annotation in place when the scope changes, and native auto sleep remains disabled until you remove it. See [Sleep tenant clusters](https://vcluster.com/docs/vcluster/key-features/sleep.md) for the lifecycle and [Pause and resume a tenant cluster](https://vcluster.com/docs/vcluster/manage/sleep-wakeup.md) for manual operations. ## Configure inactivity-based auto sleep[​](#configure-inactivity-based-auto-sleep "Direct link to Configure inactivity-based auto sleep") * vcluster.yaml * Platform UI Set `sleep.auto.afterInactivity` to the amount of idle time allowed before workloads sleep: vcluster.yaml ``` sleep: auto: afterInactivity: 1h ``` The value uses Go duration syntax. Use `s`, `m`, or `h`, such as `30s`, `15m`, or `24h`. The syntax doesn't support `d`; use `24h` instead of `1d`. 1. Select the project that contains the tenant cluster. 2. Click **Tenant Clusters** in the sidebar. 3. Click **Edit** for the tenant cluster. 4. Expand **Auto-Sleep and Auto-Delete**. 5. Set **Sleep after inactivity**. 6. Click **Save**. For a tenant cluster created from a template, edit the template instead of the instance. See [Create a tenant cluster template](https://vcluster.com/docs/platform/administer/templates/create-templates.md). ## Configure scheduled sleep and wakeup[​](#configure-scheduled-sleep-and-wakeup "Direct link to Configure scheduled sleep and wakeup") Set five-field cron expressions for `schedule` and `wakeup.schedule`. Set an [IANA time zone](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) with `timezone`; the default is UTC. Sleep Friday evening and wake Monday morning ``` sleep: auto: timezone: America/Denver schedule: 30 17 * * 5 wakeup: schedule: 0 7 * * 1 ``` In vCluster Platform, select the project and click **Tenant Clusters**. Click **Edit** for the tenant cluster, expand **Auto-Sleep and Auto-Delete**, then set **Sleep Schedule**, **Wake up Schedule**, and **Schedule timezone**. ## Exclude workloads from auto sleep[​](#exclude-workloads-from-auto-sleep "Direct link to Exclude workloads from auto sleep") Use a label selector when you know which workloads must stay running. The following configuration excludes workloads with the label `sleep: no-thanks`: Exclude workloads by label ``` sleep: auto: afterInactivity: 3h exclude: selector: labels: sleep: no-thanks ``` For platform-managed sleep, you can instead add the `sleepmode.loft.sh/exclude: "true"` annotation to a Deployment, StatefulSet, ReplicaSet, or Pod. Sleep creates a quota that prevents new Pods, so Kubernetes can't replace an excluded Pod that stops while the resource sleeps: Exclude a Deployment ``` apiVersion: apps/v1 kind: Deployment metadata: name: connectivity-agent annotations: sleepmode.loft.sh/exclude: "true" spec: replicas: 1 selector: matchLabels: app: connectivity-agent template: metadata: labels: app: connectivity-agent spec: containers: - name: agent image: example.com/connectivity-agent:1.0 ``` ## Activity detection and wakeup[​](#activity-detection-and-wakeup "Direct link to Activity detection and wakeup") This section applies to platform-managed namespaces and tenant clusters alike, except where noted. Auto sleep resets the inactivity timer when it detects relevant activity. If the resource is already sleeping, supported activity wakes it. For platform-managed sleep started by a sleep schedule, activity doesn't wake the resource. It remains asleep until its wakeup schedule or a manual wakeup. Standalone auto sleep, which only applies to a tenant cluster without a platform agent, can wake from activity after scheduled sleep. ### Kubernetes API activity[​](#kubernetes-api-activity "Direct link to Kubernetes API activity") For a platform-managed namespace or tenant cluster, Kubernetes API requests routed through vCluster Platform count as activity. This includes `kubectl` requests when the kubeconfig routes through the platform proxy. Direct requests to a tenant cluster API endpoint that bypasses the platform proxy aren't visible to platform activity detection. A standalone tenant cluster detects API activity through its running control plane. ### Detect web traffic[​](#detect-web-traffic "Direct link to Detect web traffic") For both namespaces and tenant clusters, vCluster Platform can detect traffic to the following resources: * ingress-nginx Ingress resources * Istio Gateway and VirtualService resources * Gateway API HTTPRoute resources when the Gateway controller supports request mirroring For ingress-nginx, the platform adds `mirror-target` and `mirror-request-body` annotations unless the Ingress already has either annotation. The platform preserves existing user-managed mirror annotations, but traffic to that Ingress doesn't refresh the last-activity timestamp through this integration. ingress-nginx is deprecated; for new endpoint deployments, prefer [Gateway API](https://vcluster.com/docs/vcluster/key-features/gateway-api.md). For Istio, enable the [Istio integration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/istio.md). The platform uses `istio-proxy` metrics to determine activity. #### Gateway API HTTPRoute activity[​](#gateway-api-httproute-activity-detection "Direct link to Gateway API HTTPRoute activity") Gateway-backed activity detection requires the controller to support the `HTTPRoute` request-mirror filter. An HTTPRoute can serve traffic without this feature, but its awake traffic doesn't refresh the last-activity timestamp. The namespace or tenant cluster can therefore enter sleep while the route is receiving traffic. Don't rely on activity-based auto sleep through that route unless its controller supports request mirroring. If a compatible controller doesn't advertise request mirroring in `GatewayClass` status, add the `sleepmode.loft.sh/request-mirror-controller-allowlist` annotation to the platform `Cluster` resource. Use the `GatewayClass.spec.controllerName` value. Separate multiple controller names with commas. Restart the agent after installing Gateway API CRDs If you install Gateway API CRDs after the connected-cluster agent starts, restart the agent Deployment so it discovers the HTTPRoute controller: Restart the vCluster Platform agent ``` kubectl rollout restart deployment/loft --namespace vcluster-platform ``` The agent Deployment name is always `loft`. Its namespace defaults to `vcluster-platform` for newly connected clusters; clusters connected before this default existed may still use `loft`. Check the platform `Cluster` resource's `spec.managementNamespace` if you're unsure. ### Ingress wakeup[​](#ingress-wakeup "Direct link to Ingress wakeup") When platform activity detection supports the route, opening its hostname displays a wakeup page while the namespace or tenant cluster wakes. The page reloads and reaches the application after wakeup completes. ### Last activity update interval[​](#last-activity-update-interval "Direct link to Last activity update interval") The platform limits how often mirrored Ingress and HTTPRoute traffic updates the last-activity timestamp. The default interval is five minutes. Requests received during the interval don't write another timestamp. Istio activity uses its metrics polling interval instead. Set `SLEEP_MODE_LAST_ACTIVITY_UPDATE_INTERVAL` on the connected-cluster agent to change the interval. See [Agent values](https://vcluster.com/docs/platform/administer/clusters/advanced/agent-config.md#agent-values) for how to set agent environment variables platform-wide or per cluster. ## Filter platform activity[​](#filter-platform-activity "Direct link to Filter platform activity") Use `sleepmode.loft.sh` annotations when automated clients or selected requests shouldn't keep a platform-managed namespace or tenant cluster awake. For a namespace, add them to the Namespace. For a template-backed tenant cluster, add the supported annotations under `spec.template.metadata.annotations` in its VirtualClusterTemplate. For an inline VirtualClusterInstance, use the same path on the instance. `ignore-groups`, `ignore-resources`, `ignore-verbs`, `ignore-resource-verbs`, and `ignore-resource-names` apply to namespace sleep mode only. They have no effect on a tenant cluster. Common filters include: | Annotation | Effect | | --------------------------------------------- | -------------------------------------------------------------------------- | | `sleepmode.loft.sh/ignore-all` | Ignore all activity. | | `sleepmode.loft.sh/ignore-ingresses` | Ignore supported Ingress, HTTPRoute, and Istio traffic. | | `sleepmode.loft.sh/ignore-groups` | Ignore requests from selected user groups. Namespace only. | | `sleepmode.loft.sh/ignore-resources` | Ignore requests to selected Kubernetes resources. Namespace only. | | `sleepmode.loft.sh/ignore-verbs` | Ignore selected Kubernetes API verbs. Namespace only. | | `sleepmode.loft.sh/ignore-user-agents` | Ignore matching user agents; a trailing wildcard is supported. | | `sleepmode.loft.sh/ignore-active-connections` | Don't treat open streaming connections as activity. | | `sleepmode.loft.sh/disable-ingress-wakeup` | Don't wake the resource for supported Ingress, HTTPRoute, or Istio access. | You can also add the `X-Sleep-Mode-Ignore` header to an individual request. See the [platform annotation reference](https://vcluster.com/docs/platform/reference/platform-annotations.md#sleepmode-annotations) for where to set each filter and all value formats. ## Handle long-running connections[​](#handle-long-running-connections "Direct link to Handle long-running connections") Streaming requests such as `kubectl exec` and `kubectl port-forward` can keep a namespace or tenant cluster active. Choose one of these controls when idle streams should no longer prevent sleep: 1. Add `loft.sh/streaming-connection-idle-timeout: "3600"` to the platform `Cluster` resource. The platform closes new streaming connections after the configured number of idle seconds. Existing connections aren't affected. 2. Add `sleepmode.loft.sh/ignore-active-connections: "true"` to the Namespace or tenant cluster template metadata. Open streams then don't count as activity and can be interrupted when the resource sleeps. 3. Configure the kubelet `--streaming-connection-idle-timeout` flag on the control plane cluster independently of vCluster Platform. ## Configure auto-delete[​](#configure-auto-delete "Direct link to Configure auto-delete") Auto-delete permanently deletes a platform-managed tenant cluster after an inactivity period. It is separate from sleep and can run whether the tenant cluster is awake or sleeping. See [Configure automatic deletion](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/deletion.md) for the `vcluster.yaml` and vCluster Platform procedures. ## Examples[​](#examples "Direct link to Examples") Standalone deployment example ### Configure auto sleep for a deployment resource[​](#configure-auto-sleep-for-a-deployment-resource "Direct link to Configure auto sleep for a deployment resource") 1. Create the `kind` cluster. create kind cluster ``` kind create cluster --name sleep-mode-demo ``` 2. Deploy a tenant cluster. Use the following `vcluster.yaml` to create a tenant cluster on your control plane cluster. Save this file as `vcluster.yaml` vCluster config for auto sleep ``` pro: true sleep: auto: afterInactivity: 30s exclude: selector: labels: sleep: no-thanks ``` And run the following command: Create vCluster with autoSleep config ``` vcluster create my-vcluster -f vcluster.yaml ``` Workloads with the label `sleep: no-thanks` don't enter auto sleep after 30 seconds. 3. Create demo deployments in your tenant cluster. Use the following deployment YAML to create two deployments. Example deployments ``` apiVersion: apps/v1 kind: Deployment metadata: name: sleepy-deployment labels: app: sleepy-dep spec: replicas: 2 selector: matchLabels: app: demo-dep-1 template: metadata: labels: app: demo-dep-1 spec: containers: - command: - /agnhost - serve-hostname - --http=true - --port=8080 image: registry.k8s.io/e2e-test-images/agnhost:2.39 name: sleepy-demo --- apiVersion: apps/v1 kind: Deployment metadata: name: no-sleep-deployment labels: sleep: no-thanks spec: replicas: 2 selector: matchLabels: app: demo-dep-2 template: metadata: labels: app: demo-dep-2 spec: containers: - command: - /agnhost - serve-hostname - --http=true - --port=8080 image: registry.k8s.io/e2e-test-images/agnhost:2.39 name: not-sleepy-demo ``` The first deployment does not have any special configurations for auto sleep. You can replace it with another deployment if needed. The second deployment includes a special label on the Deployment, preventing it from scaling down after 30 seconds. You can verify this by waiting `30 seconds` and then getting information about the Deployments. For example 4. Verify Deployments sleep status. deployment sleep check ``` > sleep 30; kubectl get deployments NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE default no-sleep-deployment 2/2 2 2 1m default sleepy-deployment 0/2 0 0 1m ``` The `sleepy-deployment` reports `0/2` replicas after 30 seconds. Running `kubectl` counts as cluster activity, which is why it reports `0/2` instead of `0/0`. The `kubectl` command triggers `vCluster` to update the replica count back to the original value of 2, but the replicas haven't become ready by the time `kubectl get ...` returns. #### Next steps[​](#next-steps "Direct link to Next steps") Experiment with the auto sleep feature by trying the following: * Add the `sleep: no-thanks` label to the first deployment and verify neither sleeps. * Remove the `sleep: no-thanks` label from both the deployments and verify that both go to sleep. Ingress example ### Configure auto sleep for an ingress controller[​](#configure-auto-sleep-for-an-ingress-controller "Direct link to Configure auto sleep for an ingress controller") 1. Create the `kind` cluster. Create a kind cluster ``` kind create cluster --name ingress-demo --config - < -f vcluster.yaml --upgrade ``` 6. Enable local DNS resolution for the tenant cluster. Add `127.0.0.1 smdemo.local` to your `/etc/hosts` file to match the host configured in the Gateway configuration. 7. Add the Gateway you created and the `smdemo.local` host to the VirtualService from the \[Istio integration example]\(../vcluster- yaml/integrations/istio). The updated VirtualService should look like the following: Example resources ``` apiVersion: networking.istio.io/v1 kind: VirtualService metadata: name: nginx-service namespace: test spec: gateways: - istio-sm-gateway hosts: - smdemo.local http: - match: - uri: prefix: /v2 name: nginx-v2 rewrite: uri: / route: - destination: host: nginx-service.test.svc.cluster.local subset: v2 - name: nginx-v1 route: - destination: host: nginx-service.test.svc.cluster.local subset: v1 ``` 8. Use `curl` to verify that the Gateway is working correctly. Test the Gateway endpoint within the 30-second activity window by running either `curl --silent smdemo.local/v1` or `curl --silent smdemo.local/v2`. 9. Allow the tenant cluster to go to sleep. Wait 30 seconds for the cluster to sleep. Then run the `curl` command again. To repeatedly test the endpoint, run `watch -d curl --silent smdemo.local/v2`. While the cluster is asleep or waking up, a `503` error is displayed. ## Configuration reference[​](#configuration-reference "Direct link to Configuration reference") ## `sleep` object [​](#sleep "Direct link to sleep")Sleep holds configuration for automatically putting the virtual cluster to sleep. ### `auto` object [​](#sleep-auto "Direct link to sleep-auto")Auto holds automatic sleep configuration #### `afterInactivity` string [​](#sleep-auto-afterInactivity "Direct link to sleep-auto-afterInactivity")AfterInactivity represents how long a vCluster can be idle before workloads are automatically put to sleep #### `schedule` string [​](#sleep-auto-schedule "Direct link to sleep-auto-schedule")Schedule represents a cron schedule for when to sleep workloads #### `exclude` object [​](#sleep-auto-exclude "Direct link to sleep-auto-exclude")Exclude holds configuration for labels that, if present, will prevent a workload from going to sleep ##### `selector` object [​](#sleep-auto-exclude-selector "Direct link to sleep-auto-exclude-selector") ##### `labels` object [​](#sleep-auto-exclude-selector-labels "Direct link to sleep-auto-exclude-selector-labels")Labels defines what labels should be looked for #### `wakeup` object [​](#sleep-auto-wakeup "Direct link to sleep-auto-wakeup")Wakeup holds configuration for waking the vCluster on a schedule ##### `schedule` string [​](#sleep-auto-wakeup-schedule "Direct link to sleep-auto-wakeup-schedule") #### `timezone` string [​](#sleep-auto-timezone "Direct link to sleep-auto-timezone")Timezone specifies time zone used for scheduled sleep operations. Defaults to UTC. Accepts the same format as time.LoadLocation() in Go (). The value should be a location name corresponding to a file in the IANA Time Zone database, such as "America/New\_York". --- | | | Enterprise | | | | -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Auto Snapshots](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/snapshots.md#enable-auto-snapshots) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. # Snapshots vCluster Platform allows you to configure taking snapshots of the vCluster at specific intervals. Manual snapshots with the vCluster CLI For on-demand snapshot creation and restore operations, see the [vCluster Snapshot and Restore](https://vcluster.com/docs/vcluster/manage/backup-restore/backup.md) documentation. This allows administrators to capture and store the vCluster state in scheduled intervals to help protect against infrastructure failures, data corruption, and configuration errors. By maintaining consistent recovery points, administrators can quickly restore the vCluster to a known good state without relying on manual backup processes. If you use private nodes, see [Use snapshots with private nodes](https://vcluster.com/docs/vcluster/manage/backup-restore/restore.md#use-snapshots-with-private-nodes) for additional restore steps. Deployed etcd isn't supported Snapshots aren't supported with [deployed etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md) (`controlPlane.backingStore.etcd.deploy`), so don't rely on scheduled snapshots to protect those tenant clusters. In the `vcluster.yaml`, it is configured under `snapshots`. Using the UI, you can configure the management of snapshots in the config options of a tenant cluster under **Snapshots**. Though snapshot configuration is configured on the tenant cluster itself, the capability and logic of scheduling snapshots is in vCluster Platform. note Auto Snapshot requires platform v4.4.0 or later. ## Configure[​](#configure "Direct link to Configure") * vcluster.yaml * vCluster CLI * Platform UI ### Enable auto snapshots[​](#enable-auto-snapshots "Direct link to Enable auto snapshots") To enable auto snapshots, add the following configuration to your `vcluster.yaml`: Auto Snapshots configuration ``` snapshots: auto: # Take a snapshot every 12 hours schedule: 0 */12 * * * # Default is UTC # Options are at https://en.wikipedia.org/wiki/List_of_tz_database_time_zones timezone: America/New_York retention: period: 30 maxSnapshots: 14 storage: type: s3 s3: url: s3://my-bucket/path ``` ### S3 bucket example[​](#s3-bucket-example "Direct link to S3 bucket example") Snapshots can be stored in an S3 bucket. Configure snapshots with S3 Standalone deployments Automatic snapshots are supported for standalone deployments, with two differences: AWS pod identity authentication is not available, and the credentials secret must be created in the project namespace of the standalone cluster. #### S3 configuration options[​](#s3-configuration-options "Direct link to S3 configuration options") | Option | Description | | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `auto.storage.type.s3.url` | URL of the AWS S3 bucket. Must be pre-fixed with `s3://`. | | `auto.storage.type.s3.credential` | References the AWS credentials as a Kubernetes secret | | `auto.storage.type.s3.credential.name` | Name of the Kubernetes secret. | | `auto.storage.type.s3.credential.namespace` | Namespace of the Kubernetes secret. The secret must be deployed on the host of where the vCluster control plane pod is deployed to. | #### Authenticate with AWS Pod identity[​](#authenticate-with-aws-pod-identity "Direct link to Authenticate with AWS Pod identity") Not available for standalone AWS pod identity authentication is not supported for standalone deployments. Use S3 credentials stored in a Kubernetes secret instead. When using AWS S3 buckets, it is recommended to authenticate using [AWS pod identity](https://aws.amazon.com/blogs/containers/amazon-eks-pod-identity-a-new-way-for-applications-on-eks-to-obtain-iam-credentials/). info The EKS Pod Identity association must be created for the vCluster control plane pod. The vCluster control plane pod is the one that runs inside the control plane cluster namespace automatically created by vCluster Platform for each vCluster—typically named: `loft--`. By default, this pod uses the service account: `vc-`. This is the service account that must be associated with your EKS Pod Identity role so that the vCluster control plane can authenticate to AWS when performing scheduled snapshot creation to S3. Example vcluster.yaml configuring snapshots into s3 with a pod identity ``` snapshots: auto: # Take a snapshot every 12 hours schedule: 0 */12 * * * storage: type: s3 s3: # URL of location of S3-compatible bucket # Must be prefixed with `s3://` url: s3:///snapshots ``` #### Authenticate with S3 credentials as a secret[​](#authenticate-with-s3-credentials-as-a-secret "Direct link to Authenticate with S3 credentials as a secret") Alternatively, you can create a Kubernetes secret with your S3 credentials. For standalone deployments, the secret must be created in the project namespace where the standalone cluster is created. The `credential.namespace` field is still part of the schema, but standalone deployments ignore it at runtime and always read the secret from that project namespace. 1. Create a Kubernetes secret of your S3 credentials. Create this secret on the host of where the vCluster control plane is deployed. It could be deployed in the namespace of the vCluster or a different namespace. If the vCluster is externally deployed, ensure the vCluster ClusterRole has permission to read the secret. The vCluster ClusterRole follows the naming pattern `vc--v-`. The secret needs to contain all these three keys: * `AWS_ACCESS_KEY_ID` * `AWS_SECRET_ACCESS_KEY` * `AWS_SESSION_TOKEN` AWS\_SESSION\_TOKEN with static credentials `AWS_SESSION_TOKEN` must be present in the secret even if you don't use one. This applies to long-term static credentials and to S3-compatible storage that doesn't issue session tokens, such as MinIO, Ceph, or Dell ECS. If you don't have a session token, set the key to an empty string. Create AWS credentials secret ``` kubectl create -f - </ # Secret must be located on the host cluster that the vCluster is deployed on credential: secretName: aws-cred secretNamespace: p-default ``` ### OCI image registries[​](#oci-image-registries "Direct link to OCI image registries") Configure snapshots with OCI #### OCI configuration options[​](#oci-configuration-options "Direct link to OCI configuration options") | Option | Description | | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `auto.storage.type.oci.repository` | OCI registry address. Must be pre-fixed with `oci://` | | `auto.storage.type.oci.credential` | Reference the OCI credentials as a Kubernetes secret | | `auto.storage.type.oci.credential.name` | Name of the Kubernetes secret | | `auto.storage.type.oci.credential.namespace` | Namespace of the Kubernetes secret. The secret must be deployed on the host of where the vCluster control plane pod is deployed to. | | `auto.storage.type.oci.username` | Username of the credentials to access the OCI registry | | `auto.storage.type.oci.password` | Password of the credentials to access the OCI registry | #### Authenticate with credentials in a Kubernetes secret[​](#authenticate-with-credentials-in-a-kubernetes-secret "Direct link to Authenticate with credentials in a Kubernetes secret") It's recommended to store your credentials to your OCI registry in a secret and reference the secret in the `vcluster.yaml` of your tenant cluster configuration. This protects the details of your credentials. For standalone deployments, the secret must be created in the project namespace where the standalone cluster is created. The `credential.namespace` field is still part of the schema, but standalone deployments ignore it at runtime and always read the secret from that project namespace. 1. Create a Kubernetes secret of your credentials to your OCI registry. Create this secret on the host of where the vCluster control plane is deployed. It could be deployed in the namespace of the vCluster or a different namespace. If the vCluster is externally deployed, ensure the vCluster ClusterRole has permission to read the secret. The vCluster ClusterRole follows the naming pattern `vc--v-`. The secret needs to contain: * `username` * `password` Create OCI credentials secret ``` kubectl create -f - < for a tenant cluster snapshot to be taken #### `timezone` string [​](#snapshots-auto-timezone "Direct link to snapshots-auto-timezone")Timezone specifies time zone used for scheduled snapshot operations. Defaults to UTC. Accepts the same format as time.LoadLocation() in Go (). The value should be a location name corresponding to a file in the IANA Time Zone database, such as "America/New\_York". #### `retention` object [​](#snapshots-auto-retention "Direct link to snapshots-auto-retention")Retention specifies how long snapshots will be kept ##### `period` integer [​](#snapshots-auto-retention-period "Direct link to snapshots-auto-retention-period")Period defines the number of days a snapshot will be kept ##### `maxSnapshots` integer [​](#snapshots-auto-retention-maxSnapshots "Direct link to snapshots-auto-retention-maxSnapshots")MaxSnapshots defines the number of snapshots that can be taken #### `storage` object [​](#snapshots-auto-storage "Direct link to snapshots-auto-storage")Storage specifies where the snapshot will be stored ##### `type` string [​](#snapshots-auto-storage-type "Direct link to snapshots-auto-storage-type")Type specifies supported type of storage services for a snapshot S3/OCI/Container, see ##### `s3` object [​](#snapshots-auto-storage-s3 "Direct link to snapshots-auto-storage-s3")S3 holds configuration for storing snapshots in S3-compatible bucket ##### `url` string [​](#snapshots-auto-storage-s3-url "Direct link to snapshots-auto-storage-s3-url")Url specifies url to the storage service ##### `credential` object [​](#snapshots-auto-storage-s3-credential "Direct link to snapshots-auto-storage-s3-credential")Credential secret with the S3 Credentials, it should contain AWS\_ACCESS\_KEY\_ID, AWS\_SECRET\_ACCESS\_KEY, AWS\_SESSION\_TOKEN ##### `secretName` string [​](#snapshots-auto-storage-s3-credential-secretName "Direct link to snapshots-auto-storage-s3-credential-secretName")SecretName is the secret name with credential ##### `secretNamespace` string [​](#snapshots-auto-storage-s3-credential-secretNamespace "Direct link to snapshots-auto-storage-s3-credential-secretNamespace")SecretNamespace is the secret namespace with credential ##### `oci` object [​](#snapshots-auto-storage-oci "Direct link to snapshots-auto-storage-oci")OCI holds configuration for storing snapshots in OCI image registries ##### `repository` string [​](#snapshots-auto-storage-oci-repository "Direct link to snapshots-auto-storage-oci-repository")Repository OCI repository to store the snapshot ##### `credential` object [​](#snapshots-auto-storage-oci-credential "Direct link to snapshots-auto-storage-oci-credential")Credential secret with the OCI Credentials ##### `secretName` string [​](#snapshots-auto-storage-oci-credential-secretName "Direct link to snapshots-auto-storage-oci-credential-secretName")SecretName is the secret name with credential ##### `secretNamespace` string [​](#snapshots-auto-storage-oci-credential-secretNamespace "Direct link to snapshots-auto-storage-oci-credential-secretNamespace")SecretNamespace is the secret namespace with credential ##### `username` string [​](#snapshots-auto-storage-oci-username "Direct link to snapshots-auto-storage-oci-username")Username to authenticate with the OCI registry ##### `password` string [​](#snapshots-auto-storage-oci-password "Direct link to snapshots-auto-storage-oci-password")Password to authenticate with the OCI registry ##### `container` object [​](#snapshots-auto-storage-container "Direct link to snapshots-auto-storage-container")Container holds configuration for storing snapshots as local files inside a vCluster container ##### `path` string [​](#snapshots-auto-storage-container-path "Direct link to snapshots-auto-storage-container-path")Path specifies directory to store the snapshot ##### `volume` object [​](#snapshots-auto-storage-container-volume "Direct link to snapshots-auto-storage-container-volume")Volume specifies which volume needs to be mounted into the container to store the snapshot ##### `name` string [​](#snapshots-auto-storage-container-volume-name "Direct link to snapshots-auto-storage-container-volume-name")Name to be used to mount the volume ##### `path` string [​](#snapshots-auto-storage-container-volume-path "Direct link to snapshots-auto-storage-container-volume-path")Path to the volume mount ##### `azure` object [​](#snapshots-auto-storage-azure "Direct link to snapshots-auto-storage-azure")Azure holds configuration for storing snapshots in Azure Blob Storage ##### `blobUrl` string [​](#snapshots-auto-storage-azure-blobUrl "Direct link to snapshots-auto-storage-azure-blobUrl")BlobURL specifies the Azure Blob Storage URL in the format https\://{account}.blob.core.windows.net/{container}/{path} ##### `credential` object [​](#snapshots-auto-storage-azure-credential "Direct link to snapshots-auto-storage-azure-credential")Credential secret with the Azure credentials. The secret should contain either: AZURE\_STORAGE\_KEY (storage account access key), or AZURE\_TENANT\_ID, AZURE\_CLIENT\_ID, AZURE\_CLIENT\_SECRET, AZURE\_SUBSCRIPTION\_ID, AZURE\_RESOURCE\_GROUP (service principal) ##### `secretName` string [​](#snapshots-auto-storage-azure-credential-secretName "Direct link to snapshots-auto-storage-azure-credential-secretName")SecretName is the secret name with credential ##### `secretNamespace` string [​](#snapshots-auto-storage-azure-credential-secretNamespace "Direct link to snapshots-auto-storage-azure-credential-secretNamespace")SecretNamespace is the secret namespace with credential --- # Sync from host to virtual Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Read more about [how syncing works](https://vcluster.com/docs/vcluster/configure/vcluster-yaml) before deciding which resources to sync from the control plane cluster to the tenant cluster. vCluster can sync certain resources from the control plane cluster to make them available inside the tenant cluster, but when these resources are synced, they are only synced in read-only mode. No changes to the resource in the tenant cluster syncs back to the control plane cluster as the resources are shared across the control plane cluster. A good example would be nodes, which are nice to view inside the tenant cluster and can be also used to enabled certain features such as [scheduling inside the vCluster](https://vcluster.com/docs/vcluster/next/configure/vcluster-yaml/control-plane/other/advanced/virtual-scheduler), but you wouldn't want your tenant cluster to change the node itself. Another benefit of only syncing from the control plane cluster is that vCluster itself only requires read-only RBAC permissions. vCluster also lets you sync custom resources with the [custom resource syncer](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/custom-resources.md) There are a couple of labels that are created on the control plane cluster by vCluster that never get synced to the tenant cluster resource. These labels are: * `release` (Label is needed to avoid conflicts with the vCluster pods themselves). * `vcluster.loft.sh/namespace` * `vcluster.loft.sh/managed-by` * `vcluster.loft.sh/controlled-by` * `vcluster.loft.sh/ns-label-*` These labels are added as management labels when syncing from the tenant cluster to the control plane cluster, but skipped when syncing from the control plane cluster to the tenant cluster to prevent sync loops. ### Resource creation behavior[​](#resource-creation-behavior "Direct link to Resource creation behavior") #### Cluster-scoped resources[​](#cluster-scoped-resources "Direct link to Cluster-scoped resources") The following resources are not created during sync if they don’t already exist in the tenant cluster: * [CSINodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-nodes.md) * [CSIDrivers](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-drivers.md) The following resources are created in the tenant cluster as read-only if they don't already exist during the sync: * [Nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md) * [IngressClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/ingress-classes.md) * [Gateways and GatewayClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/gateways.md) * [StorageClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/storage-classes.md) * Cluster-scoped [CustomResources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/custom-resources.md) #### Namespaced resources[​](#namespaced-resources "Direct link to Namespaced resources") Namespaced resources are automatically created in the tenant cluster if they do not already exist. Before synchronization occurs, the mapping between the control plane cluster source namespace name and the corresponding tenant cluster target namespace name must be defined. This applies to: * [Events](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/events.md) * [ConfigMaps](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/configmaps.md) * [Secrets](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/secrets.md) * Namespace-scoped [CustomResources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/custom-resources.md) For CSIStorageCapacities, synchronization occurs only when a corresponding mapping exists between the physical StorageClass name in the control plane cluster and the StorageClass name in the tenant cluster and the capacity aligns with the associated physical node: * [CSIStorageCapacities](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-storage-capacities.md) ### Enabled by default[​](#enabled-by-default "Direct link to Enabled by default") * [Events](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/events.md) ### Disabled by default[​](#disabled-by-default "Direct link to Disabled by default") * [CustomResources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/custom-resources.md) * [Nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md) * [IngressClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/ingress-classes.md) * [Gateways and GatewayClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/gateways.md) * [StorageClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/storage-classes.md) * [CSINodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-nodes.md) * [CSIDrivers](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-drivers.md) * [CSIStorageCapacities](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-storage-capacities.md) * [ConfigMaps](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/configmaps.md) * [Secrets](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/secrets.md) ### No bi-directional syncing[​](#no-bi-directional-syncing "Direct link to No bi-directional syncing") Since syncing resources from the control plane cluster is in read only mode and changes in the tenant cluster do not get applied to the resource in the control plane cluster, bi-directional syncing does not exist across these resources. ## Config reference[​](#config-reference "Direct link to Config reference") ## `fromHost` object [​](#fromHost "Direct link to fromHost")Configure what resources vCluster should sync from the host cluster to the virtual cluster. ### `nodes` object [​](#fromHost-nodes "Direct link to fromHost-nodes")Nodes defines if nodes should get synced from the host cluster to the virtual cluster, but not back. #### `enabled` boolean false [​](#fromHost-nodes-enabled "Direct link to fromHost-nodes-enabled")Enabled specifies if syncing real nodes should be enabled. If this is disabled, vCluster will create fake nodes instead. #### `syncBackChanges` boolean false [​](#fromHost-nodes-syncBackChanges "Direct link to fromHost-nodes-syncBackChanges")SyncBackChanges enables syncing labels and taints from the virtual cluster to the host cluster. If this is enabled someone within the virtual cluster will be able to change the labels and taints of the host cluster node. #### `clearImageStatus` boolean false [​](#fromHost-nodes-clearImageStatus "Direct link to fromHost-nodes-clearImageStatus")ClearImageStatus will erase the image status when syncing a node. This allows to hide images that are pulled by the node. #### `selector` object [​](#fromHost-nodes-selector "Direct link to fromHost-nodes-selector")Selector can be used to define more granular what nodes should get synced from the host cluster to the virtual cluster. ##### `all` boolean false [​](#fromHost-nodes-selector-all "Direct link to fromHost-nodes-selector-all")All specifies if all nodes should get synced by vCluster from the host to the virtual cluster or only the ones where pods are assigned to. ##### `labels` object {} [​](#fromHost-nodes-selector-labels "Direct link to fromHost-nodes-selector-labels")Labels are the node labels used to sync nodes from host cluster to virtual cluster. This will also set the node selector when syncing a pod from virtual cluster to host cluster to the same value. #### `patches` object\[] [​](#fromHost-nodes-patches "Direct link to fromHost-nodes-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-nodes-patches-path "Direct link to fromHost-nodes-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-nodes-patches-expression "Direct link to fromHost-nodes-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-nodes-patches-reverseExpression "Direct link to fromHost-nodes-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-nodes-patches-reference "Direct link to fromHost-nodes-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-nodes-patches-reference-apiVersion "Direct link to fromHost-nodes-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-nodes-patches-reference-apiVersionPath "Direct link to fromHost-nodes-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-nodes-patches-reference-kind "Direct link to fromHost-nodes-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-nodes-patches-reference-kindPath "Direct link to fromHost-nodes-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-nodes-patches-reference-namePath "Direct link to fromHost-nodes-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-nodes-patches-reference-namespacePath "Direct link to fromHost-nodes-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-nodes-patches-labels "Direct link to fromHost-nodes-patches-labels")Labels treats the path value as a labels selector. ### `events` object [​](#fromHost-events "Direct link to fromHost-events")Events defines if events should get synced from the host cluster to the virtual cluster, but not back. #### `enabled` boolean true [​](#fromHost-events-enabled "Direct link to fromHost-events-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#fromHost-events-patches "Direct link to fromHost-events-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-events-patches-path "Direct link to fromHost-events-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-events-patches-expression "Direct link to fromHost-events-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-events-patches-reverseExpression "Direct link to fromHost-events-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-events-patches-reference "Direct link to fromHost-events-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-events-patches-reference-apiVersion "Direct link to fromHost-events-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-events-patches-reference-apiVersionPath "Direct link to fromHost-events-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-events-patches-reference-kind "Direct link to fromHost-events-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-events-patches-reference-kindPath "Direct link to fromHost-events-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-events-patches-reference-namePath "Direct link to fromHost-events-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-events-patches-reference-namespacePath "Direct link to fromHost-events-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-events-patches-labels "Direct link to fromHost-events-patches-labels")Labels treats the path value as a labels selector. ### `ingressClasses` object [​](#fromHost-ingressClasses "Direct link to fromHost-ingressClasses")IngressClasses defines if ingress classes should get synced from the host cluster to the virtual cluster, but not back. #### `enabled` boolean false [​](#fromHost-ingressClasses-enabled "Direct link to fromHost-ingressClasses-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#fromHost-ingressClasses-patches "Direct link to fromHost-ingressClasses-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-ingressClasses-patches-path "Direct link to fromHost-ingressClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-ingressClasses-patches-expression "Direct link to fromHost-ingressClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-ingressClasses-patches-reverseExpression "Direct link to fromHost-ingressClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-ingressClasses-patches-reference "Direct link to fromHost-ingressClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-ingressClasses-patches-reference-apiVersion "Direct link to fromHost-ingressClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-ingressClasses-patches-reference-apiVersionPath "Direct link to fromHost-ingressClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-ingressClasses-patches-reference-kind "Direct link to fromHost-ingressClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-ingressClasses-patches-reference-kindPath "Direct link to fromHost-ingressClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-ingressClasses-patches-reference-namePath "Direct link to fromHost-ingressClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-ingressClasses-patches-reference-namespacePath "Direct link to fromHost-ingressClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-ingressClasses-patches-labels "Direct link to fromHost-ingressClasses-patches-labels")Labels treats the path value as a labels selector. #### `selector` object [​](#fromHost-ingressClasses-selector "Direct link to fromHost-ingressClasses-selector")Selector defines the selector to use for the resource. If not set, all resources of that type will be synced. ##### `matchLabels` object [​](#fromHost-ingressClasses-selector-matchLabels "Direct link to fromHost-ingressClasses-selector-matchLabels") ##### `matchExpressions` object\[] [​](#fromHost-ingressClasses-selector-matchExpressions "Direct link to fromHost-ingressClasses-selector-matchExpressions") ##### `key` string [​](#fromHost-ingressClasses-selector-matchExpressions-key "Direct link to fromHost-ingressClasses-selector-matchExpressions-key") ##### `operator` string [​](#fromHost-ingressClasses-selector-matchExpressions-operator "Direct link to fromHost-ingressClasses-selector-matchExpressions-operator") ##### `values` string\[] [​](#fromHost-ingressClasses-selector-matchExpressions-values "Direct link to fromHost-ingressClasses-selector-matchExpressions-values") ### `gatewayClasses` object [​](#fromHost-gatewayClasses "Direct link to fromHost-gatewayClasses")GatewayClasses defines if gateway classes should get synced from the control plane cluster to the tenant cluster, but not back. #### `enabled` boolean false [​](#fromHost-gatewayClasses-enabled "Direct link to fromHost-gatewayClasses-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#fromHost-gatewayClasses-patches "Direct link to fromHost-gatewayClasses-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-gatewayClasses-patches-path "Direct link to fromHost-gatewayClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-gatewayClasses-patches-expression "Direct link to fromHost-gatewayClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-gatewayClasses-patches-reverseExpression "Direct link to fromHost-gatewayClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-gatewayClasses-patches-reference "Direct link to fromHost-gatewayClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-gatewayClasses-patches-reference-apiVersion "Direct link to fromHost-gatewayClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-gatewayClasses-patches-reference-apiVersionPath "Direct link to fromHost-gatewayClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-gatewayClasses-patches-reference-kind "Direct link to fromHost-gatewayClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-gatewayClasses-patches-reference-kindPath "Direct link to fromHost-gatewayClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-gatewayClasses-patches-reference-namePath "Direct link to fromHost-gatewayClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-gatewayClasses-patches-reference-namespacePath "Direct link to fromHost-gatewayClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-gatewayClasses-patches-labels "Direct link to fromHost-gatewayClasses-patches-labels")Labels treats the path value as a labels selector. #### `selector` object [​](#fromHost-gatewayClasses-selector "Direct link to fromHost-gatewayClasses-selector")Selector defines the selector to use for the resource. If not set, all resources of that type will be synced. ##### `matchLabels` object [​](#fromHost-gatewayClasses-selector-matchLabels "Direct link to fromHost-gatewayClasses-selector-matchLabels") ##### `matchExpressions` object\[] [​](#fromHost-gatewayClasses-selector-matchExpressions "Direct link to fromHost-gatewayClasses-selector-matchExpressions") ##### `key` string [​](#fromHost-gatewayClasses-selector-matchExpressions-key "Direct link to fromHost-gatewayClasses-selector-matchExpressions-key") ##### `operator` string [​](#fromHost-gatewayClasses-selector-matchExpressions-operator "Direct link to fromHost-gatewayClasses-selector-matchExpressions-operator") ##### `values` string\[] [​](#fromHost-gatewayClasses-selector-matchExpressions-values "Direct link to fromHost-gatewayClasses-selector-matchExpressions-values") ### `gateways` object [​](#fromHost-gateways "Direct link to fromHost-gateways")Gateways defines if selected control plane Gateways should get synced from the control plane cluster to the tenant cluster, but not back. #### `enabled` boolean false [​](#fromHost-gateways-enabled "Direct link to fromHost-gateways-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#fromHost-gateways-patches "Direct link to fromHost-gateways-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-gateways-patches-path "Direct link to fromHost-gateways-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-gateways-patches-expression "Direct link to fromHost-gateways-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-gateways-patches-reverseExpression "Direct link to fromHost-gateways-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-gateways-patches-reference "Direct link to fromHost-gateways-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-gateways-patches-reference-apiVersion "Direct link to fromHost-gateways-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-gateways-patches-reference-apiVersionPath "Direct link to fromHost-gateways-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-gateways-patches-reference-kind "Direct link to fromHost-gateways-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-gateways-patches-reference-kindPath "Direct link to fromHost-gateways-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-gateways-patches-reference-namePath "Direct link to fromHost-gateways-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-gateways-patches-reference-namespacePath "Direct link to fromHost-gateways-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-gateways-patches-labels "Direct link to fromHost-gateways-patches-labels")Labels treats the path value as a labels selector. #### `selector` object [​](#fromHost-gateways-selector "Direct link to fromHost-gateways-selector")Selector defines the selector to use for the resource. If not set, all resources of that type will be synced. ##### `matchLabels` object [​](#fromHost-gateways-selector-matchLabels "Direct link to fromHost-gateways-selector-matchLabels") ##### `matchExpressions` object\[] [​](#fromHost-gateways-selector-matchExpressions "Direct link to fromHost-gateways-selector-matchExpressions") ##### `key` string [​](#fromHost-gateways-selector-matchExpressions-key "Direct link to fromHost-gateways-selector-matchExpressions-key") ##### `operator` string [​](#fromHost-gateways-selector-matchExpressions-operator "Direct link to fromHost-gateways-selector-matchExpressions-operator") ##### `values` string\[] [​](#fromHost-gateways-selector-matchExpressions-values "Direct link to fromHost-gateways-selector-matchExpressions-values") #### `mappings` object [​](#fromHost-gateways-mappings "Direct link to fromHost-gateways-mappings")Mappings define control plane Gateway namespace/name to tenant-facing namespace/name placement. ##### `byName` object {} [​](#fromHost-gateways-mappings-byName "Direct link to fromHost-gateways-mappings-byName")ByName is a map of control-plane-object-namespace/control-plane-object-name: tenant-object-namespace/tenant-object-name. There are several wildcards supported:1) To match all objects in a control plane namespace and sync them to a different namespace in the tenant cluster: byName: "foo/*": "foo-in-virtual/*" 2) To match a specific object in the control plane namespace and sync it to the same namespace with the same name: byName: "foo/my-object": "foo/my-object" 3) To match a specific object in the control plane namespace and sync it to the same namespace with a different name: byName: "foo/my-object": "foo/my-virtual-object" 4) To match all objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "": "my-virtual-namespace/\*" 5) To match specific objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "/my-object": "my-virtual-namespace/my-object" #### `allowedRoutes` object [​](#fromHost-gateways-allowedRoutes "Direct link to fromHost-gateways-allowedRoutes")AllowedRoutes configures the tenant-facing allowedRoutes policy shown on imported Gateways and enforced for Routes. ##### `defaultVirtualNamespacePolicy` object [​](#fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy "Direct link to fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy") ##### `from` string [​](#fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy-from "Direct link to fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy-from") ##### `selector` object [​](#fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy-selector "Direct link to fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy-selector") ##### `matchLabels` object [​](#fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy-selector-matchLabels "Direct link to fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy-selector-matchLabels") ##### `matchExpressions` object\[] [​](#fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy-selector-matchExpressions "Direct link to fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy-selector-matchExpressions") ##### `key` string [​](#fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy-selector-matchExpressions-key "Direct link to fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy-selector-matchExpressions-key") ##### `operator` string [​](#fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy-selector-matchExpressions-operator "Direct link to fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy-selector-matchExpressions-operator") ##### `values` string\[] [​](#fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy-selector-matchExpressions-values "Direct link to fromHost-gateways-allowedRoutes-defaultVirtualNamespacePolicy-selector-matchExpressions-values") ##### `overrides` object\[] [​](#fromHost-gateways-allowedRoutes-overrides "Direct link to fromHost-gateways-allowedRoutes-overrides") ##### `hostNamespace` string [​](#fromHost-gateways-allowedRoutes-overrides-hostNamespace "Direct link to fromHost-gateways-allowedRoutes-overrides-hostNamespace") ##### `name` string [​](#fromHost-gateways-allowedRoutes-overrides-name "Direct link to fromHost-gateways-allowedRoutes-overrides-name") ##### `virtualNamespacePolicy` object [​](#fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy "Direct link to fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy") ##### `from` string [​](#fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy-from "Direct link to fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy-from") ##### `selector` object [​](#fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy-selector "Direct link to fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy-selector") ##### `matchLabels` object [​](#fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy-selector-matchLabels "Direct link to fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy-selector-matchLabels") ##### `matchExpressions` object\[] [​](#fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy-selector-matchExpressions "Direct link to fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy-selector-matchExpressions") ##### `key` string [​](#fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy-selector-matchExpressions-key "Direct link to fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy-selector-matchExpressions-key") ##### `operator` string [​](#fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy-selector-matchExpressions-operator "Direct link to fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy-selector-matchExpressions-operator") ##### `values` string\[] [​](#fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy-selector-matchExpressions-values "Direct link to fromHost-gateways-allowedRoutes-overrides-virtualNamespacePolicy-selector-matchExpressions-values") ##### `allowedHostnames` string\[] [​](#fromHost-gateways-allowedRoutes-overrides-allowedHostnames "Direct link to fromHost-gateways-allowedRoutes-overrides-allowedHostnames") #### `status` object [​](#fromHost-gateways-status "Direct link to fromHost-gateways-status")Status configures how Gateway status is mirrored. ##### `exposeAddresses` boolean false [​](#fromHost-gateways-status-exposeAddresses "Direct link to fromHost-gateways-status-exposeAddresses") #### `metadata` object [​](#fromHost-gateways-metadata "Direct link to fromHost-gateways-metadata")Metadata configures imported Gateway metadata visibility. ##### `exposeSourceGateway` boolean false [​](#fromHost-gateways-metadata-exposeSourceGateway "Direct link to fromHost-gateways-metadata-exposeSourceGateway") #### `sanitize` object [​](#fromHost-gateways-sanitize "Direct link to fromHost-gateways-sanitize")Sanitize configures sensitive control plane field sanitization. ##### `certificateRefs` boolean true [​](#fromHost-gateways-sanitize-certificateRefs "Direct link to fromHost-gateways-sanitize-certificateRefs") ##### `infrastructure` boolean true [​](#fromHost-gateways-sanitize-infrastructure "Direct link to fromHost-gateways-sanitize-infrastructure") ### `runtimeClasses` object [​](#fromHost-runtimeClasses "Direct link to fromHost-runtimeClasses")RuntimeClasses defines if runtime classes should get synced from the host cluster to the virtual cluster, but not back. #### `enabled` boolean false [​](#fromHost-runtimeClasses-enabled "Direct link to fromHost-runtimeClasses-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#fromHost-runtimeClasses-patches "Direct link to fromHost-runtimeClasses-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-runtimeClasses-patches-path "Direct link to fromHost-runtimeClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-runtimeClasses-patches-expression "Direct link to fromHost-runtimeClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-runtimeClasses-patches-reverseExpression "Direct link to fromHost-runtimeClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-runtimeClasses-patches-reference "Direct link to fromHost-runtimeClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-runtimeClasses-patches-reference-apiVersion "Direct link to fromHost-runtimeClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-runtimeClasses-patches-reference-apiVersionPath "Direct link to fromHost-runtimeClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-runtimeClasses-patches-reference-kind "Direct link to fromHost-runtimeClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-runtimeClasses-patches-reference-kindPath "Direct link to fromHost-runtimeClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-runtimeClasses-patches-reference-namePath "Direct link to fromHost-runtimeClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-runtimeClasses-patches-reference-namespacePath "Direct link to fromHost-runtimeClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-runtimeClasses-patches-labels "Direct link to fromHost-runtimeClasses-patches-labels")Labels treats the path value as a labels selector. #### `selector` object [​](#fromHost-runtimeClasses-selector "Direct link to fromHost-runtimeClasses-selector")Selector defines the selector to use for the resource. If not set, all resources of that type will be synced. ##### `matchLabels` object [​](#fromHost-runtimeClasses-selector-matchLabels "Direct link to fromHost-runtimeClasses-selector-matchLabels") ##### `matchExpressions` object\[] [​](#fromHost-runtimeClasses-selector-matchExpressions "Direct link to fromHost-runtimeClasses-selector-matchExpressions") ##### `key` string [​](#fromHost-runtimeClasses-selector-matchExpressions-key "Direct link to fromHost-runtimeClasses-selector-matchExpressions-key") ##### `operator` string [​](#fromHost-runtimeClasses-selector-matchExpressions-operator "Direct link to fromHost-runtimeClasses-selector-matchExpressions-operator") ##### `values` string\[] [​](#fromHost-runtimeClasses-selector-matchExpressions-values "Direct link to fromHost-runtimeClasses-selector-matchExpressions-values") ### `priorityClasses` object [​](#fromHost-priorityClasses "Direct link to fromHost-priorityClasses")PriorityClasses defines if priority classes classes should get synced from the host cluster to the virtual cluster, but not back. #### `enabled` boolean false [​](#fromHost-priorityClasses-enabled "Direct link to fromHost-priorityClasses-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#fromHost-priorityClasses-patches "Direct link to fromHost-priorityClasses-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-priorityClasses-patches-path "Direct link to fromHost-priorityClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-priorityClasses-patches-expression "Direct link to fromHost-priorityClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-priorityClasses-patches-reverseExpression "Direct link to fromHost-priorityClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-priorityClasses-patches-reference "Direct link to fromHost-priorityClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-priorityClasses-patches-reference-apiVersion "Direct link to fromHost-priorityClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-priorityClasses-patches-reference-apiVersionPath "Direct link to fromHost-priorityClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-priorityClasses-patches-reference-kind "Direct link to fromHost-priorityClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-priorityClasses-patches-reference-kindPath "Direct link to fromHost-priorityClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-priorityClasses-patches-reference-namePath "Direct link to fromHost-priorityClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-priorityClasses-patches-reference-namespacePath "Direct link to fromHost-priorityClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-priorityClasses-patches-labels "Direct link to fromHost-priorityClasses-patches-labels")Labels treats the path value as a labels selector. #### `selector` object [​](#fromHost-priorityClasses-selector "Direct link to fromHost-priorityClasses-selector")Selector defines the selector to use for the resource. If not set, all resources of that type will be synced. ##### `matchLabels` object [​](#fromHost-priorityClasses-selector-matchLabels "Direct link to fromHost-priorityClasses-selector-matchLabels") ##### `matchExpressions` object\[] [​](#fromHost-priorityClasses-selector-matchExpressions "Direct link to fromHost-priorityClasses-selector-matchExpressions") ##### `key` string [​](#fromHost-priorityClasses-selector-matchExpressions-key "Direct link to fromHost-priorityClasses-selector-matchExpressions-key") ##### `operator` string [​](#fromHost-priorityClasses-selector-matchExpressions-operator "Direct link to fromHost-priorityClasses-selector-matchExpressions-operator") ##### `values` string\[] [​](#fromHost-priorityClasses-selector-matchExpressions-values "Direct link to fromHost-priorityClasses-selector-matchExpressions-values") ### `storageClasses` object [​](#fromHost-storageClasses "Direct link to fromHost-storageClasses")StorageClasses defines if storage classes should get synced from the host cluster to the virtual cluster, but not back. If auto, is automatically enabled when the virtual scheduler is enabled. #### `enabled` string|boolean auto [​](#fromHost-storageClasses-enabled "Direct link to fromHost-storageClasses-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#fromHost-storageClasses-patches "Direct link to fromHost-storageClasses-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-storageClasses-patches-path "Direct link to fromHost-storageClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-storageClasses-patches-expression "Direct link to fromHost-storageClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-storageClasses-patches-reverseExpression "Direct link to fromHost-storageClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-storageClasses-patches-reference "Direct link to fromHost-storageClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-storageClasses-patches-reference-apiVersion "Direct link to fromHost-storageClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-storageClasses-patches-reference-apiVersionPath "Direct link to fromHost-storageClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-storageClasses-patches-reference-kind "Direct link to fromHost-storageClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-storageClasses-patches-reference-kindPath "Direct link to fromHost-storageClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-storageClasses-patches-reference-namePath "Direct link to fromHost-storageClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-storageClasses-patches-reference-namespacePath "Direct link to fromHost-storageClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-storageClasses-patches-labels "Direct link to fromHost-storageClasses-patches-labels")Labels treats the path value as a labels selector. #### `selector` object [​](#fromHost-storageClasses-selector "Direct link to fromHost-storageClasses-selector")Selector defines the selector to use for the resource. If not set, all resources of that type will be synced. ##### `matchLabels` object [​](#fromHost-storageClasses-selector-matchLabels "Direct link to fromHost-storageClasses-selector-matchLabels") ##### `matchExpressions` object\[] [​](#fromHost-storageClasses-selector-matchExpressions "Direct link to fromHost-storageClasses-selector-matchExpressions") ##### `key` string [​](#fromHost-storageClasses-selector-matchExpressions-key "Direct link to fromHost-storageClasses-selector-matchExpressions-key") ##### `operator` string [​](#fromHost-storageClasses-selector-matchExpressions-operator "Direct link to fromHost-storageClasses-selector-matchExpressions-operator") ##### `values` string\[] [​](#fromHost-storageClasses-selector-matchExpressions-values "Direct link to fromHost-storageClasses-selector-matchExpressions-values") ### `csiNodes` object [​](#fromHost-csiNodes "Direct link to fromHost-csiNodes")CSINodes defines if csi nodes should get synced from the host cluster to the virtual cluster, but not back. If auto, is automatically enabled when the virtual scheduler is enabled. #### `enabled` string|boolean auto [​](#fromHost-csiNodes-enabled "Direct link to fromHost-csiNodes-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#fromHost-csiNodes-patches "Direct link to fromHost-csiNodes-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-csiNodes-patches-path "Direct link to fromHost-csiNodes-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-csiNodes-patches-expression "Direct link to fromHost-csiNodes-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-csiNodes-patches-reverseExpression "Direct link to fromHost-csiNodes-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-csiNodes-patches-reference "Direct link to fromHost-csiNodes-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-csiNodes-patches-reference-apiVersion "Direct link to fromHost-csiNodes-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-csiNodes-patches-reference-apiVersionPath "Direct link to fromHost-csiNodes-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-csiNodes-patches-reference-kind "Direct link to fromHost-csiNodes-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-csiNodes-patches-reference-kindPath "Direct link to fromHost-csiNodes-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-csiNodes-patches-reference-namePath "Direct link to fromHost-csiNodes-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-csiNodes-patches-reference-namespacePath "Direct link to fromHost-csiNodes-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-csiNodes-patches-labels "Direct link to fromHost-csiNodes-patches-labels")Labels treats the path value as a labels selector. ### `csiDrivers` object [​](#fromHost-csiDrivers "Direct link to fromHost-csiDrivers")CSIDrivers defines if csi drivers should get synced from the host cluster to the virtual cluster, but not back. If auto, is automatically enabled when the virtual scheduler is enabled. #### `enabled` string|boolean auto [​](#fromHost-csiDrivers-enabled "Direct link to fromHost-csiDrivers-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#fromHost-csiDrivers-patches "Direct link to fromHost-csiDrivers-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-csiDrivers-patches-path "Direct link to fromHost-csiDrivers-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-csiDrivers-patches-expression "Direct link to fromHost-csiDrivers-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-csiDrivers-patches-reverseExpression "Direct link to fromHost-csiDrivers-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-csiDrivers-patches-reference "Direct link to fromHost-csiDrivers-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-csiDrivers-patches-reference-apiVersion "Direct link to fromHost-csiDrivers-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-csiDrivers-patches-reference-apiVersionPath "Direct link to fromHost-csiDrivers-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-csiDrivers-patches-reference-kind "Direct link to fromHost-csiDrivers-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-csiDrivers-patches-reference-kindPath "Direct link to fromHost-csiDrivers-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-csiDrivers-patches-reference-namePath "Direct link to fromHost-csiDrivers-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-csiDrivers-patches-reference-namespacePath "Direct link to fromHost-csiDrivers-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-csiDrivers-patches-labels "Direct link to fromHost-csiDrivers-patches-labels")Labels treats the path value as a labels selector. ### `csiStorageCapacities` object [​](#fromHost-csiStorageCapacities "Direct link to fromHost-csiStorageCapacities")CSIStorageCapacities defines if csi storage capacities should get synced from the host cluster to the virtual cluster, but not back. If auto, is automatically enabled when the virtual scheduler is enabled. #### `enabled` string|boolean auto [​](#fromHost-csiStorageCapacities-enabled "Direct link to fromHost-csiStorageCapacities-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#fromHost-csiStorageCapacities-patches "Direct link to fromHost-csiStorageCapacities-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-csiStorageCapacities-patches-path "Direct link to fromHost-csiStorageCapacities-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-csiStorageCapacities-patches-expression "Direct link to fromHost-csiStorageCapacities-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-csiStorageCapacities-patches-reverseExpression "Direct link to fromHost-csiStorageCapacities-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-csiStorageCapacities-patches-reference "Direct link to fromHost-csiStorageCapacities-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-csiStorageCapacities-patches-reference-apiVersion "Direct link to fromHost-csiStorageCapacities-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-csiStorageCapacities-patches-reference-apiVersionPath "Direct link to fromHost-csiStorageCapacities-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-csiStorageCapacities-patches-reference-kind "Direct link to fromHost-csiStorageCapacities-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-csiStorageCapacities-patches-reference-kindPath "Direct link to fromHost-csiStorageCapacities-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-csiStorageCapacities-patches-reference-namePath "Direct link to fromHost-csiStorageCapacities-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-csiStorageCapacities-patches-reference-namespacePath "Direct link to fromHost-csiStorageCapacities-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-csiStorageCapacities-patches-labels "Direct link to fromHost-csiStorageCapacities-patches-labels")Labels treats the path value as a labels selector. ### `customResources` {key: object} [​](#fromHost-customResources "Direct link to fromHost-customResources")CustomResources defines what custom resources should get synced read-only to the virtual cluster from the host cluster. vCluster will automatically add any required RBAC to the vCluster cluster role. #### `enabled` required boolean [​](#fromHost-customResources-enabled "Direct link to fromHost-customResources-enabled")Enabled defines if this option should be enabled. #### `scope` required string [​](#fromHost-customResources-scope "Direct link to fromHost-customResources-scope")Scope defines the scope of the resource #### `patches` object\[] [​](#fromHost-customResources-patches "Direct link to fromHost-customResources-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-customResources-patches-path "Direct link to fromHost-customResources-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-customResources-patches-expression "Direct link to fromHost-customResources-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-customResources-patches-reverseExpression "Direct link to fromHost-customResources-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-customResources-patches-reference "Direct link to fromHost-customResources-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-customResources-patches-reference-apiVersion "Direct link to fromHost-customResources-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-customResources-patches-reference-apiVersionPath "Direct link to fromHost-customResources-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-customResources-patches-reference-kind "Direct link to fromHost-customResources-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-customResources-patches-reference-kindPath "Direct link to fromHost-customResources-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-customResources-patches-reference-namePath "Direct link to fromHost-customResources-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-customResources-patches-reference-namespacePath "Direct link to fromHost-customResources-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-customResources-patches-labels "Direct link to fromHost-customResources-patches-labels")Labels treats the path value as a labels selector. #### `mappings` object [​](#fromHost-customResources-mappings "Direct link to fromHost-customResources-mappings")Mappings for Namespace and Object ##### `byName` object [​](#fromHost-customResources-mappings-byName "Direct link to fromHost-customResources-mappings-byName")ByName is a map of control-plane-object-namespace/control-plane-object-name: tenant-object-namespace/tenant-object-name. There are several wildcards supported:1) To match all objects in a control plane namespace and sync them to a different namespace in the tenant cluster: byName: "foo/*": "foo-in-virtual/*" 2) To match a specific object in the control plane namespace and sync it to the same namespace with the same name: byName: "foo/my-object": "foo/my-object" 3) To match a specific object in the control plane namespace and sync it to the same namespace with a different name: byName: "foo/my-object": "foo/my-virtual-object" 4) To match all objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "": "my-virtual-namespace/\*" 5) To match specific objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "/my-object": "my-virtual-namespace/my-object" ### `volumeSnapshotClasses` object [​](#fromHost-volumeSnapshotClasses "Direct link to fromHost-volumeSnapshotClasses")VolumeSnapshotClasses defines if volume snapshot classes created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean false [​](#fromHost-volumeSnapshotClasses-enabled "Direct link to fromHost-volumeSnapshotClasses-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#fromHost-volumeSnapshotClasses-patches "Direct link to fromHost-volumeSnapshotClasses-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-volumeSnapshotClasses-patches-path "Direct link to fromHost-volumeSnapshotClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-volumeSnapshotClasses-patches-expression "Direct link to fromHost-volumeSnapshotClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-volumeSnapshotClasses-patches-reverseExpression "Direct link to fromHost-volumeSnapshotClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-volumeSnapshotClasses-patches-reference "Direct link to fromHost-volumeSnapshotClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-volumeSnapshotClasses-patches-reference-apiVersion "Direct link to fromHost-volumeSnapshotClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-volumeSnapshotClasses-patches-reference-apiVersionPath "Direct link to fromHost-volumeSnapshotClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-volumeSnapshotClasses-patches-reference-kind "Direct link to fromHost-volumeSnapshotClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-volumeSnapshotClasses-patches-reference-kindPath "Direct link to fromHost-volumeSnapshotClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-volumeSnapshotClasses-patches-reference-namePath "Direct link to fromHost-volumeSnapshotClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-volumeSnapshotClasses-patches-reference-namespacePath "Direct link to fromHost-volumeSnapshotClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-volumeSnapshotClasses-patches-labels "Direct link to fromHost-volumeSnapshotClasses-patches-labels")Labels treats the path value as a labels selector. ### `configMaps` object [​](#fromHost-configMaps "Direct link to fromHost-configMaps")ConfigMaps defines if config maps in the host should get synced to the virtual cluster. #### `enabled` boolean false [​](#fromHost-configMaps-enabled "Direct link to fromHost-configMaps-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#fromHost-configMaps-patches "Direct link to fromHost-configMaps-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-configMaps-patches-path "Direct link to fromHost-configMaps-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-configMaps-patches-expression "Direct link to fromHost-configMaps-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-configMaps-patches-reverseExpression "Direct link to fromHost-configMaps-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-configMaps-patches-reference "Direct link to fromHost-configMaps-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-configMaps-patches-reference-apiVersion "Direct link to fromHost-configMaps-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-configMaps-patches-reference-apiVersionPath "Direct link to fromHost-configMaps-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-configMaps-patches-reference-kind "Direct link to fromHost-configMaps-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-configMaps-patches-reference-kindPath "Direct link to fromHost-configMaps-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-configMaps-patches-reference-namePath "Direct link to fromHost-configMaps-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-configMaps-patches-reference-namespacePath "Direct link to fromHost-configMaps-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-configMaps-patches-labels "Direct link to fromHost-configMaps-patches-labels")Labels treats the path value as a labels selector. #### `mappings` object [​](#fromHost-configMaps-mappings "Direct link to fromHost-configMaps-mappings")Mappings for Namespace and Object ##### `byName` object {} [​](#fromHost-configMaps-mappings-byName "Direct link to fromHost-configMaps-mappings-byName")ByName is a map of control-plane-object-namespace/control-plane-object-name: tenant-object-namespace/tenant-object-name. There are several wildcards supported:1) To match all objects in a control plane namespace and sync them to a different namespace in the tenant cluster: byName: "foo/*": "foo-in-virtual/*" 2) To match a specific object in the control plane namespace and sync it to the same namespace with the same name: byName: "foo/my-object": "foo/my-object" 3) To match a specific object in the control plane namespace and sync it to the same namespace with a different name: byName: "foo/my-object": "foo/my-virtual-object" 4) To match all objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "": "my-virtual-namespace/\*" 5) To match specific objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "/my-object": "my-virtual-namespace/my-object" ### `secrets` object [​](#fromHost-secrets "Direct link to fromHost-secrets")Secrets defines if secrets in the host should get synced to the virtual cluster. #### `enabled` boolean false [​](#fromHost-secrets-enabled "Direct link to fromHost-secrets-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#fromHost-secrets-patches "Direct link to fromHost-secrets-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-secrets-patches-path "Direct link to fromHost-secrets-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-secrets-patches-expression "Direct link to fromHost-secrets-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-secrets-patches-reverseExpression "Direct link to fromHost-secrets-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-secrets-patches-reference "Direct link to fromHost-secrets-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-secrets-patches-reference-apiVersion "Direct link to fromHost-secrets-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-secrets-patches-reference-apiVersionPath "Direct link to fromHost-secrets-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-secrets-patches-reference-kind "Direct link to fromHost-secrets-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-secrets-patches-reference-kindPath "Direct link to fromHost-secrets-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-secrets-patches-reference-namePath "Direct link to fromHost-secrets-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-secrets-patches-reference-namespacePath "Direct link to fromHost-secrets-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-secrets-patches-labels "Direct link to fromHost-secrets-patches-labels")Labels treats the path value as a labels selector. #### `mappings` object [​](#fromHost-secrets-mappings "Direct link to fromHost-secrets-mappings")Mappings for Namespace and Object ##### `byName` object {} [​](#fromHost-secrets-mappings-byName "Direct link to fromHost-secrets-mappings-byName")ByName is a map of control-plane-object-namespace/control-plane-object-name: tenant-object-namespace/tenant-object-name. There are several wildcards supported:1) To match all objects in a control plane namespace and sync them to a different namespace in the tenant cluster: byName: "foo/*": "foo-in-virtual/*" 2) To match a specific object in the control plane namespace and sync it to the same namespace with the same name: byName: "foo/my-object": "foo/my-object" 3) To match a specific object in the control plane namespace and sync it to the same namespace with a different name: byName: "foo/my-object": "foo/my-virtual-object" 4) To match all objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "": "my-virtual-namespace/\*" 5) To match specific objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "/my-object": "my-virtual-namespace/my-object" ### `deviceClasses` object [​](#fromHost-deviceClasses "Direct link to fromHost-deviceClasses")DeviceClasses defines if device classes in the host should get synced to the virtual cluster #### `enabled` boolean false [​](#fromHost-deviceClasses-enabled "Direct link to fromHost-deviceClasses-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#fromHost-deviceClasses-patches "Direct link to fromHost-deviceClasses-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#fromHost-deviceClasses-patches-path "Direct link to fromHost-deviceClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#fromHost-deviceClasses-patches-expression "Direct link to fromHost-deviceClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#fromHost-deviceClasses-patches-reverseExpression "Direct link to fromHost-deviceClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#fromHost-deviceClasses-patches-reference "Direct link to fromHost-deviceClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#fromHost-deviceClasses-patches-reference-apiVersion "Direct link to fromHost-deviceClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#fromHost-deviceClasses-patches-reference-apiVersionPath "Direct link to fromHost-deviceClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#fromHost-deviceClasses-patches-reference-kind "Direct link to fromHost-deviceClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#fromHost-deviceClasses-patches-reference-kindPath "Direct link to fromHost-deviceClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#fromHost-deviceClasses-patches-reference-namePath "Direct link to fromHost-deviceClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#fromHost-deviceClasses-patches-reference-namespacePath "Direct link to fromHost-deviceClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#fromHost-deviceClasses-patches-labels "Direct link to fromHost-deviceClasses-patches-labels")Labels treats the path value as a labels selector. #### `selector` object [​](#fromHost-deviceClasses-selector "Direct link to fromHost-deviceClasses-selector")Selector defines the selector to use for the resource. If not set, all resources of that type will be synced. ##### `matchLabels` object [​](#fromHost-deviceClasses-selector-matchLabels "Direct link to fromHost-deviceClasses-selector-matchLabels") ##### `matchExpressions` object\[] [​](#fromHost-deviceClasses-selector-matchExpressions "Direct link to fromHost-deviceClasses-selector-matchExpressions") ##### `key` string [​](#fromHost-deviceClasses-selector-matchExpressions-key "Direct link to fromHost-deviceClasses-selector-matchExpressions-key") ##### `operator` string [​](#fromHost-deviceClasses-selector-matchExpressions-operator "Direct link to fromHost-deviceClasses-selector-matchExpressions-operator") ##### `values` string\[] [​](#fromHost-deviceClasses-selector-matchExpressions-values "Direct link to fromHost-deviceClasses-selector-matchExpressions-values") --- # ConfigMaps Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is turned off. No need to configure RBAC vCluster automatically adds the required cluster RBAC permissions for retrieving the ConfigMaps and syncing the resources from the control plane cluster to the tenant cluster. Enabling sync from host allows you to sync ConfigMaps from the specified namespaces in the control plane cluster to the specified namespaces in a tenant cluster. Here is how the required configuration in `vcluster.yaml` looks like: configure ConfigMap sync from host ``` sync: fromHost: configMaps: enabled: true mappings: byName: # syncs all ConfigMaps from "foo" namespace # to the "bar" namespace in a virtual cluster. ConfigMaps names are unchanged. "foo/*": "bar/*" ``` Here are a few things to remember when configuring the sync: * It is also possible to modify the name of the synced resource in the tenant cluster. * There is no option to sync from all namespaces in the control plane cluster. * Sync is one-directional, from the control plane cluster to the tenant cluster. If you modify an object in the control plane cluster, vCluster syncs the change to virtual object. * When you delete a virtual object, vCluster re-creates it if the host object still exist. * When you delete a host object, vCluster deletes the corresponding virtual object. * It is not possible to sync ConfigMaps that were already synced from virtual to host, they are skipped by vCluster. Namespaces in the tenant cluster are created automatically during the sync (if they do not exist already). kube-root-ca ConfigMap is skipped ConfigMaps named `kube-root-ca.crt` are skipped in the from host sync (even if they match the mappings specified in `vcluster.yaml`). You can use synced ConfigMaps in your workloads as an environment variables source or a volume. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Run the command `kubectl auth can-i create clusterrole -A` to verify that your current kube-context has administrative privileges. info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm`: Helm v3.10 is required for deploying the platform. Refer to the [Helm Installation Guide](https://Helm.sh/docs/intro/install/) if you need to install it. * `kubectl`: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. All the specified namespaces have to exist in the host at the vCluster startup. ## Sync all ConfigMaps from host namespace[​](#sync-all-configmaps-from-host-namespace "Direct link to Sync all ConfigMaps from host namespace") To sync all ConfigMaps from a given namespace in the control plane cluster to the given namespace in the tenant cluster, use “namespace/\*” wildcard, e.g.: configure ConfigMap sync from host namespace ``` sync: fromHost: configMaps: enabled: true mappings: byName: # syncs all ConfigMaps from "foo" namespace # to the "bar" namespace in a virtual cluster. ConfigMaps names are unchanged. "foo/*": "bar/*" ``` ## Sync only specific ConfigMaps from host namespace[​](#sync-only-specific-configmaps-from-host-namespace "Direct link to Sync only specific ConfigMaps from host namespace") To sync only specific ConfigMaps from namespaces, you need to provide `namespace/name` as the key and value: configure ConfigMap sync from host for one object ``` sync: fromHost: configMaps: enabled: true mappings: byName: # syncs ConfigMap named "cm-name" from "foo" host namespace # to the "bar" namespace in virtual. "foo/cm-name": "bar/cm-name" ``` ## Sync all ConfigMaps from the tenant cluster’s control plane cluster namespace[​](#sync-all-configmaps-from-the-tenant-clusters-control-plane-cluster-namespace "Direct link to Sync all ConfigMaps from the tenant cluster’s control plane cluster namespace") There is also a handy syntax to sync all ConfigMaps from the tenant cluster’s own control plane cluster namespace to the tenant namespace. As the tenant cluster’s namespace is not always known upfront (e.g. when the tenant cluster is created by the platform), `””` (empty string) is treated as “tenant cluster’s own control plane cluster namespace”. configure ConfigMap sync from host for tenant cluster's namespace ``` sync: fromHost: configMaps: enabled: true mappings: byName: # syncs all ConfigMaps from virtual cluster's host namespace # to "my-virtual" namespace in a virtual cluster. "": "my-virtual" ``` ## Sync specific ConfigMap from the tenant cluster’s control plane cluster namespace[​](#sync-specific-configmap-from-the-tenant-clusters-control-plane-cluster-namespace "Direct link to Sync specific ConfigMap from the tenant cluster’s control plane cluster namespace") you can also specify only a few ConfigMaps from the tenant cluster’s own control plane cluster namespace this way: configure ConfigMap sync from host for objects in tenant cluster's namespace ``` sync: fromHost: configMaps: enabled: true mappings: byName: # syncs ConfigMap named "my-cm" from virtual cluster's host namespace # to "my-virtual-namespace" in a virtual cluster. "/my-cm": "my-virtual/my-cm" ``` ## Modify synced ConfigMap namespace and name in the tenant cluster[​](#modify-synced-configmap-namespace-and-name-in-the-tenant-cluster "Direct link to Modify synced ConfigMap namespace and name in the tenant cluster") It’s also possible to modify ConfigMap name during the sync: configure ConfigMap sync from host and modify name and namespace ``` sync: fromHost: configMaps: enabled: true mappings: byName: # syncs "config" ConfigMap from "system" namespace in the host # as "my-config" in "my-virtual" namespace in a virtual cluster. "system/config": "my-virtual/my-config" ``` ## Patches[​](#patches "Direct link to Patches") Use `sync.fromHost.configMaps.patches` to transform ConfigMap fields while syncing from the control plane cluster to the tenant cluster. For from-host sync, use `reverseExpression` for values that should appear in the tenant cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax and directionality. configure ConfigMap sync from host with patches ``` sync: fromHost: configMaps: enabled: true mappings: byName: "default/my-cm": "barfoo2/cm-my" patches: - path: metadata.annotations[*] # optional reverseExpression to reverse the change from the host cluster reverseExpression: "value.startsWith('www.') ? value.slice(4) : value" ``` ## From host config map sync example[​](#from-host-config-map-sync-example "Direct link to From host config map sync example") This guide shows how to sync Kubernetes config maps from control plane clusters and how you can use them in your workload running inside tenant clusters. ### Set up cluster contexts[​](#set-up-cluster-contexts "Direct link to Set up cluster contexts") Setting up the control plane and tenant cluster contexts makes it easier to switch between them. set up kubectl contexts ``` export HOST_CTX="your-host-context" export VCLUSTER_CTX="vcluster-ctx" ``` then, create a namespace in your control plane cluster, use `foobar2` as an example: create namespace ``` kubectl --context="${HOST_CTX}" create namespace foobar2 ``` tip You can find your contexts by running `kubectl config get-contexts` ### Enable from host syncing for ConfigMap[​](#enable-from-host-syncing-for-configmap "Direct link to Enable from host syncing for ConfigMap") Enable the from host syncing for ConfigMap in your tenant cluster configuration: Enable from host syncing for a ConfigMap ``` sync: fromHost: configMaps: enabled: true mappings: byName: "foobar2/config": "my-namespace/config" ``` This configuration: * Enables from host syncing of the ConfigMap named `config` in the namespace `foobar2`. * Automatically configures RBAC permissions for vCluster, so it can access this ConfigMap (you need to re-deploy vCluster for it to take place) * Makes this ConfigMap accessible as `config` in the `my-namespace` in your vCluster. create tenant cluster Create or update a tenant cluster following the [vCluster quick start guide](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md). ### Sync ConfigMap to tenant cluster and use it in pod[​](#sync-configmap-to-tenant-cluster-and-use-it-in-pod "Direct link to Sync ConfigMap to tenant cluster and use it in pod") 1. First, you create a ConfigMap that you want to sync in the control plane cluster: Copy this file and save it locally as `config.json` ``` { "name": "my-config", "hosts": ["123.456.789", "987.654.321"] } ``` then, create a ConfigMap containing this file in the control plane cluster: Create ConfigMap in the host ``` kubectl --context="${HOST_CTX}" create configmap config \ --namespace=foobar2 \ --from-file=config.json=config.json ``` 2. #### Ensure that ConfigMap got synced to the tenant cluster[​](#ensure-that-configmap-got-synced-to-the-tenant-cluster "Direct link to Ensure that ConfigMap got synced to the tenant cluster") Your ConfigMap should be now accessible in the tenant cluster. Keep in mind, that any edit made in the virtual object is overwritten by the host object data. Check ConfigMap in the tenant cluster: Get synced ConfigMap ``` kubectl --context="${VCLUSTER_CTX}" get configmap --namespace my-namespace config -o yaml ``` you should see similar output: ConfigMap contents ``` apiVersion: v1 data: config.json: | { "name": "my-config", "hosts": ["123.456.789", "987.654.321"] } kind: ConfigMap metadata: creationTimestamp: "2025-02-17T12:43:17Z" name: config namespace: my-namespace resourceVersion: "18279" uid: bb131bf9-8fed-4d34-904c-1f83ed05aa72 ``` 3. #### Use it in your workload[​](#use-it-in-your-workload "Direct link to Use it in your workload") Now, you can create a new pod in the `my-namespace` namespace and specify this ConfigMap as a volume and mount it in the container. Save this pod locally to the file called `pod.yaml`: pod.yaml ``` apiVersion: v1 kind: Pod metadata: name: my-pod namespace: my-virtual-namespace spec: containers: - name: busybox image: busybox command: - sleep - "inf" volumeMounts: - name: my-config mountPath: /tmp/ volumes: - name: my-config configMap: name: config ``` then, create it in the tenant cluster: Create pod ``` kubectl --context="${VCLUSTER_CTX}" create -f pod.yaml ``` 4. #### Verify that config is mounted[​](#verify-that-config-is-mounted "Direct link to Verify that config is mounted") Tenant Cluster Wait for pod running Wait for pod running ``` kubectl --context="${VCLUSTER_CTX}" wait --for=condition=ready pod my-pod --namespace my-virtual-namespace --timeout=300s ``` once it is running, you can check if your file is accessible in the container under `/tmp/config.json` path. Check mounted file ``` kubectl --context="${VCLUSTER_CTX}" exec -it --namespace my-virtual-namespace my-pod -- cat /tmp/config.json ``` you should see environment successfully injected from the ConfigMap: Config from ConfigMap ``` { "name": "my-config", "hosts": ["123.456.789", "987.654.321"] } ``` 5. #### Summary[​](#summary "Direct link to Summary") From host ConfigMap syncing allow you to make specific ConfigMap(s) from control plane clusters accessible inside your tenant clusters. You can make them accessible from different namespaces and/or with different names in the tenant cluster. They are always synced from the control plane to the tenant cluster, so it is also possible to sync one control plane cluster ConfigMap to multiple tenant cluster ones. They can also be used as a volume or env source in your workloads. ## Config reference[​](#config-reference "Direct link to Config reference") ## `configMaps` object [​](#configMaps "Direct link to configMaps")ConfigMaps defines if config maps in the host should get synced to the virtual cluster. ### `enabled` boolean false [​](#configMaps-enabled "Direct link to configMaps-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#configMaps-patches "Direct link to configMaps-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#configMaps-patches-path "Direct link to configMaps-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#configMaps-patches-expression "Direct link to configMaps-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#configMaps-patches-reverseExpression "Direct link to configMaps-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#configMaps-patches-reference "Direct link to configMaps-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#configMaps-patches-reference-apiVersion "Direct link to configMaps-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#configMaps-patches-reference-apiVersionPath "Direct link to configMaps-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#configMaps-patches-reference-kind "Direct link to configMaps-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#configMaps-patches-reference-kindPath "Direct link to configMaps-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#configMaps-patches-reference-namePath "Direct link to configMaps-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#configMaps-patches-reference-namespacePath "Direct link to configMaps-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#configMaps-patches-labels "Direct link to configMaps-patches-labels")Labels treats the path value as a labels selector. ### `mappings` object [​](#configMaps-mappings "Direct link to configMaps-mappings")Mappings for Namespace and Object #### `byName` object {} [​](#configMaps-mappings-byName "Direct link to configMaps-mappings-byName")ByName is a map of control-plane-object-namespace/control-plane-object-name: tenant-object-namespace/tenant-object-name. There are several wildcards supported:1) To match all objects in a control plane namespace and sync them to a different namespace in the tenant cluster: byName: "foo/*": "foo-in-virtual/*" 2) To match a specific object in the control plane namespace and sync it to the same namespace with the same name: byName: "foo/my-object": "foo/my-object" 3) To match a specific object in the control plane namespace and sync it to the same namespace with a different name: byName: "foo/my-object": "foo/my-virtual-object" 4) To match all objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "": "my-virtual-namespace/\*" 5) To match specific objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "/my-object": "my-virtual-namespace/my-object" --- # CSI drivers Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Sync [CSIDriver](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/) resources from the control plane cluster to the tenant cluster. Use this in conjunction with the virtual scheduler inside of vCluster for more advanced scheduling needs. Syncing the CSIDriver resources from the control plane cluster helps the virtual scheduler make decisions about which node to schedule pods to. ## Default value[​](#default-value "Direct link to Default value") Default: `enabled: auto`. ## Sync CSIDrivers from the control plane cluster to the tenant cluster[​](#sync-csidrivers-from-the-control-plane-cluster-to-the-tenant-cluster "Direct link to Sync CSIDrivers from the control plane cluster to the tenant cluster") Configure `enabled: true` if you aren't using the virtual scheduler but want to sync CSIDriver resources. ``` sync: fromHost: csiDrivers: enabled: true ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `csiDrivers` object [​](#csiDrivers "Direct link to csiDrivers")CSIDrivers defines if csi drivers should get synced from the host cluster to the virtual cluster, but not back. If auto, is automatically enabled when the virtual scheduler is enabled. ### `enabled` string|boolean auto [​](#csiDrivers-enabled "Direct link to csiDrivers-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#csiDrivers-patches "Direct link to csiDrivers-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#csiDrivers-patches-path "Direct link to csiDrivers-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#csiDrivers-patches-expression "Direct link to csiDrivers-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#csiDrivers-patches-reverseExpression "Direct link to csiDrivers-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#csiDrivers-patches-reference "Direct link to csiDrivers-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#csiDrivers-patches-reference-apiVersion "Direct link to csiDrivers-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#csiDrivers-patches-reference-apiVersionPath "Direct link to csiDrivers-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#csiDrivers-patches-reference-kind "Direct link to csiDrivers-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#csiDrivers-patches-reference-kindPath "Direct link to csiDrivers-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#csiDrivers-patches-reference-namePath "Direct link to csiDrivers-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#csiDrivers-patches-reference-namespacePath "Direct link to csiDrivers-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#csiDrivers-patches-labels "Direct link to csiDrivers-patches-labels")Labels treats the path value as a labels selector. --- # CSI nodes Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Sync [CSINode](https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/csi-node-v1/) resources from the control plane cluster to the tenant cluster. Use this in conjunction with the [virtual scheduler](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/virtual-scheduler.md) inside of vCluster for more advanced scheduling needs. Syncing the CSINode resources from the control plane cluster helps the virtual scheduler make decisions about which node to schedule pods to. ## Default value[​](#default-value "Direct link to Default value") Default: `enabled: auto`. ## Sync CSINodes from the control plane cluster to the tenant cluster[​](#sync-csinodes-from-the-control-plane-cluster-to-the-tenant-cluster "Direct link to Sync CSINodes from the control plane cluster to the tenant cluster") Configure `enabled: true` if you aren't using the virtual scheduler but want to sync CSINode resources. ``` sync: fromHost: csiNodes: enabled: true ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `csiNodes` object [​](#csiNodes "Direct link to csiNodes")CSINodes defines if csi nodes should get synced from the host cluster to the virtual cluster, but not back. If auto, is automatically enabled when the virtual scheduler is enabled. ### `enabled` string|boolean auto [​](#csiNodes-enabled "Direct link to csiNodes-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#csiNodes-patches "Direct link to csiNodes-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#csiNodes-patches-path "Direct link to csiNodes-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#csiNodes-patches-expression "Direct link to csiNodes-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#csiNodes-patches-reverseExpression "Direct link to csiNodes-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#csiNodes-patches-reference "Direct link to csiNodes-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#csiNodes-patches-reference-apiVersion "Direct link to csiNodes-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#csiNodes-patches-reference-apiVersionPath "Direct link to csiNodes-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#csiNodes-patches-reference-kind "Direct link to csiNodes-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#csiNodes-patches-reference-kindPath "Direct link to csiNodes-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#csiNodes-patches-reference-namePath "Direct link to csiNodes-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#csiNodes-patches-reference-namespacePath "Direct link to csiNodes-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#csiNodes-patches-labels "Direct link to csiNodes-patches-labels")Labels treats the path value as a labels selector. --- # CSI storage capacities Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Sync [CSIStorageCapacity](https://kubernetes.io/docs/concepts/storage/storage-capacity/) resources from the control plane cluster to the tenant cluster if the `.nodeTopology` matches a synced node. Use this in conjunction with the virtual scheduler inside of vCluster for more advanced scheduling needs. Syncing the CSIStorageCapacity resources from the control plane cluster helps the virtual scheduler make decisions about which node to schedule pods to. CSIStorageCapacity is a namespaced resource. When synchronizing from the control plane cluster to the tenant cluster, all CSIStorageCapacity objects are synced into the `kube-system` namespace of the tenant cluster. During this process, the object name is transformed to the format `-x-`, where `` is the original object name in the control plane cluster and `` is the namespace in which the object originally existed in the control plane cluster. When synchronizing from the tenant cluster back to the control plane cluster, a CSIStorageCapacity object is created in the control plane cluster only if it includes the physical location annotations normally added by the syncer. Specifically: * `vcluster.loft.sh/object-name` specifies the object name in the control plane cluster. * `vcluster.loft.sh/object-namespace` specifies the control plane cluster namespace. If these annotations are not present, the CSIStorageCapacity object will not be created or updated in the control plane cluster. Physical annotations of CSIStorageCapacity ``` apiVersion: storage.k8s.io/v1 kind: CSIStorageCapacity metadata: name: fast-ssd-x-storage-v namespace: kube-system annotations: vcluster.loft.sh/object-name: fast-ssd vcluster.loft.sh/object-namespace: storage-v # optional bookkeeping: # vcluster.loft.sh/object-uid: "c1a2b3c4-..." # vcluster.loft.sh/object-kind: "storage.k8s.io/v1, Kind=CSIStorageCapacity" ``` ## Default value[​](#default-value "Direct link to Default value") Default: `enabled: auto`. ## Sync CSIStorageCapacities from the control plane cluster to the tenant cluster[​](#sync-csistoragecapacities-from-the-control-plane-cluster-to-the-tenant-cluster "Direct link to Sync CSIStorageCapacities from the control plane cluster to the tenant cluster") Configure `enabled: true` if you aren't using the virtual scheduler but want to sync CSIStorageCapacity resources. ``` sync: fromHost: csiStorageCapacities: enabled: true ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `csiStorageCapacities` object [​](#csiStorageCapacities "Direct link to csiStorageCapacities")CSIStorageCapacities defines if csi storage capacities should get synced from the host cluster to the virtual cluster, but not back. If auto, is automatically enabled when the virtual scheduler is enabled. ### `enabled` string|boolean auto [​](#csiStorageCapacities-enabled "Direct link to csiStorageCapacities-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#csiStorageCapacities-patches "Direct link to csiStorageCapacities-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#csiStorageCapacities-patches-path "Direct link to csiStorageCapacities-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#csiStorageCapacities-patches-expression "Direct link to csiStorageCapacities-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#csiStorageCapacities-patches-reverseExpression "Direct link to csiStorageCapacities-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#csiStorageCapacities-patches-reference "Direct link to csiStorageCapacities-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#csiStorageCapacities-patches-reference-apiVersion "Direct link to csiStorageCapacities-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#csiStorageCapacities-patches-reference-apiVersionPath "Direct link to csiStorageCapacities-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#csiStorageCapacities-patches-reference-kind "Direct link to csiStorageCapacities-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#csiStorageCapacities-patches-reference-kindPath "Direct link to csiStorageCapacities-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#csiStorageCapacities-patches-reference-namePath "Direct link to csiStorageCapacities-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#csiStorageCapacities-patches-reference-namespacePath "Direct link to csiStorageCapacities-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#csiStorageCapacities-patches-labels "Direct link to csiStorageCapacities-patches-labels")Labels treats the path value as a labels selector. --- # Custom resources | | | Enterprise | | | | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Custom Resource Syncing](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/custom-resources.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** vCluster allows you to sync custom resources from the control plane cluster to the tenant cluster. This feature creates read-only copies of custom resources inside your tenant cluster, making them available to applications and users without giving them direct access to the control plane cluster. When you enable custom resource syncing, vCluster first makes the resource type available in the tenant cluster. It copies the control plane cluster's CustomResourceDefinition (CRD) for a CRD-backed resource. For a resource served by an extension [API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/), vCluster generates a schemaless CRD instead. It then begins syncing the resource instances. This is particularly useful for sharing cluster-wide resources or `ClusterStores` from [external-secrets](https://external-secrets.io/latest/) with tenant workloads. For details, see [Aggregated API resources](#aggregated-api-resources). The synced resources are read-only in the tenant cluster. Changes in the control plane cluster automatically update the tenant copies. Tenant changes never propagate to the control plane cluster. vCluster removes tenant-created objects and restores tenant-deleted objects. Tenant field updates remain local until the same fields change in the control plane cluster. If you need to sync resources from the tenant cluster to the control plane cluster instead, see [syncing custom resources to the control plane cluster](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/custom-resources.md). For help choosing a direction, and for lifecycle and troubleshooting guidance, see [Manage custom resources](https://vcluster.com/docs/vcluster/manage/custom-resources.md). For a full worked example, see [Sync a namespaced custom resource from the control plane cluster](https://vcluster.com/docs/vcluster/learn-how-to/sync-custom-resource-from-host.md). Automatic RBAC for configured resources vCluster automatically adds the required cluster RBAC permissions for retrieving the CustomResourceDefinition and syncing the configured resources. Extension API servers can require additional permissions for related resources. See [Authorization on the control plane cluster](#authorization-on-the-control-plane-cluster). ## Cluster-scoped example[​](#cluster-scoped-example "Direct link to Cluster-scoped example") Cluster-scoped custom resources exist at the cluster level and are not tied to any specific namespace. These are the simplest to sync because they don't require namespace mapping. To sync cluster-scoped custom resources from the control plane cluster, configure the resource in your `vcluster.yaml` file: configure cluster-scoped custom resource sync from host ``` sync: fromHost: customResources: customobjects.example.io: enabled: true scope: Cluster ``` By default, vCluster does not require you to specify an API version when syncing custom resources. When a CRD has multiple versions and you don't specify a version, vCluster uses the storage version. If you want to sync a specific version of a CRD that is not the storage version, you can explicitly specify it in the sync configuration. ## Aggregated API resources[​](#aggregated-api-resources "Direct link to Aggregated API resources") Not every extension type in a Kubernetes cluster is backed by a CustomResourceDefinition. Kubernetes can also serve types through [API aggregation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/), where an extension API server registered as an APIService owns the type instead of a CRD. vCluster syncs these resources through the same `customResources` configuration. You don't declare that a resource comes from an extension API server. vCluster resolves every configured resource against the control plane cluster on startup: * If the control plane cluster has a CRD for the resource, vCluster copies that CRD into the tenant cluster. * If it doesn't, vCluster looks the resource up in the control plane cluster's discovery data. When an extension API server serves the resource, vCluster generates a schemaless CRD for it in the tenant cluster. * If neither a CRD nor an extension API server serves the resource, the tenant cluster fails to start. This resolution happens only on startup. If the extension API server isn't serving the resource then, vCluster fails to start and tries again when the control plane pod restarts. An extension API server that appears or recovers later is picked up on that next restart. Because `kubectl get crds` doesn't list resources that an extension API server serves, use `kubectl api-resources` to find them. Configure aggregated API resource sync from host ``` sync: fromHost: customResources: customobjects.example.io: enabled: true scope: Cluster ``` ### Generated CRDs for aggregated API resources[​](#generated-crds-for-aggregated-api-resources "Direct link to Generated CRDs for aggregated API resources") vCluster builds the generated CRD from the control plane cluster's discovery data. The kind, plural name, scope, and presence of a `status` subresource come from the control plane cluster. vCluster derives the singular name by lowercasing the kind and derives the list kind by appending `List`. It marks the CRD with the `vcluster.loft.sh/generated-from: aggregated-apiserver` annotation and creates or updates it when the tenant cluster starts. Objects of a generated CRD live in the tenant cluster's own data store, like any other custom resource, so [snapshots](https://vcluster.com/docs/vcluster/key-features/snapshots.md) include them. The generated CRD has no resource-specific OpenAPI schema and preserves unknown fields. The tenant API server doesn't validate custom fields against the extension API server's schema. Keep the following behaviors in mind: * `kubectl explain` returns no field documentation for the resource inside the tenant cluster. * The tenant cluster doesn't prune unknown fields. * When the resource key includes no version, vCluster uses the API group's preferred version on the control plane cluster. For CRD-backed resources, it uses the storage version instead. * When you change the version in the resource key, vCluster serves the new version and retains previously stored versions without serving them. vCluster uses the `vcluster.loft.sh/generated-from: aggregated-apiserver` annotation as the adoption marker. If an existing CRD has this marker, vCluster adopts it and updates its specification. If a CRD of the same name doesn't have this marker, the tenant cluster fails to start. Delete that CRD or remove the resource from `customResources`. Removing an entry from `customResources`, or setting `enabled: false`, stops the syncer. vCluster doesn't delete the generated CRD, its objects, or the objects it created on the control plane cluster. The tenant cluster keeps serving the resource, and writes to it stay tenant-local because nothing validates or syncs them. ### Authorization on the control plane cluster[​](#authorization-on-the-control-plane-cluster "Direct link to Authorization on the control plane cluster") vCluster grants itself the permissions it needs for every resource listed in `customResources`. Some extension API servers authorize a request against a related resource that you never configure for sync. For example, an API server can check access to a grouping resource before it serves or accepts an object. Grant those permissions explicitly. This example grants access to a related resource in the vCluster control plane namespace: Grant extra permissions for an extension API server ``` rbac: role: extraRules: - apiGroups: ["example.io"] resources: ["customobjectgroups"] verbs: ["get", "list", "watch"] ``` Use `rbac.role.extraRules` for resources in the vCluster control plane namespace. Use `rbac.clusterRole.extraRules` for cluster-scoped resources or namespaced resources in other control plane cluster namespaces. For details, see [Add custom RBAC rules](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/rbac.md#add-custom-rbac-rules). ### Read-only sync of aggregated API resources[​](#read-only-sync-of-aggregated-api-resources "Direct link to Read-only sync of aggregated API resources") vCluster generates a CRD in the tenant cluster for cluster-scoped and namespace-scoped aggregated API resources alike. Namespace-scoped resources still require `mappings.byName`, the same as CRD-backed ones. The synced copies are read-only in the tenant cluster. To let tenant workloads create and modify a resource that an extension API server serves, sync it in the other direction with [syncing custom resources to the control plane cluster](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/custom-resources.md). ### Limitations[​](#limitations "Direct link to Limitations") The Kubernetes API accepts standard CRD operations for an aggregated API resource, but from-host reconciliation remains read-only. vCluster removes tenant-created objects and restores tenant-deleted objects. Tenant field updates remain local until the same fields change in the control plane cluster. The generated CRD serves the resource itself, plus the `status` subresource when the control plane cluster serves it. Extension API servers can expose additional subresources, such as `scale`, `logs`, or `exec`. Those aren't available inside the tenant cluster, because a CRD can't serve them. ## Versioned example[​](#versioned-example "Direct link to Versioned example") To specify a particular version of a CustomResourceDefinition (CRD), append the version to the resource type, separated by a `/`, using the format `/` in your configuration. The specified version must exist in the control plane cluster when vCluster starts. This version becomes the storage version within the vCluster. When the storage version in the vCluster differs from the control plane cluster's storage version, vCluster automatically converts custom resources between the two versions. For example, to use version `v1` of the `certificaterequests.cert-manager.io` CRD, specify: Configure cluster-scoped CR sync from host ``` sync: fromHost: customResources: certificaterequests.cert-manager.io/v1: enabled: true scope: Cluster ``` Unique CRD versions vCluster supports syncing only one version of a custom resource. If you specify multiple versions, the tenant cluster fails to start. ### Upgrade custom resources with explicit CRD API versions[​](#upgrade-custom-resources-with-explicit-crd-api-versions "Direct link to Upgrade custom resources with explicit CRD API versions") vCluster supports syncing only one version of a CRD. If multiple versions are specified in the sync configuration, the tenant cluster fails to start. vCluster modifies the CRD inside the tenant cluster during upgrades if it detects that a required version is missing. The following scenarios describe how vCluster handles CRD versions when you upgrade the tenant cluster or change the sync configuration. #### Upgrade from no version to specified version[​](#upgrade-from-no-version-to-specified-version "Direct link to Upgrade from no version to specified version") When upgrading from an unversioned CRD to a versioned one (`certificaterequests.cert-manager.io` → `certificaterequests.cert-manager.io/v1`), vCluster checks the CRD's storage version. If the storage version is not `v1`, vCluster updates the CRD to include `v1` as a new version in addition to the current storage version. #### Upgrade from one specified version to another specified version[​](#upgrade-from-one-specified-version-to-another-specified-version "Direct link to Upgrade from one specified version to another specified version") When upgrading between versioned CRDs (`certificaterequests.cert-manager.io/v1` → `certificaterequests.cert-manager.io/v2`), the CRD is updated to include the new version `v2`, and the old version `v1` is kept as well. #### Upgrade from a specified version to no version specified[​](#upgrade-from-a-specified-version-to-no-version-specified "Direct link to Upgrade from a specified version to no version specified") When upgrading from a versioned CRD to an unversioned one (`certificaterequests.cert-manager.io/v1` → `certificaterequests.cert-manager.io`), the behavior depends on the host's storage version: * If the CRD's storage version in the control plane cluster is `v1`, nothing happens and the CRD in the tenant cluster remains as `certificaterequests.cert-manager.io/v1`. * If the storage version is not `v1`, vCluster updates the CRD to include the storage version as a new version in addition to `v1`. ## Namespace-scoped custom resources[​](#namespace-scoped-custom-resources "Direct link to Namespace-scoped custom resources") By default, namespace-scoped custom resource syncing is disabled. Enabling this feature allows you to sync namespaced custom resources from specific namespaces in the control plane cluster to corresponding namespaces in the tenant cluster. Configure namespace-scoped custom resource sync from host ``` sync: fromHost: customResources: customobjects.example.io: enabled: true scope: Namespaced mappings: byName: # syncs all CustomObjects from "foo" namespace # to the "bar" namespace in a virtual cluster. CustomObjects names are unchanged. "foo/*": "bar/*" ``` Key behaviors for namespace scoped syncing: * You can modify the name of the synced resource in the tenant cluster. * You cannot sync from all namespaces in the control plane cluster. * Sync is one-directional, from the control plane cluster to the tenant cluster. If you modify an object in the control plane cluster, vCluster syncs the change to the tenant object. * When you delete a tenant object, vCluster re-creates it if the control plane cluster object still exists. * When you delete a control plane cluster object, vCluster deletes the tenant object. vCluster creates namespaces in the tenant cluster automatically during the sync if they don't already exist. ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Ensure you have the following: * All specified namespaces must exist in the control plane cluster when vCluster starts. * The control plane cluster must serve the resource when vCluster starts, either through a CustomResourceDefinition or through an extension API server. Aggregated API resources Namespace-scoped resources served by an extension API server sync the same way, including the `mappings.byName` requirement. See [Aggregated API resources](#aggregated-api-resources) for the generated CRD and its limitations. ### Example[​](#example "Direct link to Example") For namespace scoped custom resources, you must specify `mappings.byName` in the config. This tells vCluster which control plane cluster resources should be synced and where to place them in the tenant cluster. vCluster can't sync custom resources that were already synced from the tenant cluster to the control plane cluster. It skips these resources. The following is an example with a generic custom resource. To sync all custom resources from a given namespace in the control plane cluster to a given namespace in the tenant cluster, use the "namespace/\*" wildcard: Configure custom resource sync from host namespace ``` sync: fromHost: customResources: customobjects.example.io: enabled: true scope: Namespaced mappings: byName: # syncs all CustomObjects from "foo" namespace # to the "bar" namespace in a virtual cluster. CustomObjects names are unchanged. "foo/*": "bar/*" ``` To sync only specific custom resources from namespaces, provide `namespace/name` as the key and value: Configure custom resource sync from host for one object ``` sync: fromHost: customResources: customobjects.example.io: enabled: true scope: Namespaced mappings: byName: # syncs CustomObject named "my-object" from "foo" host namespace # to the "bar" namespace in virtual. "foo/my-object": "bar/my-object" ``` There is also a syntax to sync all custom resources from the tenant cluster's own control plane cluster namespace to the tenant namespace. Since the tenant cluster's namespace is not always known upfront (for example, when vCluster is created by the platform), `""` (empty string) is treated as "tenant cluster's own control plane cluster namespace": configure custom resource sync from host for tenant cluster's namespace ``` sync: fromHost: customResources: customobjects.example.io: enabled: true scope: Namespaced mappings: byName: # syncs all CustomObjects from virtual cluster's host namespace # to "my-virtual" namespace in a virtual cluster. "": "my-virtual" ``` You can also specify only a few custom resources from the tenant cluster's own control plane cluster namespace: configure custom resource sync from host for objects in tenant cluster's namespace ``` sync: fromHost: customResources: customobjects.example.io: enabled: true scope: Namespaced mappings: byName: # syncs CustomObject named "my-object" from virtual cluster's host namespace # to "my-virtual-namespace" in a virtual cluster. "/my-object": "my-virtual/my-object" ``` It's also possible to modify the custom resource name during the sync: configure custom resource sync from host and modify name and namespace ``` sync: fromHost: customResources: customobjects.example.io: enabled: true scope: Namespaced mappings: byName: # syncs "foo" CustomObject from "system" namespace in the host # as "my-foo" in "my-virtual" namespace in a virtual cluster. "system/foo": "my-virtual/my-foo" ``` ### Patches[​](#patches "Direct link to Patches") Use `sync.fromHost.customResources..patches` to transform custom resource fields while syncing from the control plane cluster to the tenant cluster. For from-host sync, use `reverseExpression` for values that should appear in the tenant cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and custom resource labels patches. configure custom resource sync from host with patches ``` sync: fromHost: customResources: customobjects.example.io: enabled: true scope: Namespaced mappings: byName: "default/my-object": "barfoo2/object-my" patches: - path: metadata.annotations[*] # optional reverseExpression to reverse the change from the host cluster reverseExpression: "value.startsWith('www.') ? value.slice(4) : value" ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `customResources` {key: object} [​](#customResources "Direct link to customResources")CustomResources defines what custom resources should get synced read-only to the virtual cluster from the host cluster. vCluster will automatically add any required RBAC to the vCluster cluster role. ### `enabled` required boolean [​](#customResources-enabled "Direct link to customResources-enabled")Enabled defines if this option should be enabled. ### `scope` required string [​](#customResources-scope "Direct link to customResources-scope")Scope defines the scope of the resource ### `patches` object\[] [​](#customResources-patches "Direct link to customResources-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#customResources-patches-path "Direct link to customResources-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#customResources-patches-expression "Direct link to customResources-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#customResources-patches-reverseExpression "Direct link to customResources-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#customResources-patches-reference "Direct link to customResources-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#customResources-patches-reference-apiVersion "Direct link to customResources-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#customResources-patches-reference-apiVersionPath "Direct link to customResources-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#customResources-patches-reference-kind "Direct link to customResources-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#customResources-patches-reference-kindPath "Direct link to customResources-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#customResources-patches-reference-namePath "Direct link to customResources-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#customResources-patches-reference-namespacePath "Direct link to customResources-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#customResources-patches-labels "Direct link to customResources-patches-labels")Labels treats the path value as a labels selector. ### `mappings` object [​](#customResources-mappings "Direct link to customResources-mappings")Mappings for Namespace and Object #### `byName` object [​](#customResources-mappings-byName "Direct link to customResources-mappings-byName")ByName is a map of control-plane-object-namespace/control-plane-object-name: tenant-object-namespace/tenant-object-name. There are several wildcards supported:1) To match all objects in a control plane namespace and sync them to a different namespace in the tenant cluster: byName: "foo/*": "foo-in-virtual/*" 2) To match a specific object in the control plane namespace and sync it to the same namespace with the same name: byName: "foo/my-object": "foo/my-object" 3) To match a specific object in the control plane namespace and sync it to the same namespace with a different name: byName: "foo/my-object": "foo/my-virtual-object" 4) To match all objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "": "my-virtual-namespace/\*" 5) To match specific objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "/my-object": "my-virtual-namespace/my-object" --- # Device classes | | | Enterprise | | | | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [DRA Sync](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md#dynamic-resource-allocation-and-device-classes) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. DeviceClass syncing allows tenant clusters to use device classes from the control plane cluster. With DeviceClass syncing enabled, tenant clusters can reference DeviceClass resources from the control plane cluster in [resourceclaims](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaims.md) and [resourceclaimtemplates](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaimtemplates.md). You can enable this feature to sync DeviceClass resources from the control plane cluster to the tenant cluster. Add labels to DeviceClass resources in your control plane cluster and configure vCluster to sync only those that match specific label selectors. When you create a ResourceClaims or ResourceClaimTemplates resource in the tenant cluster that references a synced DeviceClass, the control plane cluster's resourceclaim controller handles the resource claims allocation. This saves resources since you don't need resourceclaim controllers in every tenant cluster. It also gives you centralized control over which DeviceClass teams can access. Selector scope on shared nodes The selector controls which `DeviceClass` pool a tenant can draw from. It doesn't control which specific device within an allowed class a tenant receives. It also doesn't stop a synced node from disclosing its GPU capacity and allocatable totals to other tenants on that node. Those are schedulable inventory counts, not another tenant's real-time usage. See [GPU and device isolation](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#gpu-and-device-isolation) in the shared-node hardening guide. This approach is useful in scenarios where selective access to DeviceClass configurations is required. Common use cases include: * **Development environments**: Sync only the DeviceClass resources required for development clusters to reduce noise and avoid unnecessary exposure. * **Tenant isolation**: Enable teams to use their own DeviceClass resources while sharing a single control plane cluster. * **Security**: Restrict the DeviceClass resources available in the tenant cluster to enforce access control and prevent unintended configurations. ## Enable syncing[​](#enable-syncing "Direct link to Enable syncing") When enabled, vCluster takes control of all DeviceClass resources in the tenant cluster. It only allows DeviceClass resources that are synced from the control plane cluster to exist. This example configuration enables syncing **all** DeviceClass resources from the control plane cluster to the tenant cluster: ``` sync: fromHost: deviceClasses: enabled: true ``` If you try to create the DeviceClass resource directly in the tenant cluster, using for example `kubectl create` or `kubectl apply`, vCluster detects it and deletes it immediately. This prevents conflicts between locally created DeviceClass resources and those synced from the control plane cluster, ensuring that only DeviceClass resources from the control plane cluster exist in the tenant cluster. DeviceClass resource limitation When enabled, DeviceClass resource creation can only occur in the control plane cluster. Any DeviceClass resources created in the tenant cluster will be deleted. ## How syncing works[​](#how-syncing-works "Direct link to How syncing works") When DeviceClass syncing is enabled, vCluster can use a label selector to control which DeviceClass, ResourceClaims and ResourceClaimTemplates resources are synchronized between the control plane cluster and tenant clusters. This affects the resource types with separate unidirectional sync flows: * **DeviceClass resources (control plane → tenant)**: vCluster syncs DeviceClass resources from the control plane cluster to the tenant cluster. You cannot create DeviceClass resources directly in the tenant cluster. If a selector is defined, only DeviceClass resources matching the selector will sync. If no selector is defined, **all** DeviceClass resources are synced. * **ResourceClaims resources (tenant → control plane)**: vCluster syncs ResourceClaims resources from the tenant cluster to the control plane cluster only if all the referenced `deviceClassName` exist in the control plane cluster and any of the following is true: * No selector is defined for DeviceClass syncing, which allows all ResourceClaims resources to sync regardless of the `deviceClassName`. * Each of the ResourceClaims's `deviceClassName` references a DeviceClass resource that matches the selector and is synced from the control plane cluster. * **ResourceClaimTemplates resources (tenant → control plane)**: vCluster syncs ResourceClaimTemplates resources from the tenant cluster to the control plane cluster only if all the referenced `deviceClassName` exist in the control plane cluster and any of the following is true: * No selector is defined for DeviceClass syncing, which allows all ResourceClaimTemplates resources to sync regardless of the `deviceClassName`. * Each of the ResourceClaimTemplates's `deviceClassName` references a DeviceClass resource that matches the selector and is synced from the control plane cluster. The same selector controls both sync flows. The selector determines which DeviceClass resources are imported from the control plane cluster and which ResourceClaims and ResourceClaimTemplates resources can sync to the control plane cluster based on the referenced DeviceClass resources. ## Use selectors to filter[​](#use-selectors-to-filter "Direct link to Use selectors to filter") Selectors provide precise control over which DeviceClass resources get synced from the control plane cluster and which ResourceClaims resources gets synced to the control plane cluster. vCluster supports two types of selector criteria that follow standard [Kubernetes label selector syntax](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements). ### Filter with matchLabels[​](#filter-with-matchlabels "Direct link to Filter with matchLabels") The `matchLabels` selector defines **exact** label key-value pairs that must be present on an DeviceClass for it to be synced. This provides straightforward filtering based on specific labels. The following example syncs only DeviceClass resources with a `environment: development` label: Example with matchLabels ``` sync: toHost: resourceClaims: enabled: true fromHost: deviceClasses: enabled: true selector: matchLabels: environment: development ``` ### Filter with matchExpressions[​](#filter-with-matchexpressions "Direct link to Filter with matchExpressions") The `matchExpressions` selector allows more flexible, set-based filtering with support for multiple operators: * `In`: Select all resources that has a specific key and the value is in the set of values * `NotIn`: Select all resources that has a specific key and the value is not in the set of values * `Exists`: Select all resources including a label with the key; no values are checked * `DoesNotExist`: Select all resources without a label with the key; no values are checked The following example syncs DeviceClass resources where the key of the label is `kubernetes.io/device.class` and the value of that label is either `nvidia` or `amd`: Example with matchExpressions ``` sync: toHost: resourceClaims: enabled: true fromHost: deviceClasses: enabled: true selector: matchExpressions: - key: kubernetes.io/device.class operator: In values: - nvidia - amd ``` ### Combined filter criteria[​](#combined-filter-criteria "Direct link to Combined filter criteria") Label conditions are additive meaning all specified label conditions must match for the DeviceClass resource to be selected to be synced. This means that if you define multiple `matchLabels` and `matchExpressions`, the DeviceClass resource must satisfy all criteria to be synced to the tenant cluster. The following example syncs DeviceClass resources that match both label and expression criteria: Example with matchLabels and matchExpressions ``` sync: toHost: resourceClaims: enabled: true fromHost: deviceClasses: enabled: true selector: matchLabels: environment: development matchExpressions: - key: kubernetes.io/device.class operator: In values: - nvidia - amd ``` In this example, the DeviceClass must have the following labels to sync to the tenant cluster: * `environment: development` * Either `kubernetes.io/device.class:nvidia` or `kubernetes.io/device.class:amd` ## Sync behavior considerations[​](#sync-behavior-considerations "Direct link to Sync behavior considerations") ### Resource lifecycle[​](#resource-lifecycle "Direct link to Resource lifecycle") Synced DeviceClass resources function like any other Kubernetes resource in the tenant cluster. You can view them with `kubectl get deviceclass` and reference them in your ResourceClaims and ResourceClaimTemplates resources specifications with the `DeviceClassName` field. When you modify the DeviceClass resource in the control plane cluster, vCluster re-evaluates whether it still matches the selector criteria. If the DeviceClass continues to match, vCluster updates the corresponding resource in the tenant cluster to reflect the changes. If the DeviceClass no longer matches the selector criteria, vCluster removes it from the tenant cluster. The selector system acts as both a resource filter and a validation mechanism. As a resource filter, it ensures that vCluster creates only DeviceClass resources matching the defined selector criteria. The selector also functions as a creation validation mechanism - when you create ResourceClaims and ResourceClaimTemplates resources in the tenant cluster, the resources can only reference DeviceClass resources that exist in the tenant cluster. ### Orphaned resources[​](#orphaned-resources "Direct link to Orphaned resources") When vCluster removes a synced DeviceClass from the tenant cluster due to selector changes or deletion from the control plane cluster, any ResourceClaims and ResourceClaimTemplates resources that reference it remain in the tenant and control plane clusters. These orphaned ResourceClaims and ResourceClaimTemplates resources stop receiving updates but vCluster does not automatically delete them to prevent unintended data loss. To remove these orphaned resources, you must delete them manually in the tenant and control plane clusters. This manual approach ensures that you maintain full control over resource cleanup and can verify that deletions are intentional. ### Error handling and troubleshooting[​](#error-handling-and-troubleshooting "Direct link to Error handling and troubleshooting") When the DeviceClass resource doesn't match the selector criteria during evaluation, vCluster logs a warning in the syncer pod's output to help with troubleshooting and monitoring. This logging provides visibility into which resources are being filtered out and why. When you create ResourceClaims and ResourceClaimTemplates resources that reference a DeviceClass not matching the selector criteria, several things occur to provide clear feedback: * The ResourceClaims or ResourceClaimTemplates fails to sync to the control plane cluster * vCluster records an event on the ResourceClaims or ResourceClaimTemplates resource in the tenant cluster * The event indicates that the specified DeviceClass is not available according to the current selector configuration The error output appears as a Kubernetes event that you can view using `kubectl describe`. The event message clearly states that the ResourceClaims or ResourceClaimTemplates resource was not synced because the referenced DeviceClass does not match the defined selector criteria. The error output looks like this: Example error handling output ``` vcluster-virtual-cluster-1:~$ kubectl describe resourceclaim my-resourceclaim Name: my-resourceclaim Namespace: default DeviceClass: amd Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning SyncWarning 10s resourceclaim-syncer did not sync resourceclaim "my-resourceclaim" to host. Failures : Device class amd does not match the selector under 'sync.fromHost.deviceClasses.selector' ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `deviceClasses` object [​](#deviceClasses "Direct link to deviceClasses")DeviceClasses defines if device classes in the host should get synced to the virtual cluster ### `enabled` boolean false [​](#deviceClasses-enabled "Direct link to deviceClasses-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#deviceClasses-patches "Direct link to deviceClasses-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#deviceClasses-patches-path "Direct link to deviceClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#deviceClasses-patches-expression "Direct link to deviceClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#deviceClasses-patches-reverseExpression "Direct link to deviceClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#deviceClasses-patches-reference "Direct link to deviceClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#deviceClasses-patches-reference-apiVersion "Direct link to deviceClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#deviceClasses-patches-reference-apiVersionPath "Direct link to deviceClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#deviceClasses-patches-reference-kind "Direct link to deviceClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#deviceClasses-patches-reference-kindPath "Direct link to deviceClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#deviceClasses-patches-reference-namePath "Direct link to deviceClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#deviceClasses-patches-reference-namespacePath "Direct link to deviceClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#deviceClasses-patches-labels "Direct link to deviceClasses-patches-labels")Labels treats the path value as a labels selector. ### `selector` object [​](#deviceClasses-selector "Direct link to deviceClasses-selector")Selector defines the selector to use for the resource. If not set, all resources of that type will be synced. #### `matchLabels` object [​](#deviceClasses-selector-matchLabels "Direct link to deviceClasses-selector-matchLabels") #### `matchExpressions` object\[] [​](#deviceClasses-selector-matchExpressions "Direct link to deviceClasses-selector-matchExpressions") ##### `key` string [​](#deviceClasses-selector-matchExpressions-key "Direct link to deviceClasses-selector-matchExpressions-key") ##### `operator` string [​](#deviceClasses-selector-matchExpressions-operator "Direct link to deviceClasses-selector-matchExpressions-operator") ##### `values` string\[] [​](#deviceClasses-selector-matchExpressions-values "Direct link to deviceClasses-selector-matchExpressions-values") --- # Events Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is enabled. Sync [Events](https://kubernetes.io/docs/reference/kubernetes-api/cluster-resources/event-v1/) from the control plane cluster to the tenant cluster to be able to view things like pod events from within the tenant cluster. The synced events are associated with tenant cluster resources that are synced to the control plane cluster and provide visibility for vCluster users. ### Disable syncing Events from the control plane cluster to the tenant cluster[​](#disable-syncing-events-from-the-control-plane-cluster-to-the-tenant-cluster "Direct link to Disable syncing Events from the control plane cluster to the tenant cluster") ``` sync: fromHost: events: enabled: false ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `events` object [​](#events "Direct link to events")Events defines if events should get synced from the host cluster to the virtual cluster, but not back. ### `enabled` boolean true [​](#events-enabled "Direct link to events-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#events-patches "Direct link to events-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#events-patches-path "Direct link to events-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#events-patches-expression "Direct link to events-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#events-patches-reverseExpression "Direct link to events-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#events-patches-reference "Direct link to events-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#events-patches-reference-apiVersion "Direct link to events-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#events-patches-reference-apiVersionPath "Direct link to events-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#events-patches-reference-kind "Direct link to events-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#events-patches-reference-kindPath "Direct link to events-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#events-patches-reference-namePath "Direct link to events-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#events-patches-reference-namespacePath "Direct link to events-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#events-patches-labels "Direct link to events-patches-labels")Labels treats the path value as a labels selector. --- # Gateway classes Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. `GatewayClass` syncing lets tenant clusters use Gateway API controllers that run on the control plane cluster. Normally, each tenant cluster would need its own Gateway controller to program traffic. With `GatewayClass` syncing enabled, tenants reference host-controller `GatewayClass` resources by name instead. Add labels to `GatewayClass` resources in the control plane cluster and configure vCluster to sync only those that match a label selector. When a tenant creates a `Gateway` that references a synced `GatewayClass`, the control plane cluster's Gateway controller handles the data plane. This saves resources because you don't need a Gateway controller in every tenant cluster, and it gives you centralized control over which controllers and listener capabilities tenants can use. Common use cases: * **Development environments**: Sync only the `GatewayClass` resources required for development tenants to reduce noise and avoid unnecessary exposure. * **Tenant isolation**: Let teams use their own `GatewayClass` resources while sharing a single control plane cluster. * **Security**: Restrict which `GatewayClass` resources are available in each tenant cluster to enforce access control and prevent unintended configurations. ## Enable syncing[​](#enable-syncing "Direct link to Enable syncing") When enabled, vCluster takes control of all GatewayClass resources in the tenant cluster. It only allows GatewayClass resources that are synced from the control plane cluster to exist. This example configuration enables syncing **all** GatewayClass resources from the control plane cluster to the tenant cluster: ``` sync: fromHost: gatewayClasses: enabled: true ``` If you try to create the GatewayClass resource directly in the tenant cluster, using for example `kubectl create` or `kubectl apply`, vCluster detects it and deletes it immediately. This prevents conflicts between locally created GatewayClass resources and those synced from the control plane cluster, ensuring that only GatewayClass resources from the control plane cluster exist in the tenant cluster. GatewayClass resource limitation When enabled, GatewayClass resource creation can only occur in the control plane cluster. Any GatewayClass resources created in the tenant cluster will be deleted. ## How syncing works[​](#how-syncing-works "Direct link to How syncing works") When GatewayClass syncing is enabled, vCluster can use a label selector to control which GatewayClass and Gateway resources are synchronized between the control plane and tenant clusters. This affects the resource types with separate unidirectional sync flows: * **GatewayClass resources (control plane → tenant)**: vCluster syncs GatewayClass resources from the control plane cluster to the tenant cluster. You cannot create GatewayClass resources directly in the tenant cluster. If a selector is defined, only GatewayClass resources matching the selector will sync. If no selector is defined, **all** GatewayClass resources are synced. * **Gateway resources (tenant → control plane)**: vCluster syncs Gateway resources from the tenant cluster to the control plane cluster only if any of the following is true: * No selector is defined for GatewayClass syncing, which allows all Gateway resources to sync regardless of the `gatewayClassName`. * The Gateway's `gatewayClassName` references a GatewayClass resource that matches the selector and is synced from the control plane cluster. * The Gateway has an empty `gatewayClassName` field. The same selector controls both sync flows. The selector determines which GatewayClass resources are imported from the control plane cluster and which Gateway resources can sync to the control plane cluster based on the referenced GatewayClass resource. ## Use selectors to filter[​](#use-selectors-to-filter "Direct link to Use selectors to filter") Selectors provide precise control over which GatewayClass resources get synced from the control plane cluster and which Gateway resources gets synced to the control plane cluster. vCluster supports two types of selector criteria that follow standard [Kubernetes label selector syntax](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements). ### Filter with matchLabels[​](#filter-with-matchlabels "Direct link to Filter with matchLabels") The `matchLabels` selector defines **exact** label key-value pairs that must be present on an GatewayClass for it to be synced. This provides straightforward filtering based on specific labels. The following example syncs only GatewayClass resources with a `environment: development` label: Example with matchLabels ``` sync: toHost: gateways: enabled: true fromHost: gatewayClasses: enabled: true selector: matchLabels: environment: development ``` ### Filter with matchExpressions[​](#filter-with-matchexpressions "Direct link to Filter with matchExpressions") The `matchExpressions` selector allows more flexible, set-based filtering with support for multiple operators: * `In`: Select all resources that has a specific key and the value is in the set of values * `NotIn`: Select all resources that has a specific key and the value is not in the set of values * `Exists`: Select all resources including a label with the key; no values are checked * `DoesNotExist`: Select all resources without a label with the key; no values are checked The following example syncs GatewayClass resources where the key of the label is `gateway-demo.loft.sh/sync` and the value of that label is either `yes` or `no`: Example with matchExpressions ``` sync: toHost: gateways: enabled: true fromHost: gatewayClasses: enabled: true selector: matchExpressions: - key: gateway-demo.loft.sh/sync operator: In values: - yes - no ``` ### Combined filter criteria[​](#combined-filter-criteria "Direct link to Combined filter criteria") Label conditions are additive meaning all specified label conditions must match for the GatewayClass resource to be selected to be synced. This means that if you define multiple `matchLabels` and `matchExpressions`, the GatewayClass resource must satisfy all criteria to be synced to the tenant cluster. The following example syncs GatewayClass resources that match both label and expression criteria: Example with matchLabels and matchExpressions ``` sync: toHost: gateways: enabled: true fromHost: gatewayClasses: enabled: true selector: matchLabels: environment: development matchExpressions: - key: gateway-demo.loft.sh/sync operator: In values: - yes - no ``` In this example, the GatewayClass must have the following labels to sync to the tenant cluster: * `environment: development` * Either `gateway-demo.loft.sh/sync:yes` or `gateway-demo.loft.sh/sync:no` ## Sync behavior considerations[​](#sync-behavior-considerations "Direct link to Sync behavior considerations") ### Resource lifecycle[​](#resource-lifecycle "Direct link to Resource lifecycle") Synced GatewayClass resources function like any other Kubernetes resource in the tenant cluster. You can view them with `kubectl get gatewayclass` and reference them in your Gateway resource specifications with the `gatewayClassName` field. When you modify the GatewayClass resource in the control plane cluster, vCluster re-evaluates whether it still matches the selector criteria. If the GatewayClass continues to match, vCluster updates the corresponding resource in the tenant cluster to reflect the changes. If the GatewayClass no longer matches the selector criteria, vCluster removes it from the tenant cluster. The selector system acts as both a resource filter and a validation mechanism. As a resource filter, it ensures that vCluster creates only GatewayClass resources matching the defined selector criteria. The selector also functions as a creation validation mechanism - when you create Gateway resources in the tenant cluster, the resources can only reference GatewayClass resources that exist in the tenant cluster. ### Orphaned resources[​](#orphaned-resources "Direct link to Orphaned resources") When vCluster removes a synced GatewayClass from the tenant cluster due to selector changes or deletion from the control plane cluster, any Gateway resources that reference it remain in the tenant and control plane clusters. These orphaned Gateway resources stop receiving updates but vCluster does not automatically delete them to prevent unintended data loss. To remove these orphaned resources, you must delete them manually in the tenant and control plane clusters. This manual approach ensures that you maintain full control over resource cleanup and can verify that deletions are intentional. ### Error handling and troubleshooting[​](#error-handling-and-troubleshooting "Direct link to Error handling and troubleshooting") When the GatewayClass resource doesn't match the selector criteria during evaluation, vCluster logs a warning in the syncer pod's output to help with troubleshooting and monitoring. This logging provides visibility into which resources are being filtered out and why. When you creates the Gateway resource that references a GatewayClass not matching the selector criteria, several things occur to provide clear feedback: * The Gateway fails to sync to the control plane cluster * vCluster records an event on the Gateway resource in the tenant cluster * The event indicates that the specified GatewayClass is not available according to the current selector configuration The error output appears as a Kubernetes event that you can view using `kubectl describe`. The event message clearly states that the Gateway resource was not synced because the referenced GatewayClass does not match the defined selector criteria. The error output looks like this: Example error handling output ``` vcluster-virtual-cluster-1:~$ kubectl describe gateway my-gateway Name: my-gateway Namespace: default GatewayClass: no Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning SyncWarning 10s gateway-syncer did not sync gateway "my-gateway" to host because it does not match the selector under 'sync.fromHost.gatewayClasses.selector' ``` ## Example: select specific `GatewayClass` resources by label[​](#example-select-specific-gatewayclass-resources-by-label "Direct link to example-select-specific-gatewayclass-resources-by-label") In the control plane cluster, label the `GatewayClass` resources you want exposed: GatewayClass in the control plane cluster ``` apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: gw-demo-11-allowed labels: gateway-demo.loft.sh/sync: "yes" spec: controllerName: gateway.envoyproxy.io/gatewayclass-controller ``` Then point vCluster at that label in `vcluster.yaml`: vcluster.yaml ``` sync: fromHost: gatewayClasses: enabled: true selector: matchLabels: gateway-demo.loft.sh/sync: "yes" ``` `GatewayClass` resources without the matching label remain only in the control plane cluster. If a tenant creates a `Gateway` that references an unsynced class, vCluster records a `SyncWarning` event on the tenant `Gateway`: Tenant event for an unselected GatewayClass ``` Warning SyncWarning Gateway "filtered-edge" sync skipped. The GatewayClass "gw-demo-11-filtered" does not match the selector under 'sync.fromHost.gatewayClasses.selector' ``` Inspect with: Confirm which GatewayClasses reached the tenant ``` kubectl get gatewayclass kubectl describe gateway -n kubectl get events --field-selector type=Warning ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `gatewayClasses` object [​](#gatewayClasses "Direct link to gatewayClasses")GatewayClasses defines if gateway classes should get synced from the control plane cluster to the tenant cluster, but not back. ### `enabled` boolean false [​](#gatewayClasses-enabled "Direct link to gatewayClasses-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#gatewayClasses-patches "Direct link to gatewayClasses-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#gatewayClasses-patches-path "Direct link to gatewayClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#gatewayClasses-patches-expression "Direct link to gatewayClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#gatewayClasses-patches-reverseExpression "Direct link to gatewayClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#gatewayClasses-patches-reference "Direct link to gatewayClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#gatewayClasses-patches-reference-apiVersion "Direct link to gatewayClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#gatewayClasses-patches-reference-apiVersionPath "Direct link to gatewayClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#gatewayClasses-patches-reference-kind "Direct link to gatewayClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#gatewayClasses-patches-reference-kindPath "Direct link to gatewayClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#gatewayClasses-patches-reference-namePath "Direct link to gatewayClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#gatewayClasses-patches-reference-namespacePath "Direct link to gatewayClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#gatewayClasses-patches-labels "Direct link to gatewayClasses-patches-labels")Labels treats the path value as a labels selector. ### `selector` object [​](#gatewayClasses-selector "Direct link to gatewayClasses-selector")Selector defines the selector to use for the resource. If not set, all resources of that type will be synced. #### `matchLabels` object [​](#gatewayClasses-selector-matchLabels "Direct link to gatewayClasses-selector-matchLabels") #### `matchExpressions` object\[] [​](#gatewayClasses-selector-matchExpressions "Direct link to gatewayClasses-selector-matchExpressions") ##### `key` string [​](#gatewayClasses-selector-matchExpressions-key "Direct link to gatewayClasses-selector-matchExpressions-key") ##### `operator` string [​](#gatewayClasses-selector-matchExpressions-operator "Direct link to gatewayClasses-selector-matchExpressions-operator") ##### `values` string\[] [​](#gatewayClasses-selector-matchExpressions-values "Direct link to gatewayClasses-selector-matchExpressions-values") --- # Gateways and GatewayClasses Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Use `sync.fromHost.gatewayClasses` and `sync.fromHost.gateways` when platform administrators own Gateway API infrastructure in the Control Plane Cluster. Tenant clusters can then see approved Gateway resources. Imported Gateway API resources are read-only inside the tenant cluster. Tenants can reference them from Routes, but changes to imported `GatewayClass` or `Gateway` resources do not sync back to the Control Plane Cluster. ## Configure admin-owned shared gateways[​](#configure-admin-owned-shared-gateways "Direct link to Configure admin-owned shared gateways") In a shared Gateway model: 1. Platform administrators install the Gateway API CRDs and Gateway controller. 2. Platform administrators create the `GatewayClass` and one or more `Gateway` resources in the Control Plane Cluster. 3. vCluster imports the approved `GatewayClass` and `Gateway` resources into tenant clusters. 4. Tenants create `HTTPRoute` or `TLSRoute` resources that attach to the imported Gateway. This lets platform teams own DNS, certificates, listener policy, and controller lifecycle while tenants own their application routes. ## Route attachment policy[​](#route-attachment-policy "Direct link to Route attachment policy") Imported Gateways can expose tenant-facing `allowedRoutes` policy. Use this to make the allowed route attachment model visible to tenants and enforce which tenant namespaces or hostnames can attach to a shared Gateway. If tenants need to route to a backend in another namespace, the selected Gateway API topology may also require a `ReferenceGrant`. ## Status and metadata visibility[​](#status-and-metadata-visibility "Direct link to Status and metadata visibility") Gateway status can be mirrored into the tenant cluster so tenants can see whether a Gateway has addresses or listener status. Metadata controls can expose source Gateway information when that helps tenants understand which Control Plane Cluster Gateway they are using. Keep this information scoped to what tenants need. Shared Gateway infrastructure often carries operational details that should remain platform-owned. ## Config reference[​](#config-reference "Direct link to Config reference") See the generated [`sync.fromHost.gatewayClasses`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host.md#fromHost-gatewayClasses) and [`sync.fromHost.gateways`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host.md#fromHost-gateways) config reference for exact YAML fields, including imported Gateways, `allowedRoutes`, status, metadata, and sanitization options. --- # Ingress classes Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. IngressClass syncing allows tenant clusters to use ingress controllers from the control plane cluster. Typically, each tenant cluster needs its own ingress controller to handle incoming traffic. With IngressClass syncing enabled, tenant clusters can reference IngressClass resources from the control plane cluster instead. You can enable this feature to sync IngressClass resources from the control plane cluster to the tenant cluster. Add labels to IngressClass resources in your control plane cluster and configure vCluster to sync only those that match specific label selectors. When you create an Ingress resource in the tenant cluster that references a synced IngressClass, the control plane cluster's ingress controller handles the traffic routing. This saves resources since you don't need separate ingress controllers in every tenant cluster. It also gives you centralized control over which ingress capabilities teams can access. This approach is useful in scenarios where selective access to ingress configurations is required. Common use cases include: * **Development environments**: Sync only the IngressClass resources required for development clusters to reduce noise and avoid unnecessary exposure. * **Tenant isolation**: Enable teams to use their own IngressClass resources while sharing a single control plane cluster. * **Security**: Restrict the IngressClass resources available in the tenant cluster to enforce access control and prevent unintended configurations. ## Enable syncing[​](#enable-syncing "Direct link to Enable syncing") When enabled, vCluster takes control of all IngressClass resources in the tenant cluster. It only allows IngressClass resources that are synced from the control plane cluster to exist. This example configuration enables syncing **all** IngressClass resources from the control plane cluster to the tenant cluster: ``` sync: fromHost: ingressClasses: enabled: true ``` If you try to create the IngressClass resource directly in the tenant cluster, using for example `kubectl create` or `kubectl apply`, vCluster detects it and deletes it immediately. This prevents conflicts between locally created IngressClass resources and those synced from the control plane cluster, ensuring that only IngressClass resources from the control plane cluster exist in the tenant cluster. IngressClass resource limitation When enabled, IngressClass resource creation can only occur in the control plane cluster. Any IngressClass resources created in the tenant cluster will be deleted. ## How syncing works[​](#how-syncing-works "Direct link to How syncing works") When IngressClass syncing is enabled, vCluster can use a label selector to control which IngressClass and Ingress resources are synchronized between the control plane and tenant clusters. This affects the resource types with separate unidirectional sync flows: * **IngressClass resources (control plane → tenant)**: vCluster syncs IngressClass resources from the control plane cluster to the tenant cluster. You cannot create IngressClass resources directly in the tenant cluster. If a selector is defined, only IngressClass resources matching the selector will sync. If no selector is defined, **all** IngressClass resources are synced. * **Ingress resources (tenant → control plane)**: vCluster syncs Ingress resources from the tenant cluster to the control plane cluster only if any of the following is true: * No selector is defined for IngressClass syncing, which allows all Ingress resources to sync regardless of the `IngressClassName`. * The Ingress's `IngressClassName` references a IngressClass resource that matches the selector and is synced from the control plane cluster. * The Ingress has an empty `IngressClassName` field. The same selector controls both sync flows. The selector determines which IngressClass resources are imported from the control plane cluster and which Ingress resources can sync to the control plane cluster based on the referenced IngressClass resource. ## Use selectors to filter[​](#use-selectors-to-filter "Direct link to Use selectors to filter") Selectors provide precise control over which IngressClass resources get synced from the control plane cluster and which Ingress resources gets synced to the control plane cluster. vCluster supports two types of selector criteria that follow standard [Kubernetes label selector syntax](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements). ### Filter with matchLabels[​](#filter-with-matchlabels "Direct link to Filter with matchLabels") The `matchLabels` selector defines **exact** label key-value pairs that must be present on an IngressClass for it to be synced. This provides straightforward filtering based on specific labels. The following example syncs only IngressClass resources with a `environment: development` label: Example with matchLabels ``` sync: toHost: ingresses: enabled: true fromHost: ingressClasses: enabled: true selector: matchLabels: environment: development ``` ### Filter with matchExpressions[​](#filter-with-matchexpressions "Direct link to Filter with matchExpressions") The `matchExpressions` selector allows more flexible, set-based filtering with support for multiple operators: * `In`: Select all resources that has a specific key and the value is in the set of values * `NotIn`: Select all resources that has a specific key and the value is not in the set of values * `Exists`: Select all resources including a label with the key; no values are checked * `DoesNotExist`: Select all resources without a label with the key; no values are checked The following example syncs IngressClass resources where the key of the label is `kubernetes.io/ingress.class` and the value of that label is either `istio` or `traefik`: Example with matchExpressions ``` sync: toHost: ingresses: enabled: true fromHost: ingressClasses: enabled: true selector: matchExpressions: - key: kubernetes.io/ingress.class operator: In values: - istio - traefik ``` ### Combined filter criteria[​](#combined-filter-criteria "Direct link to Combined filter criteria") Label conditions are additive meaning all specified label conditions must match for the IngressClass resource to be selected to be synced. This means that if you define multiple `matchLabels` and `matchExpressions`, the IngressClass resource must satisfy all criteria to be synced to the tenant cluster. The following example syncs IngressClass resources that match both label and expression criteria: Example with matchLabels and matchExpressions ``` sync: toHost: ingresses: enabled: true fromHost: ingressClasses: enabled: true selector: matchLabels: environment: development matchExpressions: - key: kubernetes.io/ingress.class operator: In values: - istio - traefik ``` In this example, the IngressClass must have the following labels to sync to the tenant cluster: * `environment: development` * Either `kubernetes.io/ingress.class:istio` or `kubernetes.io/ingress.class:traefik` ## Sync behavior considerations[​](#sync-behavior-considerations "Direct link to Sync behavior considerations") ### Resource lifecycle[​](#resource-lifecycle "Direct link to Resource lifecycle") Synced IngressClass resources function like any other Kubernetes resource in the tenant cluster. You can view them with `kubectl get ingressclass` and reference them in your Ingress resource specifications with the `IngressClassName` field. When you modify the IngressClass resource in the control plane cluster, vCluster re-evaluates whether it still matches the selector criteria. If the IngressClass continues to match, vCluster updates the corresponding resource in the tenant cluster to reflect the changes. If the IngressClass no longer matches the selector criteria, vCluster removes it from the tenant cluster. The selector system acts as both a resource filter and a validation mechanism. As a resource filter, it ensures that vCluster creates only IngressClass resources matching the defined selector criteria. The selector also functions as a creation validation mechanism - when you create Ingress resources in the tenant cluster, the resources can only reference IngressClass resources that exist in the tenant cluster. ### Orphaned resources[​](#orphaned-resources "Direct link to Orphaned resources") When vCluster removes a synced IngressClass from the tenant cluster due to selector changes or deletion from the control plane cluster, any Ingress resources that reference it remain in the tenant and control plane clusters. These orphaned Ingress resources stop receiving updates but vCluster does not automatically delete them to prevent unintended data loss. To remove these orphaned resources, you must delete them manually in the tenant and control plane clusters. This manual approach ensures that you maintain full control over resource cleanup and can verify that deletions are intentional. ### Error handling and troubleshooting[​](#error-handling-and-troubleshooting "Direct link to Error handling and troubleshooting") When the IngressClass resource doesn't match the selector criteria during evaluation, vCluster logs a warning in the syncer pod's output to help with troubleshooting and monitoring. This logging provides visibility into which resources are being filtered out and why. When you creates the Ingress resource that references a IngressClass not matching the selector criteria, several things occur to provide clear feedback: * The Ingress fails to sync to the control plane cluster * vCluster records an event on the Ingress resource in the tenant cluster * The event indicates that the specified IngressClass is not available according to the current selector configuration The error output appears as a Kubernetes event that you can view using `kubectl describe`. The event message clearly states that the Ingress resource was not synced because the referenced IngressClass does not match the defined selector criteria. The error output looks like this: Example error handling output ``` vcluster-virtual-cluster-1:~$ kubectl describe ingress my-ingress Name: my-ingress Namespace: default IngressClass: traefik Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning SyncWarning 10s ingress-syncer did not sync ingress "my-ingress" to host because it does not match the selector under 'sync.fromHost.ingressClasses.selector' ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `ingressClasses` object [​](#ingressClasses "Direct link to ingressClasses")IngressClasses defines if ingress classes should get synced from the host cluster to the virtual cluster, but not back. ### `enabled` boolean false [​](#ingressClasses-enabled "Direct link to ingressClasses-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#ingressClasses-patches "Direct link to ingressClasses-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#ingressClasses-patches-path "Direct link to ingressClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#ingressClasses-patches-expression "Direct link to ingressClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#ingressClasses-patches-reverseExpression "Direct link to ingressClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#ingressClasses-patches-reference "Direct link to ingressClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#ingressClasses-patches-reference-apiVersion "Direct link to ingressClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#ingressClasses-patches-reference-apiVersionPath "Direct link to ingressClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#ingressClasses-patches-reference-kind "Direct link to ingressClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#ingressClasses-patches-reference-kindPath "Direct link to ingressClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#ingressClasses-patches-reference-namePath "Direct link to ingressClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#ingressClasses-patches-reference-namespacePath "Direct link to ingressClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#ingressClasses-patches-labels "Direct link to ingressClasses-patches-labels")Labels treats the path value as a labels selector. ### `selector` object [​](#ingressClasses-selector "Direct link to ingressClasses-selector")Selector defines the selector to use for the resource. If not set, all resources of that type will be synced. #### `matchLabels` object [​](#ingressClasses-selector-matchLabels "Direct link to ingressClasses-selector-matchLabels") #### `matchExpressions` object\[] [​](#ingressClasses-selector-matchExpressions "Direct link to ingressClasses-selector-matchExpressions") ##### `key` string [​](#ingressClasses-selector-matchExpressions-key "Direct link to ingressClasses-selector-matchExpressions-key") ##### `operator` string [​](#ingressClasses-selector-matchExpressions-operator "Direct link to ingressClasses-selector-matchExpressions-operator") ##### `values` string\[] [​](#ingressClasses-selector-matchExpressions-values "Direct link to ingressClasses-selector-matchExpressions-values") --- # Nodes Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. vCluster only displays nodes in the tenant cluster if the tenant cluster has synced pods to that worker node from the control plane cluster. If there are no more pods on a node, vCluster deletes the node. Without syncing the nodes, these nodes are considered pseudo nodes as the name is the only real value, and the rest of the node information is randomly generated. If you need specific node information, enabling to sync nodes from the control plane cluster will allow the real information from the node to be available from the tenant cluster context. In order to access node information, a cluster role is deployed onto the control plane cluster. Node and kubelet disclosure on shared nodes Syncing real nodes gives every tenant on a shared node visibility into that node's identity. `networking.advanced.proxyKubelets` enables a vCluster-mediated kubelet proxy that backs kubelet metrics scraping and `kubectl top node` against synced nodes. On vCluster v0.34.0 and later, its pod-listing, container-log, and per-pod stats responses are scoped to the tenant's own pods. Node-level metric samples aren't scoped on any version. Disable it unless you need that workflow, both to close that remaining gap and as a safeguard on any version older than v0.34.0. Always set `clearImageStatus: true` alongside this setting on shared nodes, and review [node and kubelet disclosure](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#operational-hardening-and-lifecycle) before enabling it for trusted tenants. Node IP obfuscation By default, vCluster obfuscates node IP addresses when syncing real nodes to protect sensitive information. Learn how to [control node IP visibility](https://vcluster.com/docs/vcluster/learn-how-to/worker-nodes/host-nodes/control-node-ip-visibility.md) for your use case. ## Select a set of nodes[​](#select-a-set-of-nodes "Direct link to Select a set of nodes") By default, vCluster is able to schedule pods to *any* of the worker nodes in the control plane cluster, but you can isolate a tenant cluster to only use a set of specific nodes from the control plane cluster. There are multiple reasons why you may want to dedicate nodes to a tenant cluster including: * Restricting vCluster to nodes in a specific region. * Targeting nodes with a particular architecture. * Using only spot or preemptible instances. * Preventing vCluster workloads from running on critical infrastructure nodes. Selecting nodes is based on a node label selector, where the nodes on the control plane cluster must have the matching labels. ### Select a set of nodes without syncing node information[​](#select-a-set-of-nodes-without-syncing-node-information "Direct link to Select a set of nodes without syncing node information") If you don't want to sync node information, you can still select to use only a set of nodes from the control plane cluster. Select dedicated nodes without syncing real node information ``` sync: fromHost: nodes: # Do not sync real node information enabled: false # Select a set of nodes based on the node labels selector: labels: environment: production team: backend ``` ### Select a set of nodes and sync node information[​](#select-a-set-of-nodes-and-sync-node-information "Direct link to Select a set of nodes and sync node information") You can select a set of nodes from the control plane cluster while also requesting that the nodes sync with the real information from the control plane cluster. Select dedicated nodes and syncing real node information ``` sync: fromHost: nodes: # Sync real node information enabled: true # Select a set of nodes based on the node labels selector: labels: environment: production team: backend ``` ## Sync real nodes[​](#sync-real-nodes "Direct link to Sync real nodes") Sync nodes to the tenant cluster to view real node information. The tenant cluster will only display the host nodes that have a pod deployed onto it. By enabling this sync, a cluster role is deployed by the vCluster onto the control plane cluster. Sync nodes for real node information ``` sync: fromHost: nodes: enabled: true ``` ### Sync all real nodes[​](#sync-all-real-nodes "Direct link to Sync all real nodes") When syncing nodes, only nodes that have pods deployed on them will show up in the tenant cluster, but you can sync all the nodes from the control plane cluster. When selecting all the nodes, `kubectl get nodes` will display all the host nodes whether or not a pod has been scheduled onto the control plane cluster. Make all nodes visible to the tenant cluster ``` sync: fromHost: nodes: enabled: true selector: all: true ``` ### Sync back labels and taints[​](#sync-back-labels-and-taints "Direct link to Sync back labels and taints") By default, when syncing nodes from the control plane cluster, labels, and taints are only synced from the control plane cluster to the tenant cluster. No changes from the tenant cluster context would be reflected on the host nodes. Enabling `syncBackChanges` allows labels and taints set from the tenant cluster context to be synced to the host nodes. Additional permissions are added to the cluster role to edit the host nodes. Sync labels and taints from the tenant cluster context to the host node ``` sync: fromHost: nodes: enabled: true syncBackChanges: true ``` ### Hide image information[​](#hide-image-information "Direct link to Hide image information") When multiple tenants are using the same set of host nodes, container image metadata could be used maliciously. For increased security, you can enable `clearImageStatus` to remove all `status.images` from the node object. Clear Image Status on the Node ``` sync: fromHost: nodes: enabled: true clearImageStatus: true ``` ### Patches[​](#patches "Direct link to Patches") Use `sync.fromHost.nodes.patches` to transform node fields while syncing nodes from the control plane cluster to the tenant cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") ## `nodes` object [​](#nodes "Direct link to nodes")Nodes defines if nodes should get synced from the host cluster to the virtual cluster, but not back. ### `enabled` boolean false [​](#nodes-enabled "Direct link to nodes-enabled")Enabled specifies if syncing real nodes should be enabled. If this is disabled, vCluster will create fake nodes instead. ### `syncBackChanges` boolean false [​](#nodes-syncBackChanges "Direct link to nodes-syncBackChanges")SyncBackChanges enables syncing labels and taints from the virtual cluster to the host cluster. If this is enabled someone within the virtual cluster will be able to change the labels and taints of the host cluster node. ### `clearImageStatus` boolean false [​](#nodes-clearImageStatus "Direct link to nodes-clearImageStatus")ClearImageStatus will erase the image status when syncing a node. This allows to hide images that are pulled by the node. ### `selector` object [​](#nodes-selector "Direct link to nodes-selector")Selector can be used to define more granular what nodes should get synced from the host cluster to the virtual cluster. #### `all` boolean false [​](#nodes-selector-all "Direct link to nodes-selector-all")All specifies if all nodes should get synced by vCluster from the host to the virtual cluster or only the ones where pods are assigned to. #### `labels` object {} [​](#nodes-selector-labels "Direct link to nodes-selector-labels")Labels are the node labels used to sync nodes from host cluster to virtual cluster. This will also set the node selector when syncing a pod from virtual cluster to host cluster to the same value. ### `patches` object\[] [​](#nodes-patches "Direct link to nodes-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#nodes-patches-path "Direct link to nodes-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#nodes-patches-expression "Direct link to nodes-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#nodes-patches-reverseExpression "Direct link to nodes-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#nodes-patches-reference "Direct link to nodes-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#nodes-patches-reference-apiVersion "Direct link to nodes-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#nodes-patches-reference-apiVersionPath "Direct link to nodes-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#nodes-patches-reference-kind "Direct link to nodes-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#nodes-patches-reference-kindPath "Direct link to nodes-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#nodes-patches-reference-namePath "Direct link to nodes-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#nodes-patches-reference-namespacePath "Direct link to nodes-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#nodes-patches-labels "Direct link to nodes-patches-labels")Labels treats the path value as a labels selector. --- # Priority classes Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. PriorityClass resources set which pods get scheduled first when resources are limited. High-priority pods can kick out low-priority pods to get resources. Typically, each tenant cluster needs its own copy of every PriorityClass. This means creating the same PriorityClass multiple times-once per tenant cluster. If anything changes in your PriorityClass, you must update each copy in each tenant cluster. PriorityClass syncing eliminates this duplication and maintenance by making control plane cluster PriorityClass resources available to tenant clusters. You define priority classes once in the control plane cluster, and vCluster automatically makes them accessible to tenant clusters. You can either sync all PriorityClass resources or configure syncing using label selectors. When a tenant cluster pod references a `priorityClassName`, vCluster validates that the corresponding PriorityClass exists in the tenant cluster through the syncing process. If the referenced PriorityClass is not synced (either because it does not exist in the control plane cluster or does not match your label selector), the pod is not scheduled on the control plane cluster. ## Enable syncing[​](#enable-syncing "Direct link to Enable syncing") When enabled, vCluster takes control of all PriorityClass resources in the tenant cluster. It only allows PriorityClass resources that are synced from the control plane cluster to exist. This example configuration enables syncing **all** PriorityClass resources from the control plane cluster to the tenant cluster: ``` sync: fromHost: priorityClasses: enabled: true ``` If you try to create the PriorityClass resource directly in the tenant cluster, using for example `kubectl create` or `kubectl apply`, vCluster detects it and deletes it immediately. This prevents conflicts between locally created PriorityClass resources and those synced from the control plane cluster, ensuring that only PriorityClass resources from the control plane cluster exist in the tenant cluster. PriorityClass resource limitation When enabled, PriorityClass resource creation can only occur in the control plane cluster. Any PriorityClass resources created in the tenant cluster will be deleted. ## How syncing works[​](#how-syncing-works "Direct link to How syncing works") When PriorityClass syncing is enabled, vCluster can use a label selector to control which PriorityClass and Pod resources are synchronized between the control plane and tenant clusters. This affects the resource types with separate unidirectional sync flows: * **PriorityClass resources (control plane → tenant)**: vCluster syncs PriorityClass resources from the control plane cluster to the tenant cluster. You cannot create PriorityClass resources directly in the tenant cluster. If a selector is defined, only PriorityClass resources matching the selector will sync. If no selector is defined, **all** PriorityClass resources are synced. * **Pod resources (tenant → control plane)**: vCluster syncs Pod resources from the tenant cluster to the control plane cluster only if any of the following is true: * No selector is defined for PriorityClass syncing, which allows all Pod resources to sync regardless of the `priorityClassName`. * The Pod's `priorityClassName` references a PriorityClass resource that matches the selector and is synced from the control plane cluster. * The Pod has an empty `priorityClassName` field. The same selector controls both sync flows. The selector determines which PriorityClass resources are imported from the control plane cluster and which Pod resources can sync to the control plane cluster based on the referenced PriorityClass resource. ## Use selectors to filter[​](#use-selectors-to-filter "Direct link to Use selectors to filter") Selectors provide precise control over which PriorityClass resources get synced from the control plane cluster and which Pod resources gets synced to the control plane cluster. vCluster supports two types of selector criteria that follow standard [Kubernetes label selector syntax](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements). ### Filter with matchLabels[​](#filter-with-matchlabels "Direct link to Filter with matchLabels") The `matchLabels` selector defines **exact** label key-value pairs that must be present on an PriorityClass for it to be synced. This provides straightforward filtering based on specific labels. The following example syncs only PriorityClass resources with a `environment: development` label: Example with matchLabels ``` sync: fromHost: priorityClasses: enabled: true selector: matchLabels: environment: development ``` ### Filter with matchExpressions[​](#filter-with-matchexpressions "Direct link to Filter with matchExpressions") The `matchExpressions` selector allows more flexible, set-based filtering with support for multiple operators: * `In`: Select all resources that has a specific key and the value is in the set of values * `NotIn`: Select all resources that has a specific key and the value is not in the set of values * `Exists`: Select all resources including a label with the key; no values are checked * `DoesNotExist`: Select all resources without a label with the key; no values are checked The following example syncs PriorityClass resources where the key of the label is `kubernetes.io/priority.class` and the value of that label is either `low` or `medium`: Example with matchExpressions ``` sync: fromHost: priorityClasses: enabled: true selector: matchExpressions: - key: kubernetes.io/priority.class operator: In values: - low - medium ``` ### Combined filter criteria[​](#combined-filter-criteria "Direct link to Combined filter criteria") Label conditions are additive meaning all specified label conditions must match for the PriorityClass resource to be selected to be synced. This means that if you define multiple `matchLabels` and `matchExpressions`, the PriorityClass resource must satisfy all criteria to be synced to the tenant cluster. The following example syncs PriorityClass resources that match both label and expression criteria: Example with matchLabels and matchExpressions ``` sync: fromHost: priorityClasses: enabled: true selector: matchLabels: environment: development matchExpressions: - key: kubernetes.io/priority.class operator: In values: - low - medium ``` In this example, the PriorityClass must have the following labels to sync to the tenant cluster: * `environment: development` * Either `kubernetes.io/priority.class:low` or `kubernetes.io/priority.class:medium` ## Sync behavior considerations[​](#sync-behavior-considerations "Direct link to Sync behavior considerations") ### Resource lifecycle[​](#resource-lifecycle "Direct link to Resource lifecycle") Synced PriorityClass resources function like any other Kubernetes resource in the tenant cluster. You can view them with `kubectl get priorityclass` and reference them in your Pod resource specifications with the `priorityClassName` field. When you modify the PriorityClass resource in the control plane cluster, vCluster re-evaluates whether it still matches the selector criteria. If the PriorityClass continues to match, vCluster updates the corresponding resource in the tenant cluster to reflect the changes. If the PriorityClass no longer matches the selector criteria, vCluster removes it from the tenant cluster. The selector system acts as both a resource filter and a validation mechanism. As a resource filter, it ensures that vCluster creates only PriorityClass resources matching the defined selector criteria. The selector also functions as a creation validation mechanism - when you create Pod resources in the tenant cluster, the resources can only reference PriorityClass resources that exist in the tenant cluster. ### Orphaned resources[​](#orphaned-resources "Direct link to Orphaned resources") When vCluster removes a synced PriorityClass from the tenant cluster due to selector changes or deletion from the control plane cluster, any Pod resources that reference it remain in the tenant and control plane clusters. These orphaned Pod resources stop receiving updates but vCluster does not automatically delete them to prevent unintended data loss. To remove these orphaned resources, you must delete them manually in the tenant and control plane clusters. This manual approach ensures that you maintain full control over resource cleanup and can verify that deletions are intentional. ### Error handling and troubleshooting[​](#error-handling-and-troubleshooting "Direct link to Error handling and troubleshooting") When the PriorityClass resource doesn't match the selector criteria during evaluation, vCluster logs a warning in the syncer pod's output to help with troubleshooting and monitoring. This logging provides visibility into which resources are being filtered out and why. When you creates the Pod resource that references a PriorityClass not matching the selector criteria, several things occur to provide clear feedback: * The Pod fails to sync to the control plane cluster * vCluster records an event on the Pod resource in the tenant cluster * The event indicates that the specified PriorityClass is not available according to the current selector configuration The error output appears as a Kubernetes event that you can view using `kubectl describe`. The event message clearly states that the Pod resource was not synced because the referenced PriorityClass does not match the defined selector criteria. The error output looks like this: Example error handling output ``` vcluster-virtual-cluster-1:~$ kubectl describe pod my-pod Name: my-pod Namespace: default PriorityClass: medium Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning SyncWarning 10s pod-syncer did not sync pod "my-pod" to host because it does not match the selector under 'sync.fromHost.priorityClasses.selector' ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `priorityClasses` object [​](#priorityClasses "Direct link to priorityClasses")PriorityClasses defines if priority classes classes should get synced from the host cluster to the virtual cluster, but not back. ### `enabled` boolean false [​](#priorityClasses-enabled "Direct link to priorityClasses-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#priorityClasses-patches "Direct link to priorityClasses-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#priorityClasses-patches-path "Direct link to priorityClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#priorityClasses-patches-expression "Direct link to priorityClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#priorityClasses-patches-reverseExpression "Direct link to priorityClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#priorityClasses-patches-reference "Direct link to priorityClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#priorityClasses-patches-reference-apiVersion "Direct link to priorityClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#priorityClasses-patches-reference-apiVersionPath "Direct link to priorityClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#priorityClasses-patches-reference-kind "Direct link to priorityClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#priorityClasses-patches-reference-kindPath "Direct link to priorityClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#priorityClasses-patches-reference-namePath "Direct link to priorityClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#priorityClasses-patches-reference-namespacePath "Direct link to priorityClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#priorityClasses-patches-labels "Direct link to priorityClasses-patches-labels")Labels treats the path value as a labels selector. ### `selector` object [​](#priorityClasses-selector "Direct link to priorityClasses-selector")Selector defines the selector to use for the resource. If not set, all resources of that type will be synced. #### `matchLabels` object [​](#priorityClasses-selector-matchLabels "Direct link to priorityClasses-selector-matchLabels") #### `matchExpressions` object\[] [​](#priorityClasses-selector-matchExpressions "Direct link to priorityClasses-selector-matchExpressions") ##### `key` string [​](#priorityClasses-selector-matchExpressions-key "Direct link to priorityClasses-selector-matchExpressions-key") ##### `operator` string [​](#priorityClasses-selector-matchExpressions-operator "Direct link to priorityClasses-selector-matchExpressions-operator") ##### `values` string\[] [​](#priorityClasses-selector-matchExpressions-values "Direct link to priorityClasses-selector-matchExpressions-values") --- # Runtime classes Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. RuntimeClass syncing allows tenant clusters to use runtime classes from the control plane cluster. RuntimeClass resources define container runtime configurations including security sandboxes, alternative runtimes like gVisor or Kata Containers, and runtime-specific settings. Tenant clusters are isolated from each other; a RuntimeClass created in one tenant cluster does not exist in another. For more details on enabling and disabling synced resources, see the [sync configuration documentation](https://www.vcluster.com/docs/syncer/config#enable-or-disable-synced-resources). You can enable this feature to sync RuntimeClass resources from the control plane cluster to the tenant cluster. Add labels to RuntimeClass resources in your control plane cluster and configure vCluster to sync only those matching specific selectors. When you create a Pod that references a synced RuntimeClass, the control plane cluster uses that runtime configuration. This approach provides centralized control over runtime configurations and ensures consistent container runtime behavior across tenant clusters. Common use cases include: * **Security isolation**: Control which security runtimes different teams can access while sharing a single control plane cluster. Only authorized workloads can use enhanced security runtimes such as gVisor or Kata Containers. This prevents unauthorized access to privileged runtime configurations. * **Development environments**: Sync only required RuntimeClasses to reduce noise and avoid unnecessary exposure to production runtime configurations. * **Tenant isolation**: Enable teams to use specific runtime configurations while preventing access to incompatible or resource-intensive runtimes that could affect other tenants. Each team gets only the runtime classes appropriate for their workloads and security requirements. * **Compliance**: Maintain strict control over container runtime configurations by defining runtime classes once in the control plane cluster and making only approved runtime configurations available to tenant clusters. All workloads use compliant runtime settings. * **Resource optimization**: Prevent teams from using resource-intensive runtime configurations that could impact cluster performance without proper oversight. Access to high-overhead runtimes is restricted and used only when necessary. ## Enable syncing[​](#enable-syncing "Direct link to Enable syncing") When enabled, vCluster takes control of all RuntimeClass resources in the tenant cluster. It only allows RuntimeClass resources that are synced from the control plane cluster to exist. This example configuration enables syncing **all** RuntimeClass resources from the control plane cluster to the tenant cluster: ``` sync: fromHost: runtimeClasses: enabled: true ``` If you try to create the RuntimeClass resource directly in the tenant cluster, using for example `kubectl create` or `kubectl apply`, vCluster detects it and deletes it immediately. This prevents conflicts between locally created RuntimeClass resources and those synced from the control plane cluster, ensuring that only RuntimeClass resources from the control plane cluster exist in the tenant cluster. RuntimeClass resource limitation When enabled, RuntimeClass resource creation can only occur in the control plane cluster. Any RuntimeClass resources created in the tenant cluster will be deleted. ## How syncing works[​](#how-syncing-works "Direct link to How syncing works") When RuntimeClass syncing is enabled, vCluster can use a label selector to control which RuntimeClass and Pod resources are synchronized between the control plane and tenant clusters. This affects the resource types with separate unidirectional sync flows: * **RuntimeClass resources (control plane → tenant)**: vCluster syncs RuntimeClass resources from the control plane cluster to the tenant cluster. You cannot create RuntimeClass resources directly in the tenant cluster. If a selector is defined, only RuntimeClass resources matching the selector will sync. If no selector is defined, **all** RuntimeClass resources are synced. * **Pod resources (tenant → control plane)**: vCluster syncs Pod resources from the tenant cluster to the control plane cluster only if any of the following is true: * No selector is defined for RuntimeClass syncing, which allows all Pod resources to sync regardless of the `runtimeClassName`. * The Pod's `runtimeClassName` references a RuntimeClass resource that matches the selector and is synced from the control plane cluster. * The Pod has an empty `runtimeClassName` field. The same selector controls both sync flows. The selector determines which RuntimeClass resources are imported from the control plane cluster and which Pod resources can sync to the control plane cluster based on the referenced RuntimeClass resource. ## Use selectors to filter[​](#use-selectors-to-filter "Direct link to Use selectors to filter") Selectors provide precise control over which RuntimeClass resources get synced from the control plane cluster and which Pod resources gets synced to the control plane cluster. vCluster supports two types of selector criteria that follow standard [Kubernetes label selector syntax](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements). ### Filter with matchLabels[​](#filter-with-matchlabels "Direct link to Filter with matchLabels") The `matchLabels` selector defines **exact** label key-value pairs that must be present on an RuntimeClass for it to be synced. This provides straightforward filtering based on specific labels. The following example syncs only RuntimeClass resources with a `environment: development` label: Example with matchLabels ``` sync: fromHost: runtimeClasses: enabled: true selector: matchLabels: environment: development ``` ### Filter with matchExpressions[​](#filter-with-matchexpressions "Direct link to Filter with matchExpressions") The `matchExpressions` selector allows more flexible, set-based filtering with support for multiple operators: * `In`: Select all resources that has a specific key and the value is in the set of values * `NotIn`: Select all resources that has a specific key and the value is not in the set of values * `Exists`: Select all resources including a label with the key; no values are checked * `DoesNotExist`: Select all resources without a label with the key; no values are checked The following example syncs RuntimeClass resources where the key of the label is `kubernetes.io/runtime.class` and the value of that label is either `gvisor` or `youki`: Example with matchExpressions ``` sync: fromHost: runtimeClasses: enabled: true selector: matchExpressions: - key: kubernetes.io/runtime.class operator: In values: - gvisor - youki ``` ### Combined filter criteria[​](#combined-filter-criteria "Direct link to Combined filter criteria") Label conditions are additive meaning all specified label conditions must match for the RuntimeClass resource to be selected to be synced. This means that if you define multiple `matchLabels` and `matchExpressions`, the RuntimeClass resource must satisfy all criteria to be synced to the tenant cluster. The following example syncs RuntimeClass resources that match both label and expression criteria: Example with matchLabels and matchExpressions ``` sync: fromHost: runtimeClasses: enabled: true selector: matchLabels: environment: development matchExpressions: - key: kubernetes.io/runtime.class operator: In values: - gvisor - youki ``` In this example, the RuntimeClass must have the following labels to sync to the tenant cluster: * `environment: development` * Either `kubernetes.io/runtime.class:gvisor` or `kubernetes.io/runtime.class:youki` ## Sync behavior considerations[​](#sync-behavior-considerations "Direct link to Sync behavior considerations") ### Resource lifecycle[​](#resource-lifecycle "Direct link to Resource lifecycle") Synced RuntimeClass resources function like any other Kubernetes resource in the tenant cluster. You can view them with `kubectl get runtimeclass` and reference them in your Pod resource specifications with the `runtimeClassName` field. When you modify the RuntimeClass resource in the control plane cluster, vCluster re-evaluates whether it still matches the selector criteria. If the RuntimeClass continues to match, vCluster updates the corresponding resource in the tenant cluster to reflect the changes. If the RuntimeClass no longer matches the selector criteria, vCluster removes it from the tenant cluster. The selector system acts as both a resource filter and a validation mechanism. As a resource filter, it ensures that vCluster creates only RuntimeClass resources matching the defined selector criteria. The selector also functions as a creation validation mechanism - when you create Pod resources in the tenant cluster, the resources can only reference RuntimeClass resources that exist in the tenant cluster. ### Orphaned resources[​](#orphaned-resources "Direct link to Orphaned resources") When vCluster removes a synced RuntimeClass from the tenant cluster due to selector changes or deletion from the control plane cluster, any Pod resources that reference it remain in the tenant and control plane clusters. These orphaned Pod resources stop receiving updates but vCluster does not automatically delete them to prevent unintended data loss. To remove these orphaned resources, you must delete them manually in the tenant and control plane clusters. This manual approach ensures that you maintain full control over resource cleanup and can verify that deletions are intentional. ### Error handling and troubleshooting[​](#error-handling-and-troubleshooting "Direct link to Error handling and troubleshooting") When the RuntimeClass resource doesn't match the selector criteria during evaluation, vCluster logs a warning in the syncer pod's output to help with troubleshooting and monitoring. This logging provides visibility into which resources are being filtered out and why. When you creates the Pod resource that references a RuntimeClass not matching the selector criteria, several things occur to provide clear feedback: * The Pod fails to sync to the control plane cluster * vCluster records an event on the Pod resource in the tenant cluster * The event indicates that the specified RuntimeClass is not available according to the current selector configuration The error output appears as a Kubernetes event that you can view using `kubectl describe`. The event message clearly states that the Pod resource was not synced because the referenced RuntimeClass does not match the defined selector criteria. The error output looks like this: Example error handling output ``` vcluster-virtual-cluster-1:~$ kubectl describe pod my-pod Name: my-pod Namespace: default RuntimeClass: youki Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning SyncWarning 10s pod-syncer did not sync pod "my-pod" to host because it does not match the selector under 'sync.fromHost.runtimeClasses.selector' ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `runtimeClasses` object [​](#runtimeClasses "Direct link to runtimeClasses")RuntimeClasses defines if runtime classes should get synced from the host cluster to the virtual cluster, but not back. ### `enabled` boolean false [​](#runtimeClasses-enabled "Direct link to runtimeClasses-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#runtimeClasses-patches "Direct link to runtimeClasses-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#runtimeClasses-patches-path "Direct link to runtimeClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#runtimeClasses-patches-expression "Direct link to runtimeClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#runtimeClasses-patches-reverseExpression "Direct link to runtimeClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#runtimeClasses-patches-reference "Direct link to runtimeClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#runtimeClasses-patches-reference-apiVersion "Direct link to runtimeClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#runtimeClasses-patches-reference-apiVersionPath "Direct link to runtimeClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#runtimeClasses-patches-reference-kind "Direct link to runtimeClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#runtimeClasses-patches-reference-kindPath "Direct link to runtimeClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#runtimeClasses-patches-reference-namePath "Direct link to runtimeClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#runtimeClasses-patches-reference-namespacePath "Direct link to runtimeClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#runtimeClasses-patches-labels "Direct link to runtimeClasses-patches-labels")Labels treats the path value as a labels selector. ### `selector` object [​](#runtimeClasses-selector "Direct link to runtimeClasses-selector")Selector defines the selector to use for the resource. If not set, all resources of that type will be synced. #### `matchLabels` object [​](#runtimeClasses-selector-matchLabels "Direct link to runtimeClasses-selector-matchLabels") #### `matchExpressions` object\[] [​](#runtimeClasses-selector-matchExpressions "Direct link to runtimeClasses-selector-matchExpressions") ##### `key` string [​](#runtimeClasses-selector-matchExpressions-key "Direct link to runtimeClasses-selector-matchExpressions-key") ##### `operator` string [​](#runtimeClasses-selector-matchExpressions-operator "Direct link to runtimeClasses-selector-matchExpressions-operator") ##### `values` string\[] [​](#runtimeClasses-selector-matchExpressions-values "Direct link to runtimeClasses-selector-matchExpressions-values") --- # Secrets Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is turned off. No need to configure RBAC vCluster automatically adds the required cluster RBAC permissions for retrieving the Secrets and syncing the resources from the control plane cluster to the tenant cluster. Enabling sync from host allows you to sync Secrets from the specified namespaces in the control plane cluster to the specified namespaces in a tenant cluster. Here is how the required configuration in `vcluster.yaml` looks like: configure Secret sync from host ``` sync: fromHost: secrets: enabled: true mappings: byName: # syncs all Secrets from "foo" namespace # to the "bar" namespace in a virtual cluster. Secrets names are unchanged. "foo/*": "bar/*" ``` Here are a few things to remember when configuring the sync: * It is also possible to modify the name of the synced resource in the tenant cluster. * There is no option to sync from all namespaces in the control plane cluster. * Sync is one-directional, from the control plane cluster to the tenant cluster. If you modify an object in the control plane cluster, vCluster syncs the change to virtual object. * When you delete a virtual object, vCluster re-creates it if the host object still exist. * When you delete a host object, vCluster deletes the corresponding virtual object. * It is not possible to sync Secrets that were already synced from virtual to host, they are skipped by vCluster. Namespaces in the tenant cluster are created automatically during the sync (if they do not exist already). You can use synced Secrets in your workloads as an environment variables source or a volume. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Run the command `kubectl auth can-i create clusterrole -A` to verify that your current kube-context has administrative privileges. info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm`: Helm v3.10 is required for deploying the platform. Refer to the [Helm Installation Guide](https://Helm.sh/docs/intro/install/) if you need to install it. * `kubectl`: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. All the specified namespaces have to exist in the host at the vCluster startup. ## Sync all Secrets from host namespace[​](#sync-all-secrets-from-host-namespace "Direct link to Sync all Secrets from host namespace") To sync all Secrets from a given namespace in the control plane cluster to the given namespace in the tenant cluster, use “namespace/\*” wildcard, e.g.: configure Secret sync from host namespace ``` sync: fromHost: secrets: enabled: true mappings: byName: # syncs all Secrets from "foo" namespace # to the "bar" namespace in a virtual cluster. Secrets names are unchanged. "foo/*": "bar/*" ``` ## Sync only specific Secrets from host namespace[​](#sync-only-specific-secrets-from-host-namespace "Direct link to Sync only specific Secrets from host namespace") To sync only specific Secrets from namespaces, you need to provide `namespace/name` as the key and value: configure Secret sync from host for one object ``` sync: fromHost: secrets: enabled: true mappings: byName: # syncs Secret named "cm-name" from "foo" host namespace # to the "bar" namespace in virtual. "foo/cm-name": "bar/cm-name" ``` ## Sync all Secrets from the tenant cluster’s control plane cluster namespace[​](#sync-all-secrets-from-the-tenant-clusters-control-plane-cluster-namespace "Direct link to Sync all Secrets from the tenant cluster’s control plane cluster namespace") There is also a handy syntax to sync all Secrets from the tenant cluster’s own control plane cluster namespace to the tenant namespace. As the tenant cluster’s namespace is not always known upfront (e.g. when the tenant cluster is created by the platform), `””` (empty string) is treated as “tenant cluster’s own control plane cluster namespace”. configure Secret sync from host for tenant cluster's namespace ``` sync: fromHost: secrets: enabled: true mappings: byName: # syncs all Secrets from virtual cluster's host namespace # to "my-virtual" namespace in a virtual cluster. "": "my-virtual" ``` ## Sync specific Secret from the tenant cluster’s control plane cluster namespace[​](#sync-specific-secret-from-the-tenant-clusters-control-plane-cluster-namespace "Direct link to Sync specific Secret from the tenant cluster’s control plane cluster namespace") you can also specify only a few Secrets from the tenant cluster’s own control plane cluster namespace this way: configure Secret sync from host for objects in tenant cluster's namespace ``` sync: fromHost: secrets: enabled: true mappings: byName: # syncs Secret named "my-cm" from virtual cluster's host namespace # to "my-virtual-namespace" in a virtual cluster. "/my-cm": "my-virtual/my-cm" ``` ## Modify synced Secret namespace and name in the tenant cluster[​](#modify-synced-secret-namespace-and-name-in-the-tenant-cluster "Direct link to Modify synced Secret namespace and name in the tenant cluster") It’s also possible to modify Secret name during the sync: configure Secret sync from host and modify name and namespace ``` sync: fromHost: secrets: enabled: true mappings: byName: # syncs "config" Secret from "system" namespace in the host # as "my-config" in "my-virtual" namespace in a virtual cluster. "system/config": "my-virtual/my-config" ``` ## Patches[​](#patches "Direct link to Patches") Use `sync.fromHost.secrets.patches` to transform Secret fields while syncing from the control plane cluster to the tenant cluster. For from-host sync, use `reverseExpression` for values that should appear in the tenant cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax and directionality. configure Secret sync from host with patches ``` sync: fromHost: secrets: enabled: true mappings: byName: "default/my-cm": "barfoo2/cm-my" patches: - path: metadata.annotations[*] # optional reverseExpression to reverse the change from the host cluster reverseExpression: "value.startsWith('www.') ? value.slice(4) : value" ``` ## From host Secret sync example[​](#from-host-secret-sync-example "Direct link to From host Secret sync example") This guide shows how to sync Kubernetes Secret from control plane clusters and how you can use them in your workload running inside tenant clusters. ### Set up cluster contexts[​](#set-up-cluster-contexts "Direct link to Set up cluster contexts") Setting up the control plane and tenant cluster contexts makes it easier to switch between them. set up kubectl contexts ``` export HOST_CTX="your-host-context" export VCLUSTER_CTX="vcluster-ctx" ``` then, create a namespace in your control plane cluster, use `foobar` as an example: create namespace ``` kubectl --context="${HOST_CTX}" create namespace foobar ``` tip You can find your contexts by running `kubectl config get-contexts` ### Enable from host syncing for Secret[​](#enable-from-host-syncing-for-secret "Direct link to Enable from host syncing for Secret") Enable the from host syncing for secrets in your tenant cluster configuration: Enable from host syncing for a Secret ``` sync: fromHost: secrets: enabled: true mappings: byName: "foobar/shared-user-env": "my-namespace/user-env" ``` This configuration: * Enables from host syncing of the Secret `shared-user-env` in the namespace `foobar`. * Automatically configures RBAC permissions for vCluster, so it can access this Secret (you need to re-deploy vCluster for it to take place) * Makes this Secret accessible as `user-env` in the `my-namespace` in your vCluster. create tenant cluster Create or update a tenant cluster following the [vCluster quick start guide](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md). ### Sync Secret to tenant cluster and use it in pod[​](#sync-secret-to-tenant-cluster-and-use-it-in-pod "Direct link to Sync Secret to tenant cluster and use it in pod") 1. First, create a Secret which you want to sync in the control plane cluster: Create Secret in the host ``` kubectl --context="${HOST_CTX}" create secret generic shared-user-env \ --namespace=foobar \ --from-literal=MY_ENV_1=foo \ --from-literal=MY_ENV_2=bar ``` 2. #### Ensure that Secret got synced to the tenant cluster[​](#ensure-that-secret-got-synced-to-the-tenant-cluster "Direct link to Ensure that Secret got synced to the tenant cluster") Your Secret should be now accessible in the tenant cluster. Keep in mind, that any edit made in the virtual object is overwritten by the host object data. Check if the Secret is accessible in the tenant cluster: Get synced Secret ``` kubectl --context="${VCLUSTER_CTX}" get secrets --namespace my-namespace user-env -o yaml ``` you should see similar output: Secret contents ``` apiVersion: v1 data: MY_ENV_1: Zm9v MY_ENV_2: YmFy kind: Secret metadata: creationTimestamp: "2025-02-17T12:12:59Z" name: user-env namespace: my-namespace resourceVersion: "15919" uid: 26cc03d6-5cde-4090-9f9d-3e6fcbeefd24 type: Opaque ``` 3. #### Use it in your workload[​](#use-it-in-your-workload "Direct link to Use it in your workload") Now, you can create a new pod in the `my-namespace` namespace and specify this Secret as a source for environment variables. Save this pod locally to the file called `pod.yaml`: pod.yaml ``` apiVersion: v1 kind: Pod metadata: name: my-pod namespace: my-namespace spec: containers: - name: busybox image: busybox command: - sleep - "inf" env: # Plain Text ENV - name: DEMO_GREETING value: "Hello from the environment" envFrom: - secretRef: name: user-env ``` then, create it in the tenant cluster: Create pod ``` kubectl --context="${VCLUSTER_CTX}" create -f pod.yaml ``` 4. #### Verify pod environment variables[​](#verify-pod-environment-variables "Direct link to Verify pod environment variables") Tenant Cluster Wait for pod running Wait for pod running ``` kubectl --context="${VCLUSTER_CTX}" wait --for=condition=ready pod my-pod --namespace my-namespace --timeout=300s ``` once it is running, you can list pod's environment variables: Check environment variables ``` kubectl --context="${VCLUSTER_CTX}" exec -it --namespace my-namespace my-pod -- printenv | grep "MY_ENV" ``` you should see environment successfully injected from the Secret: Environment variables from Secret ``` MY_ENV_1=foo MY_ENV_2=bar ``` 5. #### Summary[​](#summary "Direct link to Summary") From host Secret syncing allow you to make specific Secret(s) from control plane clusters accessible inside your tenant clusters. You can make them accessible from different namespaces and/or with different names in the tenant cluster. They are always synced from the control plane to the tenant cluster, so it is also possible to sync one control plane cluster Secret to multiple tenant cluster ones. They can also be used as a volume or env source in your workloads. ## Config reference[​](#config-reference "Direct link to Config reference") ## `secrets` object [​](#secrets "Direct link to secrets")Secrets defines if secrets in the host should get synced to the virtual cluster. ### `enabled` boolean false [​](#secrets-enabled "Direct link to secrets-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#secrets-patches "Direct link to secrets-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#secrets-patches-path "Direct link to secrets-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#secrets-patches-expression "Direct link to secrets-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#secrets-patches-reverseExpression "Direct link to secrets-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#secrets-patches-reference "Direct link to secrets-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#secrets-patches-reference-apiVersion "Direct link to secrets-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#secrets-patches-reference-apiVersionPath "Direct link to secrets-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#secrets-patches-reference-kind "Direct link to secrets-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#secrets-patches-reference-kindPath "Direct link to secrets-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#secrets-patches-reference-namePath "Direct link to secrets-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#secrets-patches-reference-namespacePath "Direct link to secrets-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#secrets-patches-labels "Direct link to secrets-patches-labels")Labels treats the path value as a labels selector. ### `mappings` object [​](#secrets-mappings "Direct link to secrets-mappings")Mappings for Namespace and Object #### `byName` object {} [​](#secrets-mappings-byName "Direct link to secrets-mappings-byName")ByName is a map of control-plane-object-namespace/control-plane-object-name: tenant-object-namespace/tenant-object-name. There are several wildcards supported:1) To match all objects in a control plane namespace and sync them to a different namespace in the tenant cluster: byName: "foo/*": "foo-in-virtual/*" 2) To match a specific object in the control plane namespace and sync it to the same namespace with the same name: byName: "foo/my-object": "foo/my-object" 3) To match a specific object in the control plane namespace and sync it to the same namespace with a different name: byName: "foo/my-object": "foo/my-virtual-object" 4) To match all objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "": "my-virtual-namespace/\*" 5) To match specific objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "/my-object": "my-virtual-namespace/my-object" --- # Storage classes Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. StorageClass resources define how tenant clusters provision storage. They specify storage backends such as AWS EBS, Azure Disk, or GCP Persistent Disk, along with parameters such as performance tiers and replication settings. By default, each tenant cluster needs its own StorageClass resources. StorageClass syncing allows tenant clusters to use StorageClass resources from the control plane cluster instead. You can enable this feature to sync StorageClass resources from the control plane cluster to the tenant cluster. Add labels to StorageClass resources in your control plane cluster and configure vCluster to sync only those matching specific selectors. When you create a PersistentVolumeClaim that references a synced StorageClass, the control plane cluster uses that storage configuration. Common use cases include: * **Cost control**: Restrict access to specific storage classes based on cost or performance characteristics. * **Compliance**: Control which storage backends and configurations are available to tenant clusters. * **Tenant isolation**: Provide different storage options to different teams while preventing access to restricted storage types. * **Simplified management**: Update storage class configurations in the control plane cluster rather than in each tenant cluster. * **Security**: Control access to storage backends and encryption options. ## Enable syncing[​](#enable-syncing "Direct link to Enable syncing") When enabled, vCluster takes control of all StorageClass resources in the tenant cluster. It only allows StorageClass resources that are synced from the control plane cluster to exist. This example configuration enables syncing **all** StorageClass resources from the control plane cluster to the tenant cluster: ``` sync: fromHost: storageClasses: enabled: true ``` If you try to create the StorageClass resource directly in the tenant cluster, using for example `kubectl create` or `kubectl apply`, vCluster detects it and deletes it immediately. This prevents conflicts between locally created StorageClass resources and those synced from the control plane cluster, ensuring that only StorageClass resources from the control plane cluster exist in the tenant cluster. StorageClass resource limitation When enabled, StorageClass resource creation can only occur in the control plane cluster. Any StorageClass resources created in the tenant cluster will be deleted. ## How syncing works[​](#how-syncing-works "Direct link to How syncing works") When StorageClass syncing is enabled, vCluster can use a label selector to control which StorageClass, PersistentVolumeClaim and PersistentVolume resources are synchronized between the control plane and tenant clusters. This affects the resource types with separate unidirectional sync flows: * **StorageClass resources (control plane → tenant)**: vCluster syncs StorageClass resources from the control plane cluster to the tenant cluster. You cannot create StorageClass resources directly in the tenant cluster. If a selector is defined, only StorageClass resources matching the selector will sync. If no selector is defined, **all** StorageClass resources are synced. * **PersistentVolume and PersistentVolumeClaim resources (tenant → control plane)**: vCluster syncs PersistentVolume and PersistentVolumeClaim resources from the tenant cluster to the control plane cluster only if any of the following is true: * No selector is defined for StorageClass syncing, which allows all PersistentVolume and PersistentVolumeClaim resources to sync regardless of the `storageClassName`. * The PersistentVolume's or PersistentVolumeClaim's `storageClassName` references a StorageClass resource that matches the selector and is synced from the control plane cluster. * The PersistentVolume and PersistentVolumeClaim has an empty `storageClassName` field. The same selector controls both sync flows. The selector determines which StorageClass resources are imported from the control plane cluster and which PersistentVolume and PersistentVolumeClaim resources can sync to the control plane cluster based on the referenced StorageClass resource. ## Use selectors to filter[​](#use-selectors-to-filter "Direct link to Use selectors to filter") Selectors provide precise control over which StorageClass resources get synced from the control plane cluster and which PersistentVolume and PersistentVolumeClaim resources gets synced to the control plane cluster. vCluster supports two types of selector criteria that follow standard [Kubernetes label selector syntax](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements). ### Filter with matchLabels[​](#filter-with-matchlabels "Direct link to Filter with matchLabels") The `matchLabels` selector defines **exact** label key-value pairs that must be present on an StorageClass for it to be synced. This provides straightforward filtering based on specific labels. The following example syncs only StorageClass resources with a `environment: development` label: Example with matchLabels ``` sync: toHost: persistentVolumes: enabled: true persistentVolumeClaims: enabled: true fromHost: storageClasses: enabled: true selector: matchLabels: environment: development ``` ### Filter with matchExpressions[​](#filter-with-matchexpressions "Direct link to Filter with matchExpressions") The `matchExpressions` selector allows more flexible, set-based filtering with support for multiple operators: * `In`: Select all resources that has a specific key and the value is in the set of values * `NotIn`: Select all resources that has a specific key and the value is not in the set of values * `Exists`: Select all resources including a label with the key; no values are checked * `DoesNotExist`: Select all resources without a label with the key; no values are checked The following example syncs StorageClass resources where the key of the label is `kubernetes.io/storage.class` and the value of that label is either `network-nas` or `fast-sad`: Example with matchExpressions ``` sync: toHost: persistentVolumes: enabled: true persistentVolumeClaims: enabled: true fromHost: storageClasses: enabled: true selector: matchExpressions: - key: kubernetes.io/storage.class operator: In values: - network-nas - fast-sad ``` ### Combined filter criteria[​](#combined-filter-criteria "Direct link to Combined filter criteria") Label conditions are additive meaning all specified label conditions must match for the StorageClass resource to be selected to be synced. This means that if you define multiple `matchLabels` and `matchExpressions`, the StorageClass resource must satisfy all criteria to be synced to the tenant cluster. The following example syncs StorageClass resources that match both label and expression criteria: Example with matchLabels and matchExpressions ``` sync: toHost: persistentVolumes: enabled: true persistentVolumeClaims: enabled: true fromHost: storageClasses: enabled: true selector: matchLabels: environment: development matchExpressions: - key: kubernetes.io/storage.class operator: In values: - network-nas - fast-sad ``` In this example, the StorageClass must have the following labels to sync to the tenant cluster: * `environment: development` * Either `kubernetes.io/storage.class:network-nas` or `kubernetes.io/storage.class:fast-sad` ## Sync behavior considerations[​](#sync-behavior-considerations "Direct link to Sync behavior considerations") ### Resource lifecycle[​](#resource-lifecycle "Direct link to Resource lifecycle") Synced StorageClass resources function like any other Kubernetes resource in the tenant cluster. You can view them with `kubectl get storageclass` and reference them in your PersistentVolume and PersistentVolumeClaim resource specifications with the `storageClassName` field. When you modify the StorageClass resource in the control plane cluster, vCluster re-evaluates whether it still matches the selector criteria. If the StorageClass continues to match, vCluster updates the corresponding resource in the tenant cluster to reflect the changes. If the StorageClass no longer matches the selector criteria, vCluster removes it from the tenant cluster. The selector system acts as both a resource filter and a validation mechanism. As a resource filter, it ensures that vCluster creates only StorageClass resources matching the defined selector criteria. The selector also functions as a creation validation mechanism - when you create PersistentVolume and PersistentVolumeClaim resources in the tenant cluster, the resources can only reference StorageClass resources that exist in the tenant cluster. ### Orphaned resources[​](#orphaned-resources "Direct link to Orphaned resources") When vCluster removes a synced StorageClass from the tenant cluster due to selector changes or deletion from the control plane cluster, any PersistentVolume and PersistentVolumeClaim resources that reference it remain in the tenant and control plane clusters. These orphaned PersistentVolume and PersistentVolumeClaim resources stop receiving updates but vCluster does not automatically delete them to prevent unintended data loss. To remove these orphaned resources, you must delete them manually in the tenant and control plane clusters. This manual approach ensures that you maintain full control over resource cleanup and can verify that deletions are intentional. ### Error handling and troubleshooting[​](#error-handling-and-troubleshooting "Direct link to Error handling and troubleshooting") When the StorageClass resource doesn't match the selector criteria during evaluation, vCluster logs a warning in the syncer pod's output to help with troubleshooting and monitoring. This logging provides visibility into which resources are being filtered out and why. When you creates the PersistentVolume or PersistentVolumeClaim resource that references a StorageClass not matching the selector criteria, several things occur to provide clear feedback: * The PersistentVolume or PersistentVolumeClaim fails to sync to the control plane cluster * vCluster records an event on the PersistentVolume or PersistentVolumeClaim resource in the tenant cluster * The event indicates that the specified StorageClass is not available according to the current selector configuration The error output appears as a Kubernetes event that you can view using `kubectl describe`. The event message clearly states that the PersistentVolume or PersistentVolumeClaim resource was not synced because the referenced StorageClass does not match the defined selector criteria. The error output looks like this: Example error handling output ``` vcluster-virtual-cluster-1:~$ kubectl describe pvc my-pvc Name: my-pvc Namespace: default StorageClass: fast-sad Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning SyncWarning 10s pvc-syncer did not sync pvc "my-pvc" to host because it does not match the selector under 'sync.fromHost.storageClasses.selector' ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `storageClasses` object [​](#storageClasses "Direct link to storageClasses")StorageClasses defines if storage classes should get synced from the host cluster to the virtual cluster, but not back. If auto, is automatically enabled when the virtual scheduler is enabled. ### `enabled` string|boolean auto [​](#storageClasses-enabled "Direct link to storageClasses-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#storageClasses-patches "Direct link to storageClasses-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#storageClasses-patches-path "Direct link to storageClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#storageClasses-patches-expression "Direct link to storageClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#storageClasses-patches-reverseExpression "Direct link to storageClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#storageClasses-patches-reference "Direct link to storageClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#storageClasses-patches-reference-apiVersion "Direct link to storageClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#storageClasses-patches-reference-apiVersionPath "Direct link to storageClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#storageClasses-patches-reference-kind "Direct link to storageClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#storageClasses-patches-reference-kindPath "Direct link to storageClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#storageClasses-patches-reference-namePath "Direct link to storageClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#storageClasses-patches-reference-namespacePath "Direct link to storageClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#storageClasses-patches-labels "Direct link to storageClasses-patches-labels")Labels treats the path value as a labels selector. ### `selector` object [​](#storageClasses-selector "Direct link to storageClasses-selector")Selector defines the selector to use for the resource. If not set, all resources of that type will be synced. #### `matchLabels` object [​](#storageClasses-selector-matchLabels "Direct link to storageClasses-selector-matchLabels") #### `matchExpressions` object\[] [​](#storageClasses-selector-matchExpressions "Direct link to storageClasses-selector-matchExpressions") ##### `key` string [​](#storageClasses-selector-matchExpressions-key "Direct link to storageClasses-selector-matchExpressions-key") ##### `operator` string [​](#storageClasses-selector-matchExpressions-operator "Direct link to storageClasses-selector-matchExpressions-operator") ##### `values` string\[] [​](#storageClasses-selector-matchExpressions-values "Direct link to storageClasses-selector-matchExpressions-values") --- # Volume snapshot classes Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. Sync [VolumeSnapshotClass](https://kubernetes.io/docs/concepts/storage/volume-snapshots/#volumesnapshotclasses) resources from the control plane cluster to the tenant cluster. A VolumeSnapshotClass tells a CSI driver how to take a snapshot, similar to how a StorageClass tells it how to provision a volume. A [VolumeSnapshot](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/volume-snapshots.md) created in the tenant cluster normally references a VolumeSnapshotClass by name. Sync the classes that exist in the control plane cluster before enabling VolumeSnapshot syncing. ### Sync VolumeSnapshotClasses from the control plane cluster to the tenant cluster[​](#sync-volumesnapshotclasses-from-the-control-plane-cluster-to-the-tenant-cluster "Direct link to Sync VolumeSnapshotClasses from the control plane cluster to the tenant cluster") ``` sync: fromHost: volumeSnapshotClasses: enabled: true ``` ## Patches[​](#patches "Direct link to Patches") Use `sync.fromHost.volumeSnapshotClasses.patches` to transform VolumeSnapshotClass fields while syncing to the tenant cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") ## `volumeSnapshotClasses` object [​](#volumeSnapshotClasses "Direct link to volumeSnapshotClasses")VolumeSnapshotClasses defines if volume snapshot classes created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean false [​](#volumeSnapshotClasses-enabled "Direct link to volumeSnapshotClasses-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#volumeSnapshotClasses-patches "Direct link to volumeSnapshotClasses-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#volumeSnapshotClasses-patches-path "Direct link to volumeSnapshotClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#volumeSnapshotClasses-patches-expression "Direct link to volumeSnapshotClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#volumeSnapshotClasses-patches-reverseExpression "Direct link to volumeSnapshotClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#volumeSnapshotClasses-patches-reference "Direct link to volumeSnapshotClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#volumeSnapshotClasses-patches-reference-apiVersion "Direct link to volumeSnapshotClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#volumeSnapshotClasses-patches-reference-apiVersionPath "Direct link to volumeSnapshotClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#volumeSnapshotClasses-patches-reference-kind "Direct link to volumeSnapshotClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#volumeSnapshotClasses-patches-reference-kindPath "Direct link to volumeSnapshotClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#volumeSnapshotClasses-patches-reference-namePath "Direct link to volumeSnapshotClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#volumeSnapshotClasses-patches-reference-namespacePath "Direct link to volumeSnapshotClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#volumeSnapshotClasses-patches-labels "Direct link to volumeSnapshotClasses-patches-labels")Labels treats the path value as a labels selector. --- # Patching | | | Enterprise | | | | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Sync Patches](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Use patches when a resource needs different field values on each side of the sync boundary. For example, you can rewrite image references to a private registry, or rewrite fields that contain the name of another synced object that vCluster translates. Without a patch, vCluster applies its built-in translations and otherwise leaves synced fields unchanged. Patches are translation rules. When a resource changes, vCluster rewrites the fields that match your patch configuration and writes the result to the target cluster. Patch lists live under the resource's sync direction. Use `sync.toHost..patches` for resources synced from the tenant cluster to the control plane cluster, and `sync.fromHost..patches` for resources synced from the control plane cluster into the tenant cluster. Sync patches rewrite matched fields as changes move between the tenant and control plane clusters ## Path syntax[​](#path-syntax "Direct link to Path syntax") A path identifies the field in the Kubernetes object the patch targets. Patch paths use a dot notation. ### Bracket notation[​](#bracket-notation "Direct link to Bracket notation") Use bracket notation for map keys that contain `.`, `[`, or `]`. ``` patches: - path: metadata.annotations["example.com/key"] expression: '"patched-" + value' ``` ### All entries[​](#all-entries "Direct link to All entries") Use `[*]` to match all elements of an array or all keys of a map. `*` does not match patterns or globs. Rather, it selects every entry at that position in the path. vCluster runs the patch once for each. You can use `[*]` multiple times in a single path. `spec.containers[*].volumeMounts[*].mountPath` applies the patch to every `mountPath` across every container. ``` patches: - path: spec.containers[*].env[*].value expression: 'value.replace("internal.svc", "internal.svc.cluster.local")' ``` Omitting expression on an array-nested path Omitting `expression` or `reverseExpression` drops that value from the patch, which normally leaves the target's existing value alone. Arrays are the exception. JSON Merge Patch replaces an array wholesale whenever it appears in the patch. If the array shows up in the patch for any other reason, dropping one nested value doesn't preserve the rest of the target's array. It overwrites the whole array without them. Remove the array's own key from the patch instead of a value nested inside it, if you need the target's array left untouched. ## Patch modes[​](#patch-modes "Direct link to Patch modes") Choose the mode based on what the field contains: | To transform | Use | | --------------------------------------------- | ---------------------------------- | | A field value | `expression` / `reverseExpression` | | A field referencing another Kubernetes object | `reference` | | Labels on a custom resource | `labels` | You can use `expression` and `reverseExpression` together in one entry. ## Validation rules[​](#validation-rules "Direct link to Validation rules") * Each patch requires a `path`. * Each patch uses exactly one mode. `expression` and `reverseExpression` can coexist in one entry, but neither can be combined with `reference` or `labels`. * A resource's patch list cannot contain duplicate paths. * `metadata.name` and `metadata.namespace` are not valid patch paths. * Reference patches require a vCluster mapper for the referenced resource type. ## Expression patches[​](#expression-patches "Direct link to Expression patches") Expression patches run JavaScript against each matched value. The expression receives `value` as the current field value and its return value replaces it. For `sync.toHost`, `expression` transforms changes from the tenant cluster to the control plane cluster. `reverseExpression` transforms changes from the control plane cluster back to the tenant cluster. If you omit `expression`, vCluster drops that field from the patch when syncing to the control plane cluster. If you omit `reverseExpression`, it drops the field when syncing back to the tenant cluster. Write only one to create a one-way transform. For `sync.fromHost`, the transform only happens in one direction, from the control plane cluster to the tenant cluster. For most fromHost resources, `expression` transforms values as they arrive from the control plane cluster. ConfigMaps, Secrets, and custom resources are exceptions that use `reverseExpression` for this direction. See the [Supported resources](#from-the-control-plane-cluster) table for which keyword each resource requires. If you configure the wrong keyword for a resource, vCluster silently skips the expression and syncs the field unchanged. Rewrite container images to a private registry ``` sync: toHost: pods: enabled: true patches: - path: spec.containers[*].image expression: 'value.replace("docker.io/", "registry.internal/")' ``` In this example, any container image from `docker.io` in the tenant cluster is rewritten when the Pod syncs to the control plane cluster. A Pod with `image: docker.io/nginx:latest` runs as `image: registry.internal/nginx:latest` on the control plane cluster. Prefix service port names when syncing to the control plane cluster ``` sync: toHost: services: enabled: true patches: - path: spec.ports[*].name expression: '"tenant-" + value' reverseExpression: 'value.replace(/^tenant-/, "")' ``` In this example, a port named `http` in the tenant cluster becomes `tenant-http` on the control plane cluster. When a change flows back from the control plane cluster, the prefix is stripped and the port is `http` again in the tenant cluster. Remove a prefix from imported ConfigMap annotations ``` sync: fromHost: configMaps: enabled: true mappings: byName: "default/my-cm": "tenant/my-cm" patches: - path: metadata.annotations[*] reverseExpression: 'value.startsWith("www.") ? value.slice(4) : value' ``` In this example, the ConfigMap `my-cm` from the `default` namespace on the control plane cluster is synced into the `tenant` namespace in the tenant cluster. For every annotation value, if the value starts with `www.`, that prefix is stripped in the tenant cluster. An annotation value of `www.example.com` on the control plane cluster appears as `example.com` in the tenant cluster. The control plane cluster object is unchanged. ### JavaScript runtime[​](#javascript-runtime "Direct link to JavaScript runtime") Expressions can use these variables: | Variable | Description | | ---------------------------- | --------------------------------------------------------- | | `value` | The matched value. | | `valueExists` | `true` when the matched path exists in the patch. | | `context.vcluster.name` | Tenant cluster name. | | `context.vcluster.namespace` | Tenant cluster namespace in the control plane cluster. | | `context.vcluster.config` | Effective `vcluster.yaml` configuration. | | `context.hostObject` | Control plane cluster object, or `null` when unavailable. | | `context.virtualObject` | Tenant cluster object, or `null` when unavailable. | | `context.path` | The concrete matched path, useful with wildcards. | Expressions can use these helper functions: | Function | Description | | ------------------------- | -------------------------------------------------------------------------------- | | `btoa(value)` | Base64-encode a string. | | `atob(value)` | Base64-decode a string. | | `virtualToHostDNS(value)` | Rewrite a tenant service DNS name to the control plane cluster service DNS name. | `virtualToHostDNS` supports service DNS names in the form `..svc` or `..svc.`. ### Empty paths[​](#empty-paths "Direct link to Empty paths") Empty paths are an advanced use case. Use them when you need to read multiple fields to decide what to change, or when you need to set multiple fields in a single expression. They're also the only way to set a field a path-specific patch can't reach, because that field isn't present in the change being synced. With a normal path, `value` is the single field the patch targets. With an empty path, `value` is the entire set of fields being changed in this sync operation. This is only the changed fields, not the full object. The expression must return the modified change set. ``` patches: - path: "" expression: | (value => { if (!value.metadata) value.metadata = {}; if (!value.metadata.annotations) value.metadata.annotations = {}; value.metadata.annotations["patched-by"] = "vcluster"; return value; })(value) ``` If the incoming change set is `{"spec": {"replicas": 3}}`, the expression receives that object, adds the annotation, and returns `{"spec": {"replicas": 3}, "metadata": {"annotations": {"patched-by": "vcluster"}}}`. Both changes are then applied together. ## Reference patches[​](#reference-patches "Direct link to Reference patches") Reference patches tell vCluster that a field contains the name of another synced object. When syncing, vCluster rewrites the name to its translated value in the target cluster. vCluster renames synced objects on the control plane cluster to avoid collisions across tenant clusters. A Secret named `my-secret` in the `default` namespace becomes `default-x-my-secret-x-my-vcluster` on the control plane cluster. Any field that holds that name must be rewritten to match, or the reference breaks. ### Simple reference patch[​](#simple-reference-patch "Direct link to Simple reference patch") Use a simple reference patch when the field value is a plain string holding only the object name. Treat an annotation value as a Secret name ``` sync: toHost: pods: enabled: true patches: - path: metadata.annotations["example.com/secret-name"] reference: apiVersion: v1 kind: Secret ``` If a Pod in the tenant cluster has the annotation `example.com/secret-name: my-secret`, vCluster rewrites it when syncing the Pod to the control plane cluster. On the control plane cluster, the annotation becomes `example.com/secret-name: default-x-my-secret-x-my-vcluster`. Without the patch, the annotation stays as `my-secret`, which would not match the actual Secret name on the control plane cluster. ### Structured reference patch[​](#structured-reference-patch "Direct link to Structured reference patch") Use a structured reference patch when the field value is a nested object that contains the name as a sub-field, such as a Kubernetes `ObjectReference` or `LocalObjectReference`. Rewrite a structured Secret reference ``` sync: toHost: customResources: widgets.example.com: enabled: true patches: - path: spec.secretRef reference: apiVersion: v1 kind: Secret namePath: name namespacePath: namespace ``` In the tenant cluster, `spec.secretRef` holds an object like: ``` name: my-secret namespace: default ``` `namePath: name` and `namespacePath: namespace` are paths relative to that object, pointing to the sub-fields vCluster should rewrite. On the control plane cluster, those values become the translated Secret name and namespace: ``` name: default-x-my-secret-x-my-vcluster namespace: my-vcluster-ns ``` The values you set for `namePath` and `namespacePath` depend on how the resource defines its reference. If the sub-field were named `secretName` instead of `name`, you would write `namePath: secretName`. `namePath` is required when you set `namespacePath`, `kindPath`, or `apiVersionPath`. ## Labels patches[​](#labels-patches "Direct link to Labels patches") Use `labels` patches only with custom resource sync. They are not supported on built-in resources. vCluster translates pod label keys when syncing to the control plane cluster to prevent collisions across tenant clusters. If a custom resource has a field containing label selectors, such as `spec.selector.matchLabels`, those keys must be translated the same way or the selector will not match any pods on the control plane cluster. Setting `labels: {}` on a path tells vCluster to apply that translation to the field. The `labels` key takes an empty object — there are no configuration options inside it. Translate selector labels on a custom resource ``` sync: toHost: customResources: widgets.example.com: enabled: true patches: - path: spec.selector.matchLabels labels: {} ``` If `spec.selector.matchLabels` held `{app: my-app}` in the tenant cluster, vCluster rewrites the key to its translated form on the control plane cluster so the selector continues to match the correct pods. ## Supported resources[​](#supported-resources "Direct link to Supported resources") ### To the control plane cluster[​](#to-the-control-plane-cluster "Direct link to To the control plane cluster") These resources sync tenant cluster state to the control plane cluster. Use `expression` for changes flowing outbound and `reverseExpression` for changes flowing back. | Resource | Config key | | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------- | | [Pods](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/pods.md) | `sync.toHost.pods.patches` | | [ConfigMaps](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/config-maps.md) | `sync.toHost.configMaps.patches` | | [Secrets](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/secrets.md) | `sync.toHost.secrets.patches` | | [Services](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/services.md) | `sync.toHost.services.patches` | | [Endpoints](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/endpoints.md) | `sync.toHost.endpoints.patches` | | [EndpointSlices](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/endpointslices.md) | `sync.toHost.endpointSlices.patches` | | [Ingresses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/ingresses.md) | `sync.toHost.ingresses.patches` | | [NetworkPolicies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/network-policies.md) | `sync.toHost.networkPolicies.patches` | | [Gateway API: HTTPRoutes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md) | `sync.toHost.gatewayApi.httpRoutes.patches` | | [Gateway API: TLSRoutes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md) | `sync.toHost.gatewayApi.tlsRoutes.patches` | | [Gateway API: BackendTLSPolicies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md) | `sync.toHost.gatewayApi.backendTLSPolicies.patches` | | [Gateway API: ReferenceGrants](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md) | `sync.toHost.gatewayApi.referenceGrants.patches` | | [PersistentVolumeClaims](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/persistent-volume-claims.md) | `sync.toHost.persistentVolumeClaims.patches` | | [PersistentVolumes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/persistent-volumes.md) | `sync.toHost.persistentVolumes.patches` | | [StorageClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/storage-classes.md) | `sync.toHost.storageClasses.patches` | | [ResourceClaims](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaims.md) | `sync.toHost.resourceClaims.patches` | | [ResourceClaimTemplates](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaimtemplates.md) | `sync.toHost.resourceClaimTemplates.patches` | | [PodDisruptionBudgets](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/pod-disruption-budgets.md) | `sync.toHost.podDisruptionBudgets.patches` | | [Custom resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/custom-resources.md) | `sync.toHost.customResources..patches` | ### From the control plane cluster[​](#from-the-control-plane-cluster "Direct link to From the control plane cluster") These resources sync control plane cluster state into the tenant cluster. For most resources, `expression` transforms values as they arrive from the control plane cluster, which matches the natural reading of the keyword for this direction. ConfigMaps, Secrets, and custom resources are the exception: their sync implementations share the same direction model as `sync.toHost`, where `reverseExpression` means the control plane→tenant direction. Use the third column to confirm which keyword applies to each resource rather than reasoning from the keyword names alone. | Resource | Config key | Use for control plane→tenant | | ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | ---------------------------- | | [ConfigMaps](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/configmaps.md) | `sync.fromHost.configMaps.patches` | `reverseExpression` | | [Secrets](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/secrets.md) | `sync.fromHost.secrets.patches` | `reverseExpression` | | [Custom resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/custom-resources.md) | `sync.fromHost.customResources..patches` | `reverseExpression` | | [CSI Drivers](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-drivers.md) | `sync.fromHost.csiDrivers.patches` | `expression` | | [CSI Nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-nodes.md) | `sync.fromHost.csiNodes.patches` | `expression` | | [CSI Storage Capacities](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-storage-capacities.md) | `sync.fromHost.csiStorageCapacities.patches` | `expression` | | [Device Classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/device-classes.md) | `sync.fromHost.deviceClasses.patches` | `expression` | | [Events](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/events.md) | `sync.fromHost.events.patches` | `expression` | | [Gateway Classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/gateway-classes.md) | `sync.fromHost.gatewayClasses.patches` | `expression` | | [Gateways](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/gateways.md) | `sync.fromHost.gateways.patches` | `expression` | | [Ingress Classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/ingress-classes.md) | `sync.fromHost.ingressClasses.patches` | `expression` | | [Nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md) | `sync.fromHost.nodes.patches` | `expression` | | [Priority Classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/priority-classes.md) | `sync.fromHost.priorityClasses.patches` | `expression` | | [Runtime Classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/runtime-classes.md) | `sync.fromHost.runtimeClasses.patches` | `expression` | | [Storage Classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/storage-classes.md) | `sync.fromHost.storageClasses.patches` | `expression` | | Volume Snapshot Classes | `sync.fromHost.volumeSnapshotClasses.patches` | `expression` | ## Troubleshoot[​](#troubleshoot "Direct link to Troubleshoot") | Symptom | What to check | | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Pro feature license error | Confirm the tenant cluster uses the Pro image and has a license from vCluster Platform. See [Resolve Pro feature license errors](https://vcluster.com/docs/vcluster/troubleshoot/pro-feature-license-error.md). | | Patch appears to do nothing | Confirm the path exists in the merge patch for the update. Patches run on changed fields only, not the full object. On create, vCluster converts the object to a patch, so more fields are available. | | `sync.fromHost` patch appears to do nothing | Confirm you are using the correct expression keyword for the resource. vCluster silently skips the wrong keyword with no error. See the [Supported resources](#from-the-control-plane-cluster) table. | | A field stops syncing in one direction | Check whether you have an `expression` or `reverseExpression` entry for that direction. Omitting the expression for a direction causes vCluster to drop that field from the patch. | | An object is stuck in a failing sync loop | An expression error is likely. Check vCluster pod logs for messages containing `apply patches host object` or `apply patches virtual object`. No event or status condition is set on the object. | | A field is being deleted instead of transformed | The expression may be returning `null` or `undefined`. vCluster deletes the field when an expression returns either value. | | Reference patch fails with a missing mapper error | The referenced resource type must be synced or otherwise known to vCluster. | --- # Sync to host from virtual Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Read more about [how syncing works](https://vcluster.com/docs/vcluster/configure/vcluster-yaml) before deciding which resources to sync from the tenant cluster to the control plane cluster. vCluster syncs specific Kubernetes resources from the tenant cluster to the control plane cluster. These resources are expected to be created and updated within the tenant cluster. In addition to standard Kubernetes resources, custom resources can also be synced if configured. ### Enabled by default[​](#enabled-by-default "Direct link to Enabled by default") The following resources are synced by default. * [Pods](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/pods.md) * [Secrets](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/secrets.md) * [ConfigMaps](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/config-maps.md) * [Services](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/services.md) * [Endpoints](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/endpoints.md) * [PersistentVolumeClaims](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/persistent-volume-claims.md) ### Disabled by default[​](#disabled-by-default "Direct link to Disabled by default") The following resources are **not** synced by default. You can explicitly enable them in your `vcluster.yaml` configuration. * [CustomResources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/custom-resources.md) * [Ingresses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/ingresses.md) * [Gateway API](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md) * [NetworkPolicies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/network-policies.md) * [PersistentVolumes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/persistent-volumes.md) * [StorageClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/storage-classes.md) * [ServiceAccounts](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/service-accounts.md) * [PodDisruptionBudgets](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/pod-disruption-budgets.md) * [PriorityClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/priority-classes.md) * [Namespaces](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/namespaces.md) ### Added by the tenant cluster[​](#added-by-the-tenant-cluster "Direct link to Added by the tenant cluster") When syncing objects from the tenant cluster to the control plane cluster, the tenant cluster adds management labels to the control plane cluster-side objects. In the default (single-namespace) mode, the following labels are added: * `release` * `vcluster.loft.sh/namespace` * `vcluster.loft.sh/managed-by` * `vcluster.loft.sh/controlled-by` note If `sync.toHost.namespaces.enabled` is `true` (multi-namespace sync), the tenant cluster does not add the release label. Only the `vcluster.loft.sh/*` labels are applied. ### Bidirectional syncing[​](#bidirectional-syncing "Direct link to Bidirectional syncing") For all resources synced from the tenant cluster to the control plane cluster, `metadata.labels` and `metadata.annotations` are synchronized in both directions. This means changes made in the tenant cluster are reflected in the control plane cluster, and changes made in the control plane cluster are also reflected in the tenant cluster. vCluster specific Labels Any labels prefixed with `vcluster.loft.sh` on the control plane cluster resources should not be updated by the user. These labels are managed by the vCluster control plane to track resources and prevent conflicts. #### Additional fields that support bidirectional sync[​](#additional-fields-that-support-bidirectional-sync "Direct link to Additional fields that support bidirectional sync") All `metadata.labels` and `metadata.annotations` are bidirectionally synced for resources synced from the tenant cluster to the control plane cluster.
In addition to these fields, vCluster also supports bidirectional syncing for the following resource-specific fields: | Resource | Fields | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Pods | - `status.conditions` | | Secrets | - `spec.data`
- `type` | | ConfigMaps | - `spec.data`
- `spec.binaryData` | | Services | - `spec.externalIPs`
- `spec.loadBalancerIP`
- `spec.ports`
- `spec.publishNotReadyAddresses`
- `spec.type`
- `spec.externalName`
- `spec.externalTrafficPolicy`
- `spec.sessionAffinity`
- `spec.sessionAffinityConfig`
- `spec.loadBalancerSourceRanges`
- `spec.healthCheckNodePort`
- `spec.selector` | | Ingresses | - `spec.ingressClassName` | | Gateway API | - `HTTPRoute`, `Gateway`, `TLSRoute`, `BackendTLSPolicy`, and `ReferenceGrant` metadata and status fields vary by resource and controller support. | | PersistentVolumes | - `spec.persistentVolumeSource`
- `spec.capacity`
- `spec.accessModes`
- `spec.persistentVolumeReclaimPolicy`
- `spec.nodeAffinity`
- `spec.volumeMode`
- `spec.mountOptions` | | StorageClass | - `spec.provisioner`
- `spec.parameters`
- `spec.reclaimPolicy`
- `spec.mountOptions`
- `spec.allowVolumeExpansion`
- `spec.volumeBindingMode`
- `spec.allowedTopologies` | ## Config reference[​](#config-reference "Direct link to Config reference") * Reference * Explorer (beta) ## `toHost` object [​](#toHost "Direct link to toHost")Configure resources to sync from the virtual cluster to the host cluster. ### `pods` object [​](#toHost-pods "Direct link to toHost-pods")Pods defines if pods created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean true [​](#toHost-pods-enabled "Direct link to toHost-pods-enabled")Enabled defines if pod syncing should be enabled. #### `translateImage` object {} [​](#toHost-pods-translateImage "Direct link to toHost-pods-translateImage")TranslateImage maps an image to another image that should be used instead. For example this can be used to rewrite a certain image that is used within the virtual cluster to be another image on the host cluster #### `enforceTolerations` string\[] \[] [​](#toHost-pods-enforceTolerations "Direct link to toHost-pods-enforceTolerations")EnforceTolerations will add the specified tolerations to all pods synced by the virtual cluster. #### `useSecretsForSATokens` boolean false [​](#toHost-pods-useSecretsForSATokens "Direct link to toHost-pods-useSecretsForSATokens")UseSecretsForSATokens will use secrets to save the generated service account tokens by virtual cluster instead of using a pod annotation. #### `runtimeClassName` string [​](#toHost-pods-runtimeClassName "Direct link to toHost-pods-runtimeClassName")RuntimeClassName is the runtime class to set for synced pods. #### `priorityClassName` string [​](#toHost-pods-priorityClassName "Direct link to toHost-pods-priorityClassName")PriorityClassName is the priority class to set for synced pods. #### `rewriteHosts` object [​](#toHost-pods-rewriteHosts "Direct link to toHost-pods-rewriteHosts")RewriteHosts is a special option needed to rewrite statefulset containers to allow the correct FQDN. virtual cluster will add a small container to each stateful set pod that will initially rewrite the /etc/hosts file to match the FQDN expected by the virtual cluster. ##### `enabled` boolean true [​](#toHost-pods-rewriteHosts-enabled "Direct link to toHost-pods-rewriteHosts-enabled")Enabled specifies if rewriting stateful set pods should be enabled. ##### `initContainer` object [​](#toHost-pods-rewriteHosts-initContainer "Direct link to toHost-pods-rewriteHosts-initContainer")InitContainer holds extra options for the init container used by vCluster to rewrite the FQDN for stateful set pods. ##### `image` object [​](#toHost-pods-rewriteHosts-initContainer-image "Direct link to toHost-pods-rewriteHosts-initContainer-image")Image is the image virtual cluster should use to rewrite this FQDN. ##### `registry` string mirror.gcr.io [​](#toHost-pods-rewriteHosts-initContainer-image-registry "Direct link to toHost-pods-rewriteHosts-initContainer-image-registry")Registry is the registry of the container image, e.g. my-registry.com or ghcr.io. This setting can be globally overridden via the controlPlane.advanced.defaultImageRegistry option. Empty means docker hub. ##### `repository` string library/alpine [​](#toHost-pods-rewriteHosts-initContainer-image-repository "Direct link to toHost-pods-rewriteHosts-initContainer-image-repository")Repository is the repository of the container image, e.g. my-repo/my-image ##### `tag` string 3.20 [​](#toHost-pods-rewriteHosts-initContainer-image-tag "Direct link to toHost-pods-rewriteHosts-initContainer-image-tag")Tag is the tag of the container image, and is the default version. ##### `resources` object [​](#toHost-pods-rewriteHosts-initContainer-resources "Direct link to toHost-pods-rewriteHosts-initContainer-resources")Resources are the resources that should be assigned to the init container for each stateful set init container. ##### `limits` object map\[cpu:30m memory:64Mi] [​](#toHost-pods-rewriteHosts-initContainer-resources-limits "Direct link to toHost-pods-rewriteHosts-initContainer-resources-limits")Limits are resource limits for the container ##### `requests` object map\[cpu:30m memory:64Mi] [​](#toHost-pods-rewriteHosts-initContainer-resources-requests "Direct link to toHost-pods-rewriteHosts-initContainer-resources-requests")Requests are minimal resources that will be consumed by the container #### `patches` object\[] [​](#toHost-pods-patches "Direct link to toHost-pods-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-pods-patches-path "Direct link to toHost-pods-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-pods-patches-expression "Direct link to toHost-pods-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-pods-patches-reverseExpression "Direct link to toHost-pods-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-pods-patches-reference "Direct link to toHost-pods-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-pods-patches-reference-apiVersion "Direct link to toHost-pods-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-pods-patches-reference-apiVersionPath "Direct link to toHost-pods-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-pods-patches-reference-kind "Direct link to toHost-pods-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-pods-patches-reference-kindPath "Direct link to toHost-pods-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-pods-patches-reference-namePath "Direct link to toHost-pods-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-pods-patches-reference-namespacePath "Direct link to toHost-pods-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-pods-patches-labels "Direct link to toHost-pods-patches-labels")Labels treats the path value as a labels selector. #### `hybridScheduling` object [​](#toHost-pods-hybridScheduling "Direct link to toHost-pods-hybridScheduling")HybridScheduling is used to enable and configure hybrid scheduling for pods in the virtual cluster. ##### `enabled` boolean false [​](#toHost-pods-hybridScheduling-enabled "Direct link to toHost-pods-hybridScheduling-enabled")Enabled specifies if hybrid scheduling is enabled. ##### `hostSchedulers` string\[] \[] [​](#toHost-pods-hybridScheduling-hostSchedulers "Direct link to toHost-pods-hybridScheduling-hostSchedulers")HostSchedulers is a list of schedulers that are deployed on the host cluster. ### `secrets` object [​](#toHost-secrets "Direct link to toHost-secrets")Secrets defines if secrets created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean true [​](#toHost-secrets-enabled "Direct link to toHost-secrets-enabled")Enabled defines if this option should be enabled. #### `all` boolean false [​](#toHost-secrets-all "Direct link to toHost-secrets-all")All defines if all resources of that type should get synced or only the necessary ones that are needed. #### `patches` object\[] [​](#toHost-secrets-patches "Direct link to toHost-secrets-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-secrets-patches-path "Direct link to toHost-secrets-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-secrets-patches-expression "Direct link to toHost-secrets-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-secrets-patches-reverseExpression "Direct link to toHost-secrets-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-secrets-patches-reference "Direct link to toHost-secrets-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-secrets-patches-reference-apiVersion "Direct link to toHost-secrets-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-secrets-patches-reference-apiVersionPath "Direct link to toHost-secrets-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-secrets-patches-reference-kind "Direct link to toHost-secrets-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-secrets-patches-reference-kindPath "Direct link to toHost-secrets-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-secrets-patches-reference-namePath "Direct link to toHost-secrets-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-secrets-patches-reference-namespacePath "Direct link to toHost-secrets-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-secrets-patches-labels "Direct link to toHost-secrets-patches-labels")Labels treats the path value as a labels selector. ### `configMaps` object [​](#toHost-configMaps "Direct link to toHost-configMaps")ConfigMaps defines if config maps created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean true [​](#toHost-configMaps-enabled "Direct link to toHost-configMaps-enabled")Enabled defines if this option should be enabled. #### `all` boolean false [​](#toHost-configMaps-all "Direct link to toHost-configMaps-all")All defines if all resources of that type should get synced or only the necessary ones that are needed. #### `patches` object\[] [​](#toHost-configMaps-patches "Direct link to toHost-configMaps-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-configMaps-patches-path "Direct link to toHost-configMaps-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-configMaps-patches-expression "Direct link to toHost-configMaps-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-configMaps-patches-reverseExpression "Direct link to toHost-configMaps-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-configMaps-patches-reference "Direct link to toHost-configMaps-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-configMaps-patches-reference-apiVersion "Direct link to toHost-configMaps-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-configMaps-patches-reference-apiVersionPath "Direct link to toHost-configMaps-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-configMaps-patches-reference-kind "Direct link to toHost-configMaps-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-configMaps-patches-reference-kindPath "Direct link to toHost-configMaps-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-configMaps-patches-reference-namePath "Direct link to toHost-configMaps-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-configMaps-patches-reference-namespacePath "Direct link to toHost-configMaps-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-configMaps-patches-labels "Direct link to toHost-configMaps-patches-labels")Labels treats the path value as a labels selector. ### `ingresses` object [​](#toHost-ingresses "Direct link to toHost-ingresses")Ingresses defines if ingresses created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean false [​](#toHost-ingresses-enabled "Direct link to toHost-ingresses-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-ingresses-patches "Direct link to toHost-ingresses-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-ingresses-patches-path "Direct link to toHost-ingresses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-ingresses-patches-expression "Direct link to toHost-ingresses-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-ingresses-patches-reverseExpression "Direct link to toHost-ingresses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-ingresses-patches-reference "Direct link to toHost-ingresses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-ingresses-patches-reference-apiVersion "Direct link to toHost-ingresses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-ingresses-patches-reference-apiVersionPath "Direct link to toHost-ingresses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-ingresses-patches-reference-kind "Direct link to toHost-ingresses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-ingresses-patches-reference-kindPath "Direct link to toHost-ingresses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-ingresses-patches-reference-namePath "Direct link to toHost-ingresses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-ingresses-patches-reference-namespacePath "Direct link to toHost-ingresses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-ingresses-patches-labels "Direct link to toHost-ingresses-patches-labels")Labels treats the path value as a labels selector. ### `gatewayApi` object [​](#toHost-gatewayApi "Direct link to toHost-gatewayApi")GatewayAPI defines Gateway API resources created within the tenant cluster that should get synced to the control plane cluster. Setting enabled: true turns on Gateway and HTTPRoute sync, imports control plane cluster GatewayClasses so tenant Gateways can resolve them, and serves tenant ReferenceGrants for validation; TLSRoutes and BackendTLSPolicies must be enabled individually. #### `enabled` boolean false [​](#toHost-gatewayApi-enabled "Direct link to toHost-gatewayApi-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-gatewayApi-patches "Direct link to toHost-gatewayApi-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-gatewayApi-patches-path "Direct link to toHost-gatewayApi-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-gatewayApi-patches-expression "Direct link to toHost-gatewayApi-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-gatewayApi-patches-reverseExpression "Direct link to toHost-gatewayApi-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-gatewayApi-patches-reference "Direct link to toHost-gatewayApi-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-gatewayApi-patches-reference-apiVersion "Direct link to toHost-gatewayApi-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-gatewayApi-patches-reference-apiVersionPath "Direct link to toHost-gatewayApi-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-gatewayApi-patches-reference-kind "Direct link to toHost-gatewayApi-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-gatewayApi-patches-reference-kindPath "Direct link to toHost-gatewayApi-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-gatewayApi-patches-reference-namePath "Direct link to toHost-gatewayApi-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-gatewayApi-patches-reference-namespacePath "Direct link to toHost-gatewayApi-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-gatewayApi-patches-labels "Direct link to toHost-gatewayApi-patches-labels")Labels treats the path value as a labels selector. #### `httpRoutes` object [​](#toHost-gatewayApi-httpRoutes "Direct link to toHost-gatewayApi-httpRoutes")HTTPRoutes configures HTTPRoute sync to the control plane cluster. ##### `enabled` boolean false [​](#toHost-gatewayApi-httpRoutes-enabled "Direct link to toHost-gatewayApi-httpRoutes-enabled")Enabled defines if this option should be enabled. ##### `patches` object\[] [​](#toHost-gatewayApi-httpRoutes-patches "Direct link to toHost-gatewayApi-httpRoutes-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-gatewayApi-httpRoutes-patches-path "Direct link to toHost-gatewayApi-httpRoutes-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-gatewayApi-httpRoutes-patches-expression "Direct link to toHost-gatewayApi-httpRoutes-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-gatewayApi-httpRoutes-patches-reverseExpression "Direct link to toHost-gatewayApi-httpRoutes-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-gatewayApi-httpRoutes-patches-reference "Direct link to toHost-gatewayApi-httpRoutes-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-gatewayApi-httpRoutes-patches-reference-apiVersion "Direct link to toHost-gatewayApi-httpRoutes-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-gatewayApi-httpRoutes-patches-reference-apiVersionPath "Direct link to toHost-gatewayApi-httpRoutes-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-gatewayApi-httpRoutes-patches-reference-kind "Direct link to toHost-gatewayApi-httpRoutes-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-gatewayApi-httpRoutes-patches-reference-kindPath "Direct link to toHost-gatewayApi-httpRoutes-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-gatewayApi-httpRoutes-patches-reference-namePath "Direct link to toHost-gatewayApi-httpRoutes-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-gatewayApi-httpRoutes-patches-reference-namespacePath "Direct link to toHost-gatewayApi-httpRoutes-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-gatewayApi-httpRoutes-patches-labels "Direct link to toHost-gatewayApi-httpRoutes-patches-labels")Labels treats the path value as a labels selector. #### `gateways` object [​](#toHost-gatewayApi-gateways "Direct link to toHost-gatewayApi-gateways")Gateways configures tenant-created Gateway sync to the control plane cluster. ##### `enabled` boolean false [​](#toHost-gatewayApi-gateways-enabled "Direct link to toHost-gatewayApi-gateways-enabled")Enabled defines if this option should be enabled. ##### `patches` object\[] [​](#toHost-gatewayApi-gateways-patches "Direct link to toHost-gatewayApi-gateways-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-gatewayApi-gateways-patches-path "Direct link to toHost-gatewayApi-gateways-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-gatewayApi-gateways-patches-expression "Direct link to toHost-gatewayApi-gateways-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-gatewayApi-gateways-patches-reverseExpression "Direct link to toHost-gatewayApi-gateways-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-gatewayApi-gateways-patches-reference "Direct link to toHost-gatewayApi-gateways-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-gatewayApi-gateways-patches-reference-apiVersion "Direct link to toHost-gatewayApi-gateways-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-gatewayApi-gateways-patches-reference-apiVersionPath "Direct link to toHost-gatewayApi-gateways-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-gatewayApi-gateways-patches-reference-kind "Direct link to toHost-gatewayApi-gateways-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-gatewayApi-gateways-patches-reference-kindPath "Direct link to toHost-gatewayApi-gateways-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-gatewayApi-gateways-patches-reference-namePath "Direct link to toHost-gatewayApi-gateways-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-gatewayApi-gateways-patches-reference-namespacePath "Direct link to toHost-gatewayApi-gateways-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-gatewayApi-gateways-patches-labels "Direct link to toHost-gatewayApi-gateways-patches-labels")Labels treats the path value as a labels selector. #### `tlsRoutes` object [​](#toHost-gatewayApi-tlsRoutes "Direct link to toHost-gatewayApi-tlsRoutes")TLSRoutes configures TLSRoute sync to the control plane cluster. ##### `enabled` boolean false [​](#toHost-gatewayApi-tlsRoutes-enabled "Direct link to toHost-gatewayApi-tlsRoutes-enabled")Enabled defines if this option should be enabled. ##### `patches` object\[] [​](#toHost-gatewayApi-tlsRoutes-patches "Direct link to toHost-gatewayApi-tlsRoutes-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-gatewayApi-tlsRoutes-patches-path "Direct link to toHost-gatewayApi-tlsRoutes-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-gatewayApi-tlsRoutes-patches-expression "Direct link to toHost-gatewayApi-tlsRoutes-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-gatewayApi-tlsRoutes-patches-reverseExpression "Direct link to toHost-gatewayApi-tlsRoutes-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-gatewayApi-tlsRoutes-patches-reference "Direct link to toHost-gatewayApi-tlsRoutes-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-gatewayApi-tlsRoutes-patches-reference-apiVersion "Direct link to toHost-gatewayApi-tlsRoutes-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-gatewayApi-tlsRoutes-patches-reference-apiVersionPath "Direct link to toHost-gatewayApi-tlsRoutes-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-gatewayApi-tlsRoutes-patches-reference-kind "Direct link to toHost-gatewayApi-tlsRoutes-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-gatewayApi-tlsRoutes-patches-reference-kindPath "Direct link to toHost-gatewayApi-tlsRoutes-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-gatewayApi-tlsRoutes-patches-reference-namePath "Direct link to toHost-gatewayApi-tlsRoutes-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-gatewayApi-tlsRoutes-patches-reference-namespacePath "Direct link to toHost-gatewayApi-tlsRoutes-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-gatewayApi-tlsRoutes-patches-labels "Direct link to toHost-gatewayApi-tlsRoutes-patches-labels")Labels treats the path value as a labels selector. #### `backendTLSPolicies` object [​](#toHost-gatewayApi-backendTLSPolicies "Direct link to toHost-gatewayApi-backendTLSPolicies")BackendTLSPolicies configures BackendTLSPolicy sync to the control plane cluster. ##### `enabled` boolean false [​](#toHost-gatewayApi-backendTLSPolicies-enabled "Direct link to toHost-gatewayApi-backendTLSPolicies-enabled")Enabled defines if this option should be enabled. ##### `patches` object\[] [​](#toHost-gatewayApi-backendTLSPolicies-patches "Direct link to toHost-gatewayApi-backendTLSPolicies-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-gatewayApi-backendTLSPolicies-patches-path "Direct link to toHost-gatewayApi-backendTLSPolicies-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-gatewayApi-backendTLSPolicies-patches-expression "Direct link to toHost-gatewayApi-backendTLSPolicies-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-gatewayApi-backendTLSPolicies-patches-reverseExpression "Direct link to toHost-gatewayApi-backendTLSPolicies-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-gatewayApi-backendTLSPolicies-patches-reference "Direct link to toHost-gatewayApi-backendTLSPolicies-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-gatewayApi-backendTLSPolicies-patches-reference-apiVersion "Direct link to toHost-gatewayApi-backendTLSPolicies-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-gatewayApi-backendTLSPolicies-patches-reference-apiVersionPath "Direct link to toHost-gatewayApi-backendTLSPolicies-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-gatewayApi-backendTLSPolicies-patches-reference-kind "Direct link to toHost-gatewayApi-backendTLSPolicies-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-gatewayApi-backendTLSPolicies-patches-reference-kindPath "Direct link to toHost-gatewayApi-backendTLSPolicies-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-gatewayApi-backendTLSPolicies-patches-reference-namePath "Direct link to toHost-gatewayApi-backendTLSPolicies-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-gatewayApi-backendTLSPolicies-patches-reference-namespacePath "Direct link to toHost-gatewayApi-backendTLSPolicies-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-gatewayApi-backendTLSPolicies-patches-labels "Direct link to toHost-gatewayApi-backendTLSPolicies-patches-labels")Labels treats the path value as a labels selector. #### `referenceGrants` object [​](#toHost-gatewayApi-referenceGrants "Direct link to toHost-gatewayApi-referenceGrants")ReferenceGrants configures ReferenceGrant sync to the control plane cluster. Enabled may be "auto", "true", or "false". In auto mode grants follow route sync and are validated within the tenant cluster; they sync to the control plane cluster only when namespace sync is also enabled. ##### `enabled` string|boolean auto [​](#toHost-gatewayApi-referenceGrants-enabled "Direct link to toHost-gatewayApi-referenceGrants-enabled")Enabled defines if this option should be enabled. ##### `patches` object\[] [​](#toHost-gatewayApi-referenceGrants-patches "Direct link to toHost-gatewayApi-referenceGrants-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-gatewayApi-referenceGrants-patches-path "Direct link to toHost-gatewayApi-referenceGrants-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-gatewayApi-referenceGrants-patches-expression "Direct link to toHost-gatewayApi-referenceGrants-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-gatewayApi-referenceGrants-patches-reverseExpression "Direct link to toHost-gatewayApi-referenceGrants-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-gatewayApi-referenceGrants-patches-reference "Direct link to toHost-gatewayApi-referenceGrants-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-gatewayApi-referenceGrants-patches-reference-apiVersion "Direct link to toHost-gatewayApi-referenceGrants-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-gatewayApi-referenceGrants-patches-reference-apiVersionPath "Direct link to toHost-gatewayApi-referenceGrants-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-gatewayApi-referenceGrants-patches-reference-kind "Direct link to toHost-gatewayApi-referenceGrants-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-gatewayApi-referenceGrants-patches-reference-kindPath "Direct link to toHost-gatewayApi-referenceGrants-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-gatewayApi-referenceGrants-patches-reference-namePath "Direct link to toHost-gatewayApi-referenceGrants-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-gatewayApi-referenceGrants-patches-reference-namespacePath "Direct link to toHost-gatewayApi-referenceGrants-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-gatewayApi-referenceGrants-patches-labels "Direct link to toHost-gatewayApi-referenceGrants-patches-labels")Labels treats the path value as a labels selector. ### `services` object [​](#toHost-services "Direct link to toHost-services")Services defines if services created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean true [​](#toHost-services-enabled "Direct link to toHost-services-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-services-patches "Direct link to toHost-services-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-services-patches-path "Direct link to toHost-services-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-services-patches-expression "Direct link to toHost-services-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-services-patches-reverseExpression "Direct link to toHost-services-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-services-patches-reference "Direct link to toHost-services-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-services-patches-reference-apiVersion "Direct link to toHost-services-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-services-patches-reference-apiVersionPath "Direct link to toHost-services-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-services-patches-reference-kind "Direct link to toHost-services-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-services-patches-reference-kindPath "Direct link to toHost-services-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-services-patches-reference-namePath "Direct link to toHost-services-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-services-patches-reference-namespacePath "Direct link to toHost-services-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-services-patches-labels "Direct link to toHost-services-patches-labels")Labels treats the path value as a labels selector. ### `endpoints` object [​](#toHost-endpoints "Direct link to toHost-endpoints")Endpoints defines if endpoints created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean true [​](#toHost-endpoints-enabled "Direct link to toHost-endpoints-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-endpoints-patches "Direct link to toHost-endpoints-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-endpoints-patches-path "Direct link to toHost-endpoints-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-endpoints-patches-expression "Direct link to toHost-endpoints-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-endpoints-patches-reverseExpression "Direct link to toHost-endpoints-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-endpoints-patches-reference "Direct link to toHost-endpoints-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-endpoints-patches-reference-apiVersion "Direct link to toHost-endpoints-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-endpoints-patches-reference-apiVersionPath "Direct link to toHost-endpoints-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-endpoints-patches-reference-kind "Direct link to toHost-endpoints-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-endpoints-patches-reference-kindPath "Direct link to toHost-endpoints-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-endpoints-patches-reference-namePath "Direct link to toHost-endpoints-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-endpoints-patches-reference-namespacePath "Direct link to toHost-endpoints-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-endpoints-patches-labels "Direct link to toHost-endpoints-patches-labels")Labels treats the path value as a labels selector. ### `endpointSlices` object [​](#toHost-endpointSlices "Direct link to toHost-endpointSlices")EndpointSlices defines if endpointslices created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean true [​](#toHost-endpointSlices-enabled "Direct link to toHost-endpointSlices-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-endpointSlices-patches "Direct link to toHost-endpointSlices-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-endpointSlices-patches-path "Direct link to toHost-endpointSlices-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-endpointSlices-patches-expression "Direct link to toHost-endpointSlices-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-endpointSlices-patches-reverseExpression "Direct link to toHost-endpointSlices-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-endpointSlices-patches-reference "Direct link to toHost-endpointSlices-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-endpointSlices-patches-reference-apiVersion "Direct link to toHost-endpointSlices-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-endpointSlices-patches-reference-apiVersionPath "Direct link to toHost-endpointSlices-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-endpointSlices-patches-reference-kind "Direct link to toHost-endpointSlices-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-endpointSlices-patches-reference-kindPath "Direct link to toHost-endpointSlices-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-endpointSlices-patches-reference-namePath "Direct link to toHost-endpointSlices-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-endpointSlices-patches-reference-namespacePath "Direct link to toHost-endpointSlices-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-endpointSlices-patches-labels "Direct link to toHost-endpointSlices-patches-labels")Labels treats the path value as a labels selector. ### `networkPolicies` object [​](#toHost-networkPolicies "Direct link to toHost-networkPolicies")NetworkPolicies defines if network policies created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean false [​](#toHost-networkPolicies-enabled "Direct link to toHost-networkPolicies-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-networkPolicies-patches "Direct link to toHost-networkPolicies-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-networkPolicies-patches-path "Direct link to toHost-networkPolicies-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-networkPolicies-patches-expression "Direct link to toHost-networkPolicies-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-networkPolicies-patches-reverseExpression "Direct link to toHost-networkPolicies-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-networkPolicies-patches-reference "Direct link to toHost-networkPolicies-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-networkPolicies-patches-reference-apiVersion "Direct link to toHost-networkPolicies-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-networkPolicies-patches-reference-apiVersionPath "Direct link to toHost-networkPolicies-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-networkPolicies-patches-reference-kind "Direct link to toHost-networkPolicies-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-networkPolicies-patches-reference-kindPath "Direct link to toHost-networkPolicies-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-networkPolicies-patches-reference-namePath "Direct link to toHost-networkPolicies-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-networkPolicies-patches-reference-namespacePath "Direct link to toHost-networkPolicies-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-networkPolicies-patches-labels "Direct link to toHost-networkPolicies-patches-labels")Labels treats the path value as a labels selector. ### `persistentVolumeClaims` object [​](#toHost-persistentVolumeClaims "Direct link to toHost-persistentVolumeClaims")PersistentVolumeClaims defines if persistent volume claims created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean true [​](#toHost-persistentVolumeClaims-enabled "Direct link to toHost-persistentVolumeClaims-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-persistentVolumeClaims-patches "Direct link to toHost-persistentVolumeClaims-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-persistentVolumeClaims-patches-path "Direct link to toHost-persistentVolumeClaims-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-persistentVolumeClaims-patches-expression "Direct link to toHost-persistentVolumeClaims-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-persistentVolumeClaims-patches-reverseExpression "Direct link to toHost-persistentVolumeClaims-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-persistentVolumeClaims-patches-reference "Direct link to toHost-persistentVolumeClaims-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-persistentVolumeClaims-patches-reference-apiVersion "Direct link to toHost-persistentVolumeClaims-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-persistentVolumeClaims-patches-reference-apiVersionPath "Direct link to toHost-persistentVolumeClaims-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-persistentVolumeClaims-patches-reference-kind "Direct link to toHost-persistentVolumeClaims-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-persistentVolumeClaims-patches-reference-kindPath "Direct link to toHost-persistentVolumeClaims-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-persistentVolumeClaims-patches-reference-namePath "Direct link to toHost-persistentVolumeClaims-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-persistentVolumeClaims-patches-reference-namespacePath "Direct link to toHost-persistentVolumeClaims-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-persistentVolumeClaims-patches-labels "Direct link to toHost-persistentVolumeClaims-patches-labels")Labels treats the path value as a labels selector. ### `persistentVolumes` object [​](#toHost-persistentVolumes "Direct link to toHost-persistentVolumes")PersistentVolumes defines if persistent volumes created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean false [​](#toHost-persistentVolumes-enabled "Direct link to toHost-persistentVolumes-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-persistentVolumes-patches "Direct link to toHost-persistentVolumes-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-persistentVolumes-patches-path "Direct link to toHost-persistentVolumes-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-persistentVolumes-patches-expression "Direct link to toHost-persistentVolumes-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-persistentVolumes-patches-reverseExpression "Direct link to toHost-persistentVolumes-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-persistentVolumes-patches-reference "Direct link to toHost-persistentVolumes-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-persistentVolumes-patches-reference-apiVersion "Direct link to toHost-persistentVolumes-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-persistentVolumes-patches-reference-apiVersionPath "Direct link to toHost-persistentVolumes-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-persistentVolumes-patches-reference-kind "Direct link to toHost-persistentVolumes-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-persistentVolumes-patches-reference-kindPath "Direct link to toHost-persistentVolumes-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-persistentVolumes-patches-reference-namePath "Direct link to toHost-persistentVolumes-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-persistentVolumes-patches-reference-namespacePath "Direct link to toHost-persistentVolumes-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-persistentVolumes-patches-labels "Direct link to toHost-persistentVolumes-patches-labels")Labels treats the path value as a labels selector. ### `volumeSnapshots` object [​](#toHost-volumeSnapshots "Direct link to toHost-volumeSnapshots")VolumeSnapshots defines if volume snapshots created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean false [​](#toHost-volumeSnapshots-enabled "Direct link to toHost-volumeSnapshots-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-volumeSnapshots-patches "Direct link to toHost-volumeSnapshots-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-volumeSnapshots-patches-path "Direct link to toHost-volumeSnapshots-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-volumeSnapshots-patches-expression "Direct link to toHost-volumeSnapshots-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-volumeSnapshots-patches-reverseExpression "Direct link to toHost-volumeSnapshots-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-volumeSnapshots-patches-reference "Direct link to toHost-volumeSnapshots-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-volumeSnapshots-patches-reference-apiVersion "Direct link to toHost-volumeSnapshots-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-volumeSnapshots-patches-reference-apiVersionPath "Direct link to toHost-volumeSnapshots-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-volumeSnapshots-patches-reference-kind "Direct link to toHost-volumeSnapshots-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-volumeSnapshots-patches-reference-kindPath "Direct link to toHost-volumeSnapshots-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-volumeSnapshots-patches-reference-namePath "Direct link to toHost-volumeSnapshots-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-volumeSnapshots-patches-reference-namespacePath "Direct link to toHost-volumeSnapshots-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-volumeSnapshots-patches-labels "Direct link to toHost-volumeSnapshots-patches-labels")Labels treats the path value as a labels selector. ### `volumeSnapshotContents` object [​](#toHost-volumeSnapshotContents "Direct link to toHost-volumeSnapshotContents")VolumeSnapshotContents defines if volume snapshot contents created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean false [​](#toHost-volumeSnapshotContents-enabled "Direct link to toHost-volumeSnapshotContents-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-volumeSnapshotContents-patches "Direct link to toHost-volumeSnapshotContents-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-volumeSnapshotContents-patches-path "Direct link to toHost-volumeSnapshotContents-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-volumeSnapshotContents-patches-expression "Direct link to toHost-volumeSnapshotContents-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-volumeSnapshotContents-patches-reverseExpression "Direct link to toHost-volumeSnapshotContents-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-volumeSnapshotContents-patches-reference "Direct link to toHost-volumeSnapshotContents-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-volumeSnapshotContents-patches-reference-apiVersion "Direct link to toHost-volumeSnapshotContents-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-volumeSnapshotContents-patches-reference-apiVersionPath "Direct link to toHost-volumeSnapshotContents-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-volumeSnapshotContents-patches-reference-kind "Direct link to toHost-volumeSnapshotContents-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-volumeSnapshotContents-patches-reference-kindPath "Direct link to toHost-volumeSnapshotContents-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-volumeSnapshotContents-patches-reference-namePath "Direct link to toHost-volumeSnapshotContents-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-volumeSnapshotContents-patches-reference-namespacePath "Direct link to toHost-volumeSnapshotContents-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-volumeSnapshotContents-patches-labels "Direct link to toHost-volumeSnapshotContents-patches-labels")Labels treats the path value as a labels selector. ### `storageClasses` object [​](#toHost-storageClasses "Direct link to toHost-storageClasses")StorageClasses defines if storage classes created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean false [​](#toHost-storageClasses-enabled "Direct link to toHost-storageClasses-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-storageClasses-patches "Direct link to toHost-storageClasses-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-storageClasses-patches-path "Direct link to toHost-storageClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-storageClasses-patches-expression "Direct link to toHost-storageClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-storageClasses-patches-reverseExpression "Direct link to toHost-storageClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-storageClasses-patches-reference "Direct link to toHost-storageClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-storageClasses-patches-reference-apiVersion "Direct link to toHost-storageClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-storageClasses-patches-reference-apiVersionPath "Direct link to toHost-storageClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-storageClasses-patches-reference-kind "Direct link to toHost-storageClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-storageClasses-patches-reference-kindPath "Direct link to toHost-storageClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-storageClasses-patches-reference-namePath "Direct link to toHost-storageClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-storageClasses-patches-reference-namespacePath "Direct link to toHost-storageClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-storageClasses-patches-labels "Direct link to toHost-storageClasses-patches-labels")Labels treats the path value as a labels selector. ### `serviceAccounts` object [​](#toHost-serviceAccounts "Direct link to toHost-serviceAccounts")ServiceAccounts defines if service accounts created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean false [​](#toHost-serviceAccounts-enabled "Direct link to toHost-serviceAccounts-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-serviceAccounts-patches "Direct link to toHost-serviceAccounts-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-serviceAccounts-patches-path "Direct link to toHost-serviceAccounts-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-serviceAccounts-patches-expression "Direct link to toHost-serviceAccounts-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-serviceAccounts-patches-reverseExpression "Direct link to toHost-serviceAccounts-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-serviceAccounts-patches-reference "Direct link to toHost-serviceAccounts-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-serviceAccounts-patches-reference-apiVersion "Direct link to toHost-serviceAccounts-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-serviceAccounts-patches-reference-apiVersionPath "Direct link to toHost-serviceAccounts-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-serviceAccounts-patches-reference-kind "Direct link to toHost-serviceAccounts-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-serviceAccounts-patches-reference-kindPath "Direct link to toHost-serviceAccounts-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-serviceAccounts-patches-reference-namePath "Direct link to toHost-serviceAccounts-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-serviceAccounts-patches-reference-namespacePath "Direct link to toHost-serviceAccounts-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-serviceAccounts-patches-labels "Direct link to toHost-serviceAccounts-patches-labels")Labels treats the path value as a labels selector. ### `podDisruptionBudgets` object [​](#toHost-podDisruptionBudgets "Direct link to toHost-podDisruptionBudgets")PodDisruptionBudgets defines if pod disruption budgets created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean false [​](#toHost-podDisruptionBudgets-enabled "Direct link to toHost-podDisruptionBudgets-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-podDisruptionBudgets-patches "Direct link to toHost-podDisruptionBudgets-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-podDisruptionBudgets-patches-path "Direct link to toHost-podDisruptionBudgets-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-podDisruptionBudgets-patches-expression "Direct link to toHost-podDisruptionBudgets-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-podDisruptionBudgets-patches-reverseExpression "Direct link to toHost-podDisruptionBudgets-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-podDisruptionBudgets-patches-reference "Direct link to toHost-podDisruptionBudgets-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-podDisruptionBudgets-patches-reference-apiVersion "Direct link to toHost-podDisruptionBudgets-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-podDisruptionBudgets-patches-reference-apiVersionPath "Direct link to toHost-podDisruptionBudgets-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-podDisruptionBudgets-patches-reference-kind "Direct link to toHost-podDisruptionBudgets-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-podDisruptionBudgets-patches-reference-kindPath "Direct link to toHost-podDisruptionBudgets-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-podDisruptionBudgets-patches-reference-namePath "Direct link to toHost-podDisruptionBudgets-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-podDisruptionBudgets-patches-reference-namespacePath "Direct link to toHost-podDisruptionBudgets-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-podDisruptionBudgets-patches-labels "Direct link to toHost-podDisruptionBudgets-patches-labels")Labels treats the path value as a labels selector. ### `priorityClasses` object [​](#toHost-priorityClasses "Direct link to toHost-priorityClasses")PriorityClasses defines if priority classes created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean false [​](#toHost-priorityClasses-enabled "Direct link to toHost-priorityClasses-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-priorityClasses-patches "Direct link to toHost-priorityClasses-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-priorityClasses-patches-path "Direct link to toHost-priorityClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-priorityClasses-patches-expression "Direct link to toHost-priorityClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-priorityClasses-patches-reverseExpression "Direct link to toHost-priorityClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-priorityClasses-patches-reference "Direct link to toHost-priorityClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-priorityClasses-patches-reference-apiVersion "Direct link to toHost-priorityClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-priorityClasses-patches-reference-apiVersionPath "Direct link to toHost-priorityClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-priorityClasses-patches-reference-kind "Direct link to toHost-priorityClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-priorityClasses-patches-reference-kindPath "Direct link to toHost-priorityClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-priorityClasses-patches-reference-namePath "Direct link to toHost-priorityClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-priorityClasses-patches-reference-namespacePath "Direct link to toHost-priorityClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-priorityClasses-patches-labels "Direct link to toHost-priorityClasses-patches-labels")Labels treats the path value as a labels selector. ### `customResources` {key: object} [​](#toHost-customResources "Direct link to toHost-customResources")CustomResources defines what custom resources should get synced from the virtual cluster to the host cluster. vCluster will copy the definition automatically from host cluster to virtual cluster on startup. vCluster will also automatically add any required RBAC permissions to the vCluster role for this to work. #### `enabled` required boolean [​](#toHost-customResources-enabled "Direct link to toHost-customResources-enabled")Enabled defines if this option should be enabled. #### `scope` string [​](#toHost-customResources-scope "Direct link to toHost-customResources-scope")Scope defines the scope of the resource. If undefined, will use Namespaced. Currently only Namespaced is supported. #### `patches` object\[] [​](#toHost-customResources-patches "Direct link to toHost-customResources-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-customResources-patches-path "Direct link to toHost-customResources-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-customResources-patches-expression "Direct link to toHost-customResources-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-customResources-patches-reverseExpression "Direct link to toHost-customResources-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-customResources-patches-reference "Direct link to toHost-customResources-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-customResources-patches-reference-apiVersion "Direct link to toHost-customResources-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-customResources-patches-reference-apiVersionPath "Direct link to toHost-customResources-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-customResources-patches-reference-kind "Direct link to toHost-customResources-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-customResources-patches-reference-kindPath "Direct link to toHost-customResources-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-customResources-patches-reference-namePath "Direct link to toHost-customResources-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-customResources-patches-reference-namespacePath "Direct link to toHost-customResources-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-customResources-patches-labels "Direct link to toHost-customResources-patches-labels")Labels treats the path value as a labels selector. ### `namespaces` object [​](#toHost-namespaces "Direct link to toHost-namespaces")Namespaces defines if namespaces created within the virtual cluster should get synced to the host cluster. #### `enabled` required boolean false [​](#toHost-namespaces-enabled "Direct link to toHost-namespaces-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-namespaces-patches "Direct link to toHost-namespaces-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-namespaces-patches-path "Direct link to toHost-namespaces-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-namespaces-patches-expression "Direct link to toHost-namespaces-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-namespaces-patches-reverseExpression "Direct link to toHost-namespaces-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-namespaces-patches-reference "Direct link to toHost-namespaces-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-namespaces-patches-reference-apiVersion "Direct link to toHost-namespaces-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-namespaces-patches-reference-apiVersionPath "Direct link to toHost-namespaces-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-namespaces-patches-reference-kind "Direct link to toHost-namespaces-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-namespaces-patches-reference-kindPath "Direct link to toHost-namespaces-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-namespaces-patches-reference-namePath "Direct link to toHost-namespaces-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-namespaces-patches-reference-namespacePath "Direct link to toHost-namespaces-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-namespaces-patches-labels "Direct link to toHost-namespaces-patches-labels")Labels treats the path value as a labels selector. #### `mappings` object [​](#toHost-namespaces-mappings "Direct link to toHost-namespaces-mappings")Mappings for Namespace and Object ##### `byName` object [​](#toHost-namespaces-mappings-byName "Direct link to toHost-namespaces-mappings-byName")ByName is a map of control-plane-object-namespace/control-plane-object-name: tenant-object-namespace/tenant-object-name. There are several wildcards supported:1) To match all objects in a control plane namespace and sync them to a different namespace in the tenant cluster: byName: "foo/*": "foo-in-virtual/*" 2) To match a specific object in the control plane namespace and sync it to the same namespace with the same name: byName: "foo/my-object": "foo/my-object" 3) To match a specific object in the control plane namespace and sync it to the same namespace with a different name: byName: "foo/my-object": "foo/my-virtual-object" 4) To match all objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "": "my-virtual-namespace/\*" 5) To match specific objects in the vCluster namespace and sync them to a different namespace in the tenant cluster: byName: "/my-object": "my-virtual-namespace/my-object" #### `mappingsOnly` boolean false [​](#toHost-namespaces-mappingsOnly "Direct link to toHost-namespaces-mappingsOnly")MappingsOnly defines if creation of namespaces not matched by mappings should be allowed. #### `extraLabels` object [​](#toHost-namespaces-extraLabels "Direct link to toHost-namespaces-extraLabels")ExtraLabels are additional labels to add to the namespace in the host cluster. ### `resourceClaims` object [​](#toHost-resourceClaims "Direct link to toHost-resourceClaims")ResourceClaim defines if resource claims created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean false [​](#toHost-resourceClaims-enabled "Direct link to toHost-resourceClaims-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-resourceClaims-patches "Direct link to toHost-resourceClaims-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-resourceClaims-patches-path "Direct link to toHost-resourceClaims-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-resourceClaims-patches-expression "Direct link to toHost-resourceClaims-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-resourceClaims-patches-reverseExpression "Direct link to toHost-resourceClaims-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-resourceClaims-patches-reference "Direct link to toHost-resourceClaims-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-resourceClaims-patches-reference-apiVersion "Direct link to toHost-resourceClaims-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-resourceClaims-patches-reference-apiVersionPath "Direct link to toHost-resourceClaims-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-resourceClaims-patches-reference-kind "Direct link to toHost-resourceClaims-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-resourceClaims-patches-reference-kindPath "Direct link to toHost-resourceClaims-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-resourceClaims-patches-reference-namePath "Direct link to toHost-resourceClaims-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-resourceClaims-patches-reference-namespacePath "Direct link to toHost-resourceClaims-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-resourceClaims-patches-labels "Direct link to toHost-resourceClaims-patches-labels")Labels treats the path value as a labels selector. ### `resourceClaimTemplates` object [​](#toHost-resourceClaimTemplates "Direct link to toHost-resourceClaimTemplates")ResourceClaimTemplates defines if resourceClaimTemplates created within the virtual cluster should get synced to the host cluster. #### `enabled` boolean false [​](#toHost-resourceClaimTemplates-enabled "Direct link to toHost-resourceClaimTemplates-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#toHost-resourceClaimTemplates-patches "Direct link to toHost-resourceClaimTemplates-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#toHost-resourceClaimTemplates-patches-path "Direct link to toHost-resourceClaimTemplates-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#toHost-resourceClaimTemplates-patches-expression "Direct link to toHost-resourceClaimTemplates-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#toHost-resourceClaimTemplates-patches-reverseExpression "Direct link to toHost-resourceClaimTemplates-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#toHost-resourceClaimTemplates-patches-reference "Direct link to toHost-resourceClaimTemplates-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#toHost-resourceClaimTemplates-patches-reference-apiVersion "Direct link to toHost-resourceClaimTemplates-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#toHost-resourceClaimTemplates-patches-reference-apiVersionPath "Direct link to toHost-resourceClaimTemplates-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#toHost-resourceClaimTemplates-patches-reference-kind "Direct link to toHost-resourceClaimTemplates-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#toHost-resourceClaimTemplates-patches-reference-kindPath "Direct link to toHost-resourceClaimTemplates-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#toHost-resourceClaimTemplates-patches-reference-namePath "Direct link to toHost-resourceClaimTemplates-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#toHost-resourceClaimTemplates-patches-reference-namespacePath "Direct link to toHost-resourceClaimTemplates-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#toHost-resourceClaimTemplates-patches-labels "Direct link to toHost-resourceClaimTemplates-patches-labels")Labels treats the path value as a labels selector. Browse the same settings as the YAML you write. Click a field for its type, default, and description; fold sections and filter to navigate quickly. --- # Custom resources Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** vCluster allows you to sync custom resources from the tenant cluster to the control plane cluster. This is useful for syncing resources that are not included in the default sync behavior. This feature works for namespace-scoped resources backed by a CustomResourceDefinition (CRD) or served through [API aggregation](#aggregated-api-resources). If a synced custom resource creates additional resources in the control plane cluster, vCluster attempts to detect and sync those resources back to the tenant cluster. For example, a custom resource that creates a Secret has that Secret automatically synced into the tenant cluster. For help choosing a sync direction, and for lifecycle and troubleshooting guidance, see [Manage custom resources](https://vcluster.com/docs/vcluster/manage/custom-resources.md).
Automatic RBAC for configured resources vCluster automatically adds the required cluster and namespace-level RBAC permissions for retrieving the CustomResourceDefinition and syncing the configured resources. Extension API servers can require additional permissions for related resources. See [Authorization on the control plane cluster](#authorization-on-the-control-plane-cluster). Only namespace-scoped resources This feature currently only works for namespace-scoped resources. Namespace syncing If you want to sync many custom resources, consider using [namespace syncing](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/namespaces.md). ## Enable custom resource syncing[​](#enable-custom-resource-syncing "Direct link to Enable custom resource syncing") To enable custom resource syncing, run `kubectl api-resources` against the control plane cluster to identify the resource. Use `.` as the key in `customResources`. For example, use `customobjects.example.io` for the `customobjects` resource in the `example.io/v1` API group. At startup, vCluster determines how the control plane cluster serves the resource. It copies a CRD-backed resource's definition into the tenant cluster. For a resource served by an extension [API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/), vCluster generates a schemaless CRD instead. You don't need to specify an API version by default. vCluster uses the storage version for a CRD-backed resource. For an aggregated API resource, it uses the API group's preferred version. To select another version, append it to the resource key in `vcluster.yaml`. ``` sync: toHost: customResources: customobjects.example.io: enabled: true ``` ## Aggregated API resources[​](#aggregated-api-resources "Direct link to Aggregated API resources") Not every extension type in a Kubernetes cluster is backed by a CustomResourceDefinition. Kubernetes can also serve types through [API aggregation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/), where an extension API server registered as an APIService owns the type instead of a CRD. vCluster syncs these resources through the same `customResources` configuration. You don't declare that a resource comes from an extension API server. vCluster resolves every configured resource against the control plane cluster on startup: * If the control plane cluster has a CRD for the resource, vCluster copies that CRD into the tenant cluster. * If it doesn't, vCluster looks the resource up in the control plane cluster's discovery data. When an extension API server serves the resource, vCluster generates a schemaless CRD for it in the tenant cluster. * If neither a CRD nor an extension API server serves the resource, the tenant cluster fails to start. This resolution happens only on startup. If the extension API server isn't serving the resource then, vCluster fails to start and tries again when the control plane pod restarts. An extension API server that appears or recovers later is picked up on that next restart. Because `kubectl get crds` doesn't list resources that an extension API server serves, use `kubectl api-resources` to find them. Configure aggregated API resource sync to host ``` sync: toHost: customResources: customobjects.example.io: enabled: true ``` ### Generated CRDs for aggregated API resources[​](#generated-crds-for-aggregated-api-resources "Direct link to Generated CRDs for aggregated API resources") vCluster builds the generated CRD from the control plane cluster's discovery data. The kind, plural name, scope, and presence of a `status` subresource come from the control plane cluster. vCluster derives the singular name by lowercasing the kind and derives the list kind by appending `List`. It marks the CRD with the `vcluster.loft.sh/generated-from: aggregated-apiserver` annotation and creates or updates it when the tenant cluster starts. Objects of a generated CRD live in the tenant cluster's own data store, like any other custom resource, so [snapshots](https://vcluster.com/docs/vcluster/key-features/snapshots.md) include them. The generated CRD has no resource-specific OpenAPI schema and preserves unknown fields. The tenant API server doesn't validate custom fields against the extension API server's schema. Keep the following behaviors in mind: * `kubectl explain` returns no field documentation for the resource inside the tenant cluster. * The tenant cluster doesn't prune unknown fields. * When the resource key includes no version, vCluster uses the API group's preferred version on the control plane cluster. For CRD-backed resources, it uses the storage version instead. * When you change the version in the resource key, vCluster serves the new version and retains previously stored versions without serving them. vCluster uses the `vcluster.loft.sh/generated-from: aggregated-apiserver` annotation as the adoption marker. If an existing CRD has this marker, vCluster adopts it and updates its specification. If a CRD of the same name doesn't have this marker, the tenant cluster fails to start. Delete that CRD or remove the resource from `customResources`. Removing an entry from `customResources`, or setting `enabled: false`, stops the syncer. vCluster doesn't delete the generated CRD, its objects, or the objects it created on the control plane cluster. The tenant cluster keeps serving the resource, and writes to it stay tenant-local because nothing validates or syncs them. ### Authorization on the control plane cluster[​](#authorization-on-the-control-plane-cluster "Direct link to Authorization on the control plane cluster") vCluster grants itself the permissions it needs for every resource listed in `customResources`. Some extension API servers authorize a request against a related resource that you never configure for sync. For example, an API server can check access to a grouping resource before it serves or accepts an object. Grant those permissions explicitly. This example grants access to a related resource in the vCluster control plane namespace: Grant extra permissions for an extension API server ``` rbac: role: extraRules: - apiGroups: ["example.io"] resources: ["customobjectgroups"] verbs: ["get", "list", "watch"] ``` Use `rbac.role.extraRules` for resources in the vCluster control plane namespace. Use `rbac.clusterRole.extraRules` for cluster-scoped resources or namespaced resources in other control plane cluster namespaces. For details, see [Add custom RBAC rules](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/rbac.md#add-custom-rbac-rules). ### Synchronous validation on the control plane cluster[​](#synchronous-validation-on-the-control-plane-cluster "Direct link to Synchronous validation on the control plane cluster") A generated CRD has no schema, so the tenant cluster can't validate an aggregated API resource on its own. vCluster validates it on the control plane cluster instead. When a client creates, updates, or patches such a resource, vCluster writes the change through to the extension API server before it answers the request. * If the extension API server accepts the object, the request succeeds. * If the extension API server rejects the object, vCluster attempts to roll the tenant write back, and returns the rejection to the client. vCluster rewrites the control plane cluster's object and namespace names in the message, details, and causes. The error refers to the tenant object. Write-through validates the request. It doesn't replace syncing. The [syncer](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync) still reconciles drift in both directions, propagates deletions, and removes control plane cluster objects that an interrupted request left behind. Key behaviors for write-through: * Only `create`, `update`, and `patch` requests on the resource itself are written through. Requests to subresources aren't. * Deletes aren't written through. The syncer removes the control plane cluster object after the tenant delete has already returned. * A rejected create exists in the tenant cluster until the rollback removes it, so watchers can observe a create followed by a delete. * A rejected write can stay in the tenant cluster for good. Rollback is best-effort, so a failed request isn't proof that nothing was stored. * vCluster pins each update and patch to the object it snapshotted for rollback. When a concurrent write lands first, the request fails with `409 Conflict` instead of being reverted. Retry it against the current object. * Dry-run requests aren't written through. `kubectl apply --dry-run=server` validates against the tenant cluster only, so it doesn't report extension API server errors. * vCluster rejects writes with `503 Service Unavailable` until the syncer finishes registering the resource on startup. This keeps writes from skipping validation. * vCluster rejects a write body it can't prepare for rollback with `415 Unsupported Media Type`. JSON, YAML, and server-side apply bodies are supported. * Each control plane replica allows up to 50 write-through requests at a time and rejects further writes with `429 Too Many Requests`. A write that the control plane cluster doesn't answer within 30 seconds fails. * The control plane cluster's audit log attributes writes that start in the tenant cluster to the vCluster identity. Deletion is asynchronous. The tenant delete returns as soon as the tenant cluster removes the object, and the syncer removes the control plane cluster object afterward. During that window the control plane cluster object still exists, and an extension API server that enforces policy keeps enforcing it. A successful delete in the tenant cluster isn't proof that enforcement stopped. vCluster can't tell its own rejected write apart from a newer one. It skips the rollback when another write reaches the object first. It also gives up after a few conflicting attempts. An abandoned rollback leaves the object in the tenant cluster. The control plane cluster keeps rejecting it, so the syncer retries without ever succeeding. The `vcluster_writethrough_rollback_failures_total` metric counts these cases. ### The control plane cluster owns the status[​](#the-control-plane-cluster-owns-the-status "Direct link to The control plane cluster owns the status") When an aggregated API resource has a `status` subresource, the status syncs from the control plane cluster to the tenant cluster only. vCluster doesn't write tenant status updates through, and the syncer doesn't push them either. Tenant status fields remain local until the same fields change on the control plane cluster. ### Limitations[​](#limitations "Direct link to Limitations") The generated CRD serves the resource itself, plus the `status` subresource when the control plane cluster serves it. Extension API servers can expose additional subresources, such as `scale`, `logs`, or `exec`. Those aren't available inside the tenant cluster, because a CRD can't serve them. Only namespace-scoped resources sync to the control plane cluster, the same as for CRD-backed resources. ## Versioned example[​](#versioned-example "Direct link to Versioned example") To specify a particular version of a CustomResourceDefinition (CRD), append the version to the resource type, separated by a `/`, using the format `/` in your configuration. The specified version must exist in the control plane cluster when vCluster starts. This version becomes the storage version within the vCluster. When the storage version in the vCluster differs from the control plane cluster's storage version, vCluster automatically converts custom resources between the two versions. For example, to use version `v1` of the `certificaterequests.cert-manager.io` CRD, specify: Configure CR sync to host ``` sync: toHost: customResources: certificaterequests.cert-manager.io/v1: enabled: true ``` Unique CRD versions vCluster supports syncing only one version of a custom resource. If you specify multiple versions, the tenant cluster fails to start. ### Upgrade custom resources with explicit CRD API versions[​](#upgrade-custom-resources-with-explicit-crd-api-versions "Direct link to Upgrade custom resources with explicit CRD API versions") vCluster supports syncing only one version of a CRD. If multiple versions are specified in the sync configuration, the tenant cluster fails to start. vCluster modifies the CRD inside the tenant cluster during upgrades if it detects that a required version is missing. The following scenarios describe how vCluster handles CRD versions when you upgrade the tenant cluster or change the sync configuration. #### Upgrade from no version to specified version[​](#upgrade-from-no-version-to-specified-version "Direct link to Upgrade from no version to specified version") When upgrading from an unversioned CRD to a versioned one (`certificaterequests.cert-manager.io` → `certificaterequests.cert-manager.io/v1`), vCluster checks the CRD's storage version. If the storage version is not `v1`, vCluster updates the CRD to include `v1` as a new version in addition to the current storage version. #### Upgrade from one specified version to another specified version[​](#upgrade-from-one-specified-version-to-another-specified-version "Direct link to Upgrade from one specified version to another specified version") When upgrading between versioned CRDs (`certificaterequests.cert-manager.io/v1` → `certificaterequests.cert-manager.io/v2`), the CRD is updated to include the new version `v2`, and the old version `v1` is kept as well. #### Upgrade from a specified version to no version specified[​](#upgrade-from-a-specified-version-to-no-version-specified "Direct link to Upgrade from a specified version to no version specified") When upgrading from a versioned CRD to an unversioned one (`certificaterequests.cert-manager.io/v1` → `certificaterequests.cert-manager.io`), the behavior depends on the host's storage version: * If the CRD's storage version in the control plane cluster is `v1`, nothing happens and the CRD in the tenant cluster remains as `certificaterequests.cert-manager.io/v1`. * If the storage version is not `v1`, vCluster updates the CRD to include the storage version as a new version in addition to `v1`. ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.customResources..patches` to transform custom resource fields while syncing to the control plane cluster. Custom resources support expression patches, reference patches, and labels patches. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for the full syntax and directionality rules. Reference patches are useful when a custom resource field points to another synced resource, such as a Secret: ``` sync: toHost: customResources: customobjects.example.io: enabled: true patches: - path: spec.secretName reference: apiVersion: v1 kind: Secret ``` Expression patches are useful when the tenant and control plane cluster need different field values: ``` sync: toHost: customResources: customobjects.example.io: enabled: true patches: - path: spec.hosts[*] expression: "value.startsWith('prod-') ? value : `prod-${value}`" reverseExpression: "value.startsWith('prod-') ? value.slice(5) : value" ``` A selector held as an opaque string can't be translated, so vCluster copies it verbatim. In the default single-namespace sync mode, every tenant namespace maps to the vCluster namespace on the control plane cluster. A copied selector can therefore match objects the policy was never written for. An expression patch can scope the field to the namespace it came from. For a worked example, see [Sync Calico network policies to the control plane cluster](https://vcluster.com/docs/vcluster/learn-how-to/sync-calico-network-policies.md). ## Sync an Istio waypoint Gateway using custom resources[​](#sync-an-istio-waypoint-gateway-using-custom-resources "Direct link to Sync an Istio waypoint Gateway using custom resources") Use native Gateway API sync for core Gateway API resources For syncing core Kubernetes Gateway API resources (`Gateway`, `HTTPRoute`, `TLSRoute`, `ReferenceGrant`, `BackendTLSPolicy`), use [native Gateway API sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md) instead of custom-resource syncing. The example below covers Istio waypoint Gateways using the `HBONE` listener, which is outside the standard Gateway API channel and still requires the custom-resource path. Install the Gateway API CRDs in the control plane cluster first. See [Create waypoint proxy in the host](https://vcluster.com/docs/vcluster/integrations/istio.md#create-waypoint-proxy-in-the-host) for the CRD installation step. To-host sync goes from the tenant cluster to the control plane cluster, so create the waypoint Gateway in the tenant cluster instead of the control plane cluster. Enable sync for the resource: Sync an Istio waypoint Gateway to the control plane cluster ``` sync: toHost: customResources: gateways.gateway.networking.k8s.io: enabled: true ``` Then create the same Gateway manifest shown in the Istio integration guide, but apply it to the tenant cluster. Point `kubectl` at your tenant cluster's context, not the control plane cluster context used in the linked step, and add `-n` if the workloads it routes for aren't in the current namespace: waypoint-gateway.yaml ``` apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: waypoint labels: istio.io/waypoint-for: service spec: gatewayClassName: istio-waypoint listeners: - name: mesh port: 15008 protocol: HBONE ``` Create Waypoint Gateway in the tenant cluster ``` kubectl --context="" create -f waypoint-gateway.yaml ``` vCluster syncs the Gateway to the control plane cluster, where Istio's waypoint controller reconciles it into a running proxy. ## Config reference[​](#config-reference "Direct link to Config reference") ## `customResources` {key: object} [​](#customResources "Direct link to customResources")CustomResources defines what custom resources should get synced from the virtual cluster to the host cluster. vCluster will copy the definition automatically from host cluster to virtual cluster on startup. vCluster will also automatically add any required RBAC permissions to the vCluster role for this to work. ### `enabled` required boolean [​](#customResources-enabled "Direct link to customResources-enabled")Enabled defines if this option should be enabled. ### `scope` string [​](#customResources-scope "Direct link to customResources-scope")Scope defines the scope of the resource. If undefined, will use Namespaced. Currently only Namespaced is supported. ### `patches` object\[] [​](#customResources-patches "Direct link to customResources-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#customResources-patches-path "Direct link to customResources-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#customResources-patches-expression "Direct link to customResources-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#customResources-patches-reverseExpression "Direct link to customResources-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#customResources-patches-reference "Direct link to customResources-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#customResources-patches-reference-apiVersion "Direct link to customResources-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#customResources-patches-reference-apiVersionPath "Direct link to customResources-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#customResources-patches-reference-kind "Direct link to customResources-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#customResources-patches-reference-kindPath "Direct link to customResources-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#customResources-patches-reference-namePath "Direct link to customResources-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#customResources-patches-reference-namespacePath "Direct link to customResources-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#customResources-patches-labels "Direct link to customResources-patches-labels")Labels treats the path value as a labels selector. --- # Namespaces | | | Enterprise | | | | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Sync Namespaces toHost](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/namespaces.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** ## Overview[​](#overview "Direct link to Overview") Namespace syncing is an advanced vCluster feature that creates dedicated namespaces on the control plane cluster corresponding to namespaces in the tenant cluster. This feature preserves original resource names and enables complex integrations with cloud providers and custom schedulers. ### Why use namespace syncing?[​](#why-use-namespace-syncing "Direct link to Why use namespace syncing?") Use namespace syncing when you need: * **Predictable resource names**: Cloud IAM integrations (like GCP Workload Identity) that require specific ServiceAccount names * **Custom scheduler support**: Tools like NVIDIA Run:ai that manage hierarchies with quotas and priorities * **Direct namespace mapping**: Scenarios where name rewriting would break external integrations * **Multi-tenant isolation**: Better separation of resources across different virtual namespaces info By default, namespace syncing is **disabled**. vCluster rewrites resource names (e.g., `test` → `test-x-my-namespace-x-my-vcluster`) to avoid conflicts and syncs everything to the vCluster's host namespace. ## How namespace syncing works[​](#how-namespace-syncing-works "Direct link to How namespace syncing works") When enabled, namespace syncing creates dedicated host namespaces that map to virtual namespaces, preserving original resource names without rewriting. This enables IAM integrations (like GCP Workload Identity) and custom schedulers (like NVIDIA Run:ai) that require predictable resource names. note If namespace syncing is not enabled or if a namespace doesn't match any mapping rule, vCluster syncs resources into the vCluster control plane's namespace on the host using rewritten names to avoid conflicts. ## Enable namespace syncing[​](#enable-namespace-syncing "Direct link to Enable namespace syncing") Configure namespace syncing by enabling the feature and defining mapping rules. Configuration is Immutable Namespace syncing configuration cannot be modified after vCluster deployment. Plan your mapping strategy carefully before initial deployment. ### Basic configuration[​](#basic-configuration "Direct link to Basic configuration") ``` sync: toHost: namespaces: enabled: true mappings: # mappings are mandatory byName: example-virtual-namespace: example-host-namespace ``` When enabled, vCluster creates corresponding namespaces on the control plane cluster for each namespace created in the vCluster, based on the defined mapping rules. ## Import existing namespaces[​](#import-existing-namespaces "Direct link to Import existing namespaces") Automatic Import Behavior Any host namespace matching a defined mapping (exact or pattern) is automatically imported into the vCluster, including all existing workloads. When a host namespace matches your mapping rules: 1. vCluster automatically imports the namespace 2. Existing pods and resources appear in the tenant cluster 3. Resources created in either direction sync bidirectionally ## Configure mapping rule types[​](#configure-mapping-rule-types "Direct link to Configure mapping rule types") ### Available mapping patterns[​](#available-mapping-patterns "Direct link to Available mapping patterns") The `mappings.byName` field supports these patterns: * **Exact mappings**: Map a specific virtual namespace name to a specific host namespace name. * **Pattern mappings**: Use a single wildcard `*` character to map a range of virtual namespaces to a corresponding range of host namespaces. The wildcard content mirrors between virtual and host. * **`${name}` variable**: This variable substitutes automatically with the vCluster instance's name. Use this to create predictable host namespaces that clearly associate with a particular vCluster instance. Combine the `${name}` variable with pattern mappings for flexible namespace organization. important Define only exact-to-exact or pattern-to-pattern mappings. Mixing these types (exact virtual namespace to patterned host namespace, or patterned virtual namespace to exact host namespace) is not supported. ### Configuration examples[​](#configuration-examples "Direct link to Configuration examples") Here are different mapping strategies you can use: vcluster-mappings-example.yaml ``` sync: toHost: namespaces: enabled: true mappings: byName: # Exact mapping: # Virtual NS "frontend-prod" -> Host NS "customer-project-alpha-prod" "frontend-prod": "customer-project-alpha-prod" # Pattern mapping: # Virtual NS "team-a-*" -> Host NS "h-team-a-*" # e.g., vNS "team-a-dev" -> hNS "h-team-a-dev" # e.g., vNS "team-a-staging" -> hNS "h-team-a-staging" "team-a-*": "h-team-a-*" # Pattern mapping using ${name} variable: # Assumes vCluster name is "my-vc" # Virtual NS "datasets-*" -> Host NS "my-vc-data-*" # e.g., vNS "datasets-raw" -> hNS "my-vc-data-raw" # e.g., vNS "datasets-processed" -> hNS "my-vc-data-processed" "datasets-*": "${name}-data-*" ``` ## Enforce only syncing defined mappings[​](#enforce-only-syncing-defined-mappings "Direct link to Enforce only syncing defined mappings") By default (`mappingsOnly: false`), virtual namespaces that don't match any mapping rules still work - they sync to the vCluster's host namespace using name-rewriting. ### Restrict to mapped namespaces only[​](#restrict-to-mapped-namespaces-only "Direct link to Restrict to mapped namespaces only") To strictly enforce mappings and block unmapped namespaces, enable `mappingsOnly:true`: ``` # vcluster-mappings-only-example.yaml sync: toHost: namespaces: enabled: true mappingsOnly: true # Enable strict mapping mappings: byName: # Only virtual namespaces starting with "synced-" are mapped "synced-*": "host-${name}-synced-*" # And the virtual namespace "important-configs" maps to "host-configs-critical" "important-configs": "host-configs-critical" ``` ## Delete a vCluster[​](#delete-a-vcluster "Direct link to Delete a vCluster") When you delete a vCluster, namespaces and resources created by the vCluster on the host are removed. Resources that originated from the control plane cluster and were imported remain unchanged - vCluster only cleans up what it created. You can test this behavior by deploying vCluster with the following configuration: vcluster-test-cleanup.yaml ``` sync: toHost: namespaces: enabled: true mappings: byName: # Test mapping: 'sync-*' in vCluster maps to 'host-*' on host # This helps demonstrate what gets deleted vs preserved sync-*: host-* ``` ## Examples[​](#examples "Direct link to Examples") The following end-to-end examples demonstrate namespace syncing behavior in both tenant and control plane clusters. ### Example: Enable namespace mappings[​](#example-enable-namespace-mappings "Direct link to Example: Enable namespace mappings") This example demonstrates how different namespace mapping rules affect resource creation and placement on the control plane cluster. 1. Deploy a vCluster named `mapping-demo` using the `vCluster-mappings-example.yaml` file from the preceding example. ``` vcluster create mapping-demo -f vCluster-mappings-example.yaml ``` 2. Inside the vCluster, create a namespace that matches the `team-a-*` pattern. ``` kubectl create namespace team-a-ns1 ``` 3. Create a namespace named `frontend-prod` inside the vCluster. This matches the exact mapping rule `"frontend-prod": "customer-project-alpha-prod"` defined in your `vcluster.yaml`. ``` kubectl create namespace frontend-prod ``` 4. Create a namespace that does not match any mapping rule: ``` kubectl create namespace some-other-namespace ``` 5. Verify the namespaces on the control plane cluster: ``` kubectl --context kind-vcluster get namespace ``` The output is similar to the following: ``` NAME STATUS AGE customer-project-alpha-prod Active 12s default Active 45h h-team-a-ns1 Active 20s kube-node-lease Active 45h kube-public Active 45h kube-system Active 45h local-path-storage Active 45h vcluster-mapping-demo Active 2m11s ``` Only namespaces matched by mappings sync directly to the control plane cluster. 6. Run a pod inside the mapped namespace: ``` kubectl -n team-a-ns1 run nginx-from-team-a --image nginx ``` 7. Run a pod inside the non-mapped namespace: ``` kubectl -n some-other-namespace run nginx-from-other-namespace --image nginx ``` 8. Verify both pods run inside vCluster: ``` kubectl get pods -A ``` The output is similar to the following: ``` NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-94f599b5-wdgzg 1/1 Running 0 1m some-other-namespace nginx-from-other-namespace 1/1 Running 0 11s team-a-ns1 nginx-from-team-a 1/1 Running 0 18s ``` 9. Verify pods running on the control plane cluster: ``` kubectl --context kind-vcluster get pods -A ``` The output is similar to the following: ``` NAMESPACE NAME READY STATUS RESTARTS AGE h-team-a-ns1 nginx-from-team-a 1/1 Running 0 58s kube-system coredns-787d4945fb-5ljfb 1/1 Running 1 (20h ago) 45h kube-system coredns-787d4945fb-zh9q7 1/1 Running 1 (20h ago) 45h kube-system etcd-vcluster-pro-control-plane 1/1 Running 1 (20h ago) 45h kube-system kindnet-8mmjr 1/1 Running 1 (20h ago) 45h kube-system kube-apiserver-vcluster-pro-control-plane 1/1 Running 1 (20h ago) 45h kube-system kube-controller-manager-vcluster-pro-control-plane 1/1 Running 1 (20h ago) 45h kube-system kube-proxy-jnwfv 1/1 Running 1 (20h ago) 45h kube-system kube-scheduler-vcluster-pro-control-plane 1/1 Running 1 (20h ago) 45h local-path-storage local-path-provisioner-75f5b54ffd-j6pjj 1/1 Running 1 (20h ago) 45h vcluster-mapping-demo coredns-94f599b5-wdgzg-x-kube-system-x-default-namespace-sync 1/1 Running 0 1m vcluster-mapping-demo mapping-demo-0 1/1 Running 0 86s vcluster-default-namespace-sync nginx-from-other-namespace-x-some-other-namespace-x--4b96e8f948 1/1 Running 0 51s ``` The pod from the mapped namespace syncs directly without renaming, while the pod from the non-mapped namespace uses default translation logic. ### Example: Import existing host namespaces[​](#example-import-existing-host-namespaces "Direct link to Example: Import existing host namespaces") This example shows how vCluster automatically imports host namespaces that match your mapping rules, including any existing workloads. 1. Create a namespace matching one of the mappings on host. This matches `"team-a-*": "h-team-a-*"` mapping. ``` kubectl --context kind-vcluster create namespace h-team-a-namespace-synced-from-host ``` 2. Start a pod inside this namespace on the control plane cluster: ``` kubectl --context kind-vcluster -n h-team-a-namespace-synced-from-host run nginx-from-host --image nginx ``` 3. Inside vCluster get namespaces: ``` kubectl get namespace ``` Namespace was imported from host following defined mapping. ``` NAME STATUS AGE default Active 58s kube-node-lease Active 58s kube-public Active 58s kube-system Active 58s team-a-namespace-synced-from-host Active 14s ``` 4. Check the pod running in this namespace: ``` kubectl -n team-a-namespace-synced-from-host get pods ``` Pod was imported into vCluster. ``` NAME READY STATUS RESTARTS AGE nginx-from-host 1/1 Running 0 36s ``` ### Example: Sync only mapped namespaces[​](#example-sync-only-mapped-namespaces "Direct link to Example: Sync only mapped namespaces") This example demonstrates strict namespace control using `mappingsOnly: true` to prevent creation of unmapped namespaces. 1. Deploy a vCluster named `strict-vc` with the `vCluster-mappings.yaml` configuration: ``` vcluster create strict-vc -f vCluster-mappings-only.yaml ``` 2. Inside the vCluster, create a namespace that matches a mapping rule: ``` kubectl create namespace synced-data-alpha ``` This matches `synced-*`. vCluster creates the `host-strict-vc-synced-data-alpha` namespace on the host, and resources created in the virtual `synced-data-alpha` namespace appear there. 3. Create a namespace that does not match any rule: ``` kubectl create namespace experimental-app ``` The output is similar to the following: ``` Error from server (Forbidden): Virtual namespace 'experimental-app' is not allowed by vcluster mappings. Allowed namespaces are important-configs, synced-* (post namespaces) ``` vCluster blocks this namespace. Any operation on namespaces not defined in mappings gets blocked when `mappingsOnly` mode is enabled. ### Example: Delete a vCluster with namespace syncing[​](#example-delete-a-vcluster-with-namespace-syncing "Direct link to Example: Delete a vCluster with namespace syncing") This example demonstrates the cleanup behavior when deleting a vCluster, showing which resources are removed and which are preserved. 1. Create a namespace on your control plane cluster that matches host-side mappings in the configuration and start a pod inside of it: ``` kubectl --context kind-vcluster create namespace host-namespace-from-host kubectl --context kind-vcluster -n host-namespace-from-host run nginx-from-host --image nginx ``` The commands run an NGINX pod directly in the control plane cluster namespace. This pod gets imported into vCluster when the vCluster starts. 2. Verify that the pod started in the control plane cluster: ``` kubectl --context kind-vcluster -n host-namespace-from-host get pods ``` The output is similar to the following: ``` NAME READY STATUS RESTARTS AGE nginx-from-host 1/1 Running 0 50s ``` 3. Create a new vCluster instance with the namespace mapping configuration that imports existing host namespaces: ``` vcluster create test-cleanup -f values-test-cleanup.yaml ``` 4. Create a namespace inside vCluster and start a pod in it: ``` kubectl create namespace sync-namespace-from-vcluster kubectl -n sync-namespace-from-vcluster run nginx-from-vcluster --image nginx ``` This runs an NGINX pod inside the tenant cluster namespace. This gets synced to the control plane cluster. 5. Verify both namespaces are available in both the control plane cluster and the tenant cluster: ``` kubectl get ns ``` The output is similar to the following: ``` NAME STATUS AGE default Active 3m35s kube-node-lease Active 3m35s kube-public Active 3m35s kube-system Active 3m35s sync-namespace-from-host Active 2m21s sync-namespace-from-vcluster Active 5s ``` List namespaces as they exist on the control plane cluster (show both original control plane cluster namespaces and synced vCluster namespaces): ``` kubectl --context kind-vcluster get ns ``` The output is similar to the following: ``` NAME STATUS AGE default Active 6d23h host-namespace-from-host Active 3m3s host-namespace-from-vcluster Active 47s kube-node-lease Active 6d23h kube-public Active 6d23h kube-system Active 6d23h local-path-storage Active 6d23h vcluster-test-cleanup Active 4m43s ``` Show all `nginx` pods as seen from inside the vCluster. The following command displays the tenant cluster's view of resources: View pods from vCluster perspective ``` kubectl get po -A | grep nginx ``` The output is similar to the following: ``` sync-namespace-from-host nginx-from-host 1/1 Running 0 3m58s sync-namespace-from-vcluster nginx-from-vcluster 1/1 Running 0 114s ``` Shows all `nginx` pods as seen from the control plane cluster. The following command displays how the same resources appear on the physical cluster: View pods from control plane cluster perspective ``` kubectl --context kind-vcluster get po -A | grep nginx ``` The output is similar to the following: ``` host-namespace-from-host nginx-from-host 1/1 Running 0 4m7s host-namespace-from-vcluster nginx-from-vcluster 1/1 Running 0 2m3s ``` 6. Delete vCluster: ``` vcluster delete test-cleanup ``` The output is similar to the following: ``` 12:29:31 info Stopping background proxy... 12:29:31 info Delete vcluster test-cleanup... 12:29:31 done Successfully deleted virtual cluster test-cleanup in namespace vcluster-test-cleanup 12:29:31 info Starting cleanup of vCluster 'test-cleanup' namespaces. 12:29:31 info Namespace 'host-namespace-from-host' was imported, cleaning up its resources and metadata... 12:29:36 info Deleting virtual cluster namespace 'host-namespace-from-vcluster' 12:29:36 info Successfully deleted virtual cluster namespace 'host-namespace-from-vcluster' 12:29:36 info Cleanup of vCluster 'test-cleanup' namespaces finished. 12:29:36 done Successfully deleted virtual cluster namespace vcluster-test-cleanup 12:29:36 info Waiting for virtual cluster to be deleted... 12:29:47 done Virtual Cluster is deleted ``` 7. Verify namespaces and resources left on host: ``` kubectl --context kind-vcluster get ns ``` The output is similar to the following: ``` NAME STATUS AGE default Active 6d23h host-namespace-from-host Active 6m55s kube-node-lease Active 6d23h kube-public Active 6d23h kube-system Active 6d23h local-path-storage Active 6d23h ``` ``` kubectl --context kind-vcluster get pods -n host-namespace-from-host ``` The output is similar to the following: ``` NAME READY STATUS RESTARTS AGE nginx-from-host 1/1 Running 0 6m48s ``` Resources synced from vCluster are gone while all resources created on the control plane cluster remained untouched. ## Known limitations[​](#known-limitations "Direct link to Known limitations") ### Incompatibility with generic sync[​](#incompatibility-with-generic-sync "Direct link to Incompatibility with generic sync") The namespace syncing feature is incompatible with the older `experimental.genericSync` feature. If you enable `sync.toHost.namespaces`, you must NOT use `genericSync`. Instead, rely on the specific `sync.toHost` and `sync.fromHost` configurations for resource synchronization. ### Automatic syncing of all secrets and ConfigMaps[​](#automatic-syncing-of-all-secrets-and-configmaps "Direct link to Automatic syncing of all secrets and ConfigMaps") When namespace syncing is enabled, all `Secrets` and `ConfigMaps` automatically sync to host namespaces (equivalent to `sync.toHost.secrets.all: true` and `sync.toHost.configMaps.all: true`). This cannot be disabled and makes these resources visible on the control plane cluster. ### NetworkPolicy syncing is disabled[​](#networkpolicy-syncing-is-disabled "Direct link to NetworkPolicy syncing is disabled") The `sync.toHost.networkPolicies` feature is not supported with namespace syncing. NetworkPolicy objects created in the vCluster won't sync to the host and have no effect on network traffic. ### Inter-namespace pod affinity[​](#inter-namespace-pod-affinity "Direct link to Inter-namespace pod affinity") `podAffinity` rules that reference pods in other namespaces are not translated by vCluster, causing pods to remain `Pending`. Intra-namespace `podAffinity` rules work as expected. #### Example: Inter-namespace affinity (fails)[​](#example-inter-namespace-affinity-fails "Direct link to Example: Inter-namespace affinity (fails)") This example attempts to schedule pods in one namespace with affinity to pods in another namespace. It fails because vCluster cannot translate cross-namespace references in pod affinity rules, causing pods to remain in Pending state indefinitely. ``` # Namespace for the target application apiVersion: v1 kind: Namespace metadata: name: sync-a --- # Namespace for the follower application apiVersion: v1 kind: Namespace metadata: name: sync-b --- # --- Target App Deployment --- apiVersion: apps/v1 kind: Deployment metadata: name: target-app namespace: sync-a spec: replicas: 2 selector: matchLabels: app: target template: metadata: labels: app: target spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - target topologyKey: "kubernetes.io/hostname" containers: - name: nginx image: nginx --- # --- Follower App with Failing Affinity --- apiVersion: apps/v1 kind: Deployment metadata: name: follower-app namespace: sync-b spec: replicas: 2 selector: matchLabels: app: follower template: metadata: labels: app: follower spec: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - target # This cross-namespace reference will not work namespaces: ["sync-a"] topologyKey: "kubernetes.io/hostname" containers: - name: busybox image: busybox command: ["sleep", "3600"] ``` After applying this manifest, the `follower-app` pods remain stuck indefinitely in the `Pending` state. If you run: ``` kubectl get po -A -o wide ``` You should see output similar to ``` NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sync-a target-app-7d8df4d5d4-tlpmc 1/1 Running 0 3s 10.244.2.24 loft-worker sync-a target-app-7d8df4d5d4-vhct5 1/1 Running 0 3s 10.244.1.17 loft-worker2 sync-b follower-app-6b5ccc86dd-8ssw6 0/1 Pending 0 3s sync-b follower-app-6b5ccc86dd-kjkkg 0/1 Pending 0 3s ``` #### Example: Intra-namespace affinity (works)[​](#example-intra-namespace-affinity-works "Direct link to Example: Intra-namespace affinity (works)") This example shows how to correctly configure pod affinity within the same namespace. By placing both deployments in the same namespace without cross-namespace references, the affinity rules work as expected. ``` # Namespace for the demo apiVersion: v1 kind: Namespace metadata: name: sync-demo --- # --- Target App Deployment --- apiVersion: apps/v1 kind: Deployment metadata: name: target-app namespace: sync-demo spec: replicas: 2 selector: matchLabels: app: target template: metadata: labels: app: target spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - target topologyKey: "kubernetes.io/hostname" containers: - name: nginx image: nginx --- # --- Follower App with Working Affinity --- apiVersion: apps/v1 kind: Deployment metadata: name: follower-app namespace: sync-demo spec: replicas: 2 selector: matchLabels: app: follower template: metadata: labels: app: follower spec: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - target topologyKey: "kubernetes.io/hostname" containers: - name: busybox image: busybox command: ["sleep", "3600"] ``` After applying this manifest, all pods are in the `Running` state, and the `follower-app` pods are co-located on the same nodes as the `target-app` pods. If you run: ``` kubectl get po -A -o wide ``` You should see output similar to ``` NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-744d9bd8bf-ck47h 1/1 Running 0 21m 10.244.1.11 loft-worker2 sync-demo follower-app-6f6df5d4fb-kvvcz 1/1 Running 0 7s 10.244.1.19 loft-worker2 sync-demo follower-app-6f6df5d4fb-rsfjf 1/1 Running 0 7s 10.244.2.26 loft-worker sync-demo target-app-7d8df4d5d4-czfrd 1/1 Running 0 8s 10.244.1.18 loft-worker2 sync-demo target-app-7d8df4d5d4-lblcc 1/1 Running 0 8s 10.244.2.25 loft-worker ``` --- # Pod disruption budgets Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. Sync [PodDisruptionBudget](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets) resources from the tenant cluster to the control plane cluster. ### Sync PodDisruptionBudgets from the tenant to control plane cluster[​](#sync-poddisruptionbudgets-from-the-tenant-to-control-plane-cluster "Direct link to Sync PodDisruptionBudgets from the tenant to control plane cluster") ``` sync: toHost: podDisruptionBudgets: enabled: true ``` ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.podDisruptionBudgets.patches` to transform PodDisruptionBudget fields while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") ## `podDisruptionBudgets` object [​](#podDisruptionBudgets "Direct link to podDisruptionBudgets")PodDisruptionBudgets defines if pod disruption budgets created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean false [​](#podDisruptionBudgets-enabled "Direct link to podDisruptionBudgets-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#podDisruptionBudgets-patches "Direct link to podDisruptionBudgets-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#podDisruptionBudgets-patches-path "Direct link to podDisruptionBudgets-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#podDisruptionBudgets-patches-expression "Direct link to podDisruptionBudgets-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#podDisruptionBudgets-patches-reverseExpression "Direct link to podDisruptionBudgets-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#podDisruptionBudgets-patches-reference "Direct link to podDisruptionBudgets-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#podDisruptionBudgets-patches-reference-apiVersion "Direct link to podDisruptionBudgets-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#podDisruptionBudgets-patches-reference-apiVersionPath "Direct link to podDisruptionBudgets-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#podDisruptionBudgets-patches-reference-kind "Direct link to podDisruptionBudgets-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#podDisruptionBudgets-patches-reference-kindPath "Direct link to podDisruptionBudgets-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#podDisruptionBudgets-patches-reference-namePath "Direct link to podDisruptionBudgets-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#podDisruptionBudgets-patches-reference-namespacePath "Direct link to podDisruptionBudgets-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#podDisruptionBudgets-patches-labels "Direct link to podDisruptionBudgets-patches-labels")Labels treats the path value as a labels selector. --- # Priority classes Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. Sync [PriorityClass](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass) resources from the tenant cluster to the control plane cluster. Enable this feature if you need to configure the host scheduler to use a certain PriorityClass that is created within the tenant cluster when scheduling the tenant cluster workloads. When enabled, will update the vCluster control plane's ClusterRole to include the necessary permissions. Cross-tenant preemption on shared nodes On shared nodes, a synced PriorityClass can preempt or starve co-tenant workloads on the same nodes. PriorityClass affects scheduling across the whole control plane cluster, not just the tenant cluster it came from. Leave this disabled unless required. If you enable it, constrain synced values through control plane cluster admission rather than trusting tenant-supplied values. See [operational hardening and lifecycle](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#operational-hardening-and-lifecycle). ### Sync PriorityClasses from the tenant to control plane cluster[​](#sync-priorityclasses-from-the-tenant-to-control-plane-cluster "Direct link to Sync PriorityClasses from the tenant to control plane cluster") ``` sync: toHost: priorityClasses: enabled: true ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `priorityClasses` object [​](#priorityClasses "Direct link to priorityClasses")PriorityClasses defines if priority classes created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean false [​](#priorityClasses-enabled "Direct link to priorityClasses-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#priorityClasses-patches "Direct link to priorityClasses-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#priorityClasses-patches-path "Direct link to priorityClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#priorityClasses-patches-expression "Direct link to priorityClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#priorityClasses-patches-reverseExpression "Direct link to priorityClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#priorityClasses-patches-reference "Direct link to priorityClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#priorityClasses-patches-reference-apiVersion "Direct link to priorityClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#priorityClasses-patches-reference-apiVersionPath "Direct link to priorityClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#priorityClasses-patches-reference-kind "Direct link to priorityClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#priorityClasses-patches-reference-kindPath "Direct link to priorityClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#priorityClasses-patches-reference-namePath "Direct link to priorityClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#priorityClasses-patches-reference-namespacePath "Direct link to priorityClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#priorityClasses-patches-labels "Direct link to priorityClasses-patches-labels")Labels treats the path value as a labels selector. --- # Service accounts Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. Sync [ServiceAccount](https://kubernetes.io/docs/concepts/security/service-accounts/) resources from the tenant cluster to the control plane cluster. This is useful for using [AWS IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) with the tenant cluster. In this scenario, the control plane cluster is defined as the identity provider to access your OIDC-compatible system, which means you don't have to configure each tenant cluster as a provider. vCluster creates a control plane cluster ServiceAccount token for your synchronized ServiceAccount from the tenant cluster. Synced ServiceAccount names are rewritten vCluster renames the ServiceAccount when it syncs it into the vCluster control plane's namespace, to `NAME-x-NAMESPACE-x-VCLUSTER_NAME`. vCluster truncates rewritten names longer than 63 characters and appends a hash suffix. Cloud IAM trust policies must reference the rewritten name, so build it with the Platform API rather than hard-coding it. With [namespace syncing](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/namespaces.md) enabled, a ServiceAccount in a mapped namespace keeps its original name and only its namespace changes. See [Resource name length and truncation](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync#resource-name-length-and-truncation) for the rule, and [Integrate EKS Pod Identity with vCluster](https://vcluster.com/docs/vcluster/third-party-integrations/pod-identity/eks-pod-identity.md) for a worked example. ### Sync ServiceAccounts from the tenant to control plane cluster[​](#sync-serviceaccounts-from-the-tenant-to-control-plane-cluster "Direct link to Sync ServiceAccounts from the tenant to control plane cluster") ``` sync: toHost: serviceAccounts: enabled: true ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `serviceAccounts` object [​](#serviceAccounts "Direct link to serviceAccounts")ServiceAccounts defines if service accounts created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean false [​](#serviceAccounts-enabled "Direct link to serviceAccounts-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#serviceAccounts-patches "Direct link to serviceAccounts-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#serviceAccounts-patches-path "Direct link to serviceAccounts-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#serviceAccounts-patches-expression "Direct link to serviceAccounts-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#serviceAccounts-patches-reverseExpression "Direct link to serviceAccounts-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#serviceAccounts-patches-reference "Direct link to serviceAccounts-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#serviceAccounts-patches-reference-apiVersion "Direct link to serviceAccounts-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#serviceAccounts-patches-reference-apiVersionPath "Direct link to serviceAccounts-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#serviceAccounts-patches-reference-kind "Direct link to serviceAccounts-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#serviceAccounts-patches-reference-kindPath "Direct link to serviceAccounts-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#serviceAccounts-patches-reference-namePath "Direct link to serviceAccounts-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#serviceAccounts-patches-reference-namespacePath "Direct link to serviceAccounts-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#serviceAccounts-patches-labels "Direct link to serviceAccounts-patches-labels")Labels treats the path value as a labels selector. --- # Config maps Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is enabled. Sync [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) resources that are used by pods from the tenant cluster to the control plane cluster. Apps frequently need configuration data to function. ### Sync only used ConfigMaps from the tenant to control plane cluster[​](#sync-only-used-configmaps-from-the-tenant-to-control-plane-cluster "Direct link to Sync only used ConfigMaps from the tenant to control plane cluster") Sync any ConfigMap that is used by a Pod synced from the tenant to control plane cluster. ``` sync: toHost: configMaps: enabled: true ``` ### Sync all ConfigMaps from the tenant to control plane cluster[​](#sync-all-configmaps-from-the-tenant-to-control-plane-cluster "Direct link to Sync all ConfigMaps from the tenant to control plane cluster") If you have a resource in the control plane cluster monitoring ConfigMap objects that aren't required by a Pod to run, then you can enable the `all` option. ``` sync: toHost: configMaps: enabled: true all: true ``` ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.configMaps.patches` to transform ConfigMap fields while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") Do Not Disable Disabling the syncing of this resource could cause the vCluster to not work properly. ## `configMaps` object [​](#configMaps "Direct link to configMaps")ConfigMaps defines if config maps created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean true [​](#configMaps-enabled "Direct link to configMaps-enabled")Enabled defines if this option should be enabled. ### `all` boolean false [​](#configMaps-all "Direct link to configMaps-all")All defines if all resources of that type should get synced or only the necessary ones that are needed. ### `patches` object\[] [​](#configMaps-patches "Direct link to configMaps-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#configMaps-patches-path "Direct link to configMaps-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#configMaps-patches-expression "Direct link to configMaps-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#configMaps-patches-reverseExpression "Direct link to configMaps-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#configMaps-patches-reference "Direct link to configMaps-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#configMaps-patches-reference-apiVersion "Direct link to configMaps-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#configMaps-patches-reference-apiVersionPath "Direct link to configMaps-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#configMaps-patches-reference-kind "Direct link to configMaps-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#configMaps-patches-reference-kindPath "Direct link to configMaps-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#configMaps-patches-reference-namePath "Direct link to configMaps-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#configMaps-patches-reference-namespacePath "Direct link to configMaps-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#configMaps-patches-labels "Direct link to configMaps-patches-labels")Labels treats the path value as a labels selector. --- # Pods Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is enabled. Sync all [Pod](https://kubernetes.io/docs/concepts/workloads/pods/) resources, including ephemeral containers, from the tenant cluster to the control plane cluster. ### Automatically apply tolerations to all pods synced by vCluster[​](#automatically-apply-tolerations-to-all-pods-synced-by-vcluster "Direct link to Automatically apply tolerations to all pods synced by vCluster") Kubernetes has a concept of [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/), which is used for controlling scheduling. If you have a use case that requires vCluster to sync all pods and automatically set a toleration on those pods, then you can achieve this with the `enforceTolerations` option. You can pass multiple toleration expressions, and the syncer adds them to every new pod that vCluster syncs. ``` sync: toHost: pods: enabled: true enforceTolerations: - example-key=example-value:NoSchedule - another-key:PreferNoSchedule ``` info vCluster does not support setting the `tolerationSeconds` field of a toleration. If your use case requires this, open an issue in [the vCluster repo on GitHub](https://github.com/loft-sh/vcluster/issues). ### Replace container images when pods are synced[​](#replace-container-images-when-pods-are-synced "Direct link to Replace container images when pods are synced") If certain images used within the tenant cluster are not accessible in the control plane cluster due to registry restrictions or security policies, `translateImage` can map these to equivalent, permitted images in the control plane cluster's registry. ``` sync: toHost: pods: enabled: true translateImage: "virtualcluster/image:tag": "hostcluster/alternative-image:tag" ``` ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.pods.patches` to transform Pod fields, such as `spec.containers[*].name`, while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Use secrets for ServiceAccount tokens[​](#use-secrets-for-serviceaccount-tokens "Direct link to Use secrets for ServiceAccount tokens") A host Pod requires a ServiceAccount token to communicate with the tenant cluster's API. If you don't want to create these secrets in the control plane cluster, disable this option. vCluster then adds annotations to the pods. ## Rewrite hosts[​](#rewrite-hosts "Direct link to Rewrite hosts") Applies only to Pods that have the field `spec.subdomain` set. In such Pods, the fqdn hostname (`hostname -f`) is constructed based on the control plane cluster namespace the vCluster runs in. This is usually not what applications expect as they are unaware of the control plane cluster. If this option is enabled, vCluster injects an initContainer to override the Pod's `/etc/hosts` file to change the fqdn hostname to match the expected domain. Configuration change In vCluster version `0.27.0`, the `rewriteHosts.initContainer.image` configuration changed from a string to an object format. This change provides granular control over the registry, repository, and tag of the image. For more information on the new object structure, see the [configuration reference](#pods-rewriteHosts-initContainer). ## Config reference[​](#config-reference "Direct link to Config reference") Do Not Disable Disabling the syncing of this resource could cause the vCluster to not work properly. ## `pods` object [​](#pods "Direct link to pods")Pods defines if pods created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean true [​](#pods-enabled "Direct link to pods-enabled")Enabled defines if pod syncing should be enabled. ### `translateImage` object {} [​](#pods-translateImage "Direct link to pods-translateImage")TranslateImage maps an image to another image that should be used instead. For example this can be used to rewrite a certain image that is used within the virtual cluster to be another image on the host cluster ### `enforceTolerations` string\[] \[] [​](#pods-enforceTolerations "Direct link to pods-enforceTolerations")EnforceTolerations will add the specified tolerations to all pods synced by the virtual cluster. ### `useSecretsForSATokens` boolean false [​](#pods-useSecretsForSATokens "Direct link to pods-useSecretsForSATokens")UseSecretsForSATokens will use secrets to save the generated service account tokens by virtual cluster instead of using a pod annotation. ### `runtimeClassName` string [​](#pods-runtimeClassName "Direct link to pods-runtimeClassName")RuntimeClassName is the runtime class to set for synced pods. ### `priorityClassName` string [​](#pods-priorityClassName "Direct link to pods-priorityClassName")PriorityClassName is the priority class to set for synced pods. ### `rewriteHosts` object [​](#pods-rewriteHosts "Direct link to pods-rewriteHosts")RewriteHosts is a special option needed to rewrite statefulset containers to allow the correct FQDN. virtual cluster will add a small container to each stateful set pod that will initially rewrite the /etc/hosts file to match the FQDN expected by the virtual cluster. #### `enabled` boolean true [​](#pods-rewriteHosts-enabled "Direct link to pods-rewriteHosts-enabled")Enabled specifies if rewriting stateful set pods should be enabled. #### `initContainer` object [​](#pods-rewriteHosts-initContainer "Direct link to pods-rewriteHosts-initContainer")InitContainer holds extra options for the init container used by vCluster to rewrite the FQDN for stateful set pods. ##### `image` object [​](#pods-rewriteHosts-initContainer-image "Direct link to pods-rewriteHosts-initContainer-image")Image is the image virtual cluster should use to rewrite this FQDN. ##### `registry` string mirror.gcr.io [​](#pods-rewriteHosts-initContainer-image-registry "Direct link to pods-rewriteHosts-initContainer-image-registry")Registry is the registry of the container image, e.g. my-registry.com or ghcr.io. This setting can be globally overridden via the controlPlane.advanced.defaultImageRegistry option. Empty means docker hub. ##### `repository` string library/alpine [​](#pods-rewriteHosts-initContainer-image-repository "Direct link to pods-rewriteHosts-initContainer-image-repository")Repository is the repository of the container image, e.g. my-repo/my-image ##### `tag` string 3.20 [​](#pods-rewriteHosts-initContainer-image-tag "Direct link to pods-rewriteHosts-initContainer-image-tag")Tag is the tag of the container image, and is the default version. ##### `resources` object [​](#pods-rewriteHosts-initContainer-resources "Direct link to pods-rewriteHosts-initContainer-resources")Resources are the resources that should be assigned to the init container for each stateful set init container. ##### `limits` object map\[cpu:30m memory:64Mi] [​](#pods-rewriteHosts-initContainer-resources-limits "Direct link to pods-rewriteHosts-initContainer-resources-limits")Limits are resource limits for the container ##### `requests` object map\[cpu:30m memory:64Mi] [​](#pods-rewriteHosts-initContainer-resources-requests "Direct link to pods-rewriteHosts-initContainer-resources-requests")Requests are minimal resources that will be consumed by the container ### `patches` object\[] [​](#pods-patches "Direct link to pods-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#pods-patches-path "Direct link to pods-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#pods-patches-expression "Direct link to pods-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#pods-patches-reverseExpression "Direct link to pods-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#pods-patches-reference "Direct link to pods-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#pods-patches-reference-apiVersion "Direct link to pods-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#pods-patches-reference-apiVersionPath "Direct link to pods-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#pods-patches-reference-kind "Direct link to pods-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#pods-patches-reference-kindPath "Direct link to pods-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#pods-patches-reference-namePath "Direct link to pods-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#pods-patches-reference-namespacePath "Direct link to pods-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#pods-patches-labels "Direct link to pods-patches-labels")Labels treats the path value as a labels selector. ### `hybridScheduling` object [​](#pods-hybridScheduling "Direct link to pods-hybridScheduling")HybridScheduling is used to enable and configure hybrid scheduling for pods in the virtual cluster. #### `enabled` boolean false [​](#pods-hybridScheduling-enabled "Direct link to pods-hybridScheduling-enabled")Enabled specifies if hybrid scheduling is enabled. #### `hostSchedulers` string\[] \[] [​](#pods-hybridScheduling-hostSchedulers "Direct link to pods-hybridScheduling-hostSchedulers")HostSchedulers is a list of schedulers that are deployed on the host cluster. --- # Hybrid scheduling | | | Enterprise | | | | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Hybrid Scheduling](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/pods/hybrid-scheduling.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. Hybrid scheduling enables tenant clusters to use multiple schedulers simultaneously - both from the control plane cluster and tenant cluster. Pods within the tenant cluster can use different schedulers based on their individual configuration. You can enable hybrid scheduling using the following: ``` sync: toHost: pods: hybridScheduling: enabled: true hostSchedulers: - my-custom-scheduler fromHost: nodes: enabled: true ``` You must also enable syncing of real nodes from the control plane cluster to the tenant cluster. If you don’t, the tenant cluster displays an error. By default, all nodes from the control plane cluster are synced to the tenant cluster. To sync only specific nodes, you can define a label selector as shown in the following example: ``` sync: toHost: pods: hybridScheduling: enabled: true hostSchedulers: - my-custom-scheduler fromHost: nodes: enabled: true selector: labels: environment: production team: backend ``` Example Checkout [this guide](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/use-custom-schedulers.md) that shows how to deploy custom host and virtual schedulers and use them with Hybrid scheduling. ## Use host schedulers[​](#use-host-schedulers "Direct link to Use host schedulers") `sync.toHost.pods.hybridScheduling.hostSchedulers` specifies a list of schedulers available on the control plane cluster that tenant cluster pods can use. The default Kubernetes scheduler on the control plane cluster is always implicitly included in the `sync.toHost.pods.hybridScheduling.hostSchedulers` list. A default scheduler can be used in two ways: by omitting `.spec.schedulerName` or by explicitly setting it to `default-scheduler`. In both cases, the default Kubernetes scheduler running on the control plane cluster schedules the pod. ### Use custom host schedulers[​](#use-custom-host-schedulers "Direct link to Use custom host schedulers") The following hybrid scheduling configuration allows tenant cluster pods to use the control plane cluster's custom schedulers my-custom-scheduler and my-gpu-scheduler. note This example omits the full vCluster configuration to highlight the relevant hybrid scheduling settings. ``` sync: toHost: pods: hybridScheduling: enabled: true hostSchedulers: - my-custom-scheduler - my-gpu-scheduler ``` You can now deploy pods in the tenant cluster that use the control plane cluster’s custom schedulers `my-custom-scheduler` and `my-gpu-scheduler`. For example: ``` apiVersion: v1 kind: Pod metadata: name: my-example spec: schedulerName: my-custom-scheduler containers: - name: pause image: registry.k8s.io/pause:3.10 --- apiVersion: v1 kind: Pod metadata: name: my-gpu-example spec: schedulerName: my-gpu-scheduler containers: - name: pause image: registry.k8s.io/pause:3.10 ``` ## Use custom virtual schedulers[​](#use-custom-virtual-schedulers "Direct link to Use custom virtual schedulers") Tenant cluster pods can use a custom scheduler that runs inside the tenant cluster. To ensure these pods are scheduled correctly, **do not** include virtual schedulers in the `sync.toHost.pods.hybridScheduling.hostSchedulers` list in your vCluster configuration. The following example shows the hybrid scheduling configuration and a pod that uses the `my-virtual-ai-scheduler` running in the tenant cluster. **Hybrid scheduling configuration**: ``` sync: toHost: pods: hybridScheduling: enabled: true ``` **Pod**: ``` apiVersion: v1 kind: Pod metadata: name: my-ai-example spec: schedulerName: my-virtual-ai-scheduler containers: - name: pause image: registry.k8s.io/pause:3.10 ``` ## Other related configurations and cluster permissions[​](#other-related-configurations-and-cluster-permissions "Direct link to Other related configurations and cluster permissions") When hybrid scheduling is enabled and PersistentVolumeClaims are synced from the tenant cluster to the control plane cluster, additional resources must also be present in the tenant cluster for scheduling to function correctly. By default, vCluster automatically syncs the following resources from the control plane cluster to the tenant cluster, unless this behavior is explicitly disabled in the configuration: * `CSINodes`, * `CSIStorageCapacities` * `CSIDrivers` When `PersistentVolumeClaims` are synced to the control plane cluster, vCluster also enables syncing of `StorageClasses` from the control plane cluster to the tenant cluster. However, this only happens if syncing `StorageClasses` from the tenant cluster to the control plane cluster is disabled in the configuration. warning Manually disabling `CSINodes`, `CSIStorageCapacities`, `CSIDrivers` and `StorageClasses` sync in the above cases can cause pod scheduling to fail in your tenant cluster. vCluster automatically adds the required `ClusterRole` rules to ensure syncing works correctly for automatically enabled `CSINodes`, `CSIStorageCapacities`, `CSIDrivers`, and `StorageClasses`. --- # Secrets Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is enabled. Sync [Secret](https://kubernetes.io/docs/concepts/configuration/secret/) resources used by pods from the tenant cluster to the control plane cluster. Apps frequently need secret data to function. ### Sync only utilized secrets from tenant to control plane cluster (Default)[​](#sync-only-utilized-secrets-from-tenant-to-control-plane-cluster-default "Direct link to Sync only utilized secrets from tenant to control plane cluster (Default)") ``` sync: toHost: secrets: enabled: true ``` ### Sync all secrets from tenant to control plane cluster[​](#sync-all-secrets-from-tenant-to-control-plane-cluster "Direct link to Sync all secrets from tenant to control plane cluster") vCluster only knows about a couple of tenant cluster resources that actually use secrets and tries to sync only those into the control plane cluster. You can enable syncing all tenant cluster secrets to avoid the problem that needed secrets are not synced to the control plane cluster. ``` sync: toHost: secrets: enabled: true all: true ``` ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.secrets.patches` to transform Secret fields while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") Do Not Disable Disabling the syncing of this resource could cause the vCluster to not work properly. ## `secrets` object [​](#secrets "Direct link to secrets")Secrets defines if secrets created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean true [​](#secrets-enabled "Direct link to secrets-enabled")Enabled defines if this option should be enabled. ### `all` boolean false [​](#secrets-all "Direct link to secrets-all")All defines if all resources of that type should get synced or only the necessary ones that are needed. ### `patches` object\[] [​](#secrets-patches "Direct link to secrets-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#secrets-patches-path "Direct link to secrets-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#secrets-patches-expression "Direct link to secrets-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#secrets-patches-reverseExpression "Direct link to secrets-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#secrets-patches-reference "Direct link to secrets-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#secrets-patches-reference-apiVersion "Direct link to secrets-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#secrets-patches-reference-apiVersionPath "Direct link to secrets-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#secrets-patches-reference-kind "Direct link to secrets-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#secrets-patches-reference-kindPath "Direct link to secrets-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#secrets-patches-reference-namePath "Direct link to secrets-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#secrets-patches-reference-namespacePath "Direct link to secrets-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#secrets-patches-labels "Direct link to secrets-patches-labels")Labels treats the path value as a labels selector. --- # ResourceClaims | | | Enterprise | | | | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [DRA Sync](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md#dynamic-resource-allocation-and-device-classes) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. When enabled, sync all [ResourceClaims](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#resourceclaims-templates) resources from the tenant cluster to the control plane cluster. Use this option to create resource claims that can be referenced in workloads created in the vCluster. When enabled, vCluster automatically tries to detect the presence of api-resource `resource.k8s.io/v1` on the control plane cluster and fails to start up if the api-resource is missing. Referenced Device Class The resource claims must reference **[device classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/device-classes.md)** synced from the host. ### Sync ResourceClaims from the tenant to control plane cluster[​](#sync-resourceclaims-from-the-tenant-to-control-plane-cluster "Direct link to Sync ResourceClaims from the tenant to control plane cluster") ``` sync: toHost: resourceClaims: enabled: true ``` ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.resourceClaims.patches` to transform ResourceClaim fields while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") ## `resourceClaims` object [​](#resourceClaims "Direct link to resourceClaims")ResourceClaim defines if resource claims created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean false [​](#resourceClaims-enabled "Direct link to resourceClaims-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#resourceClaims-patches "Direct link to resourceClaims-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#resourceClaims-patches-path "Direct link to resourceClaims-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#resourceClaims-patches-expression "Direct link to resourceClaims-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#resourceClaims-patches-reverseExpression "Direct link to resourceClaims-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#resourceClaims-patches-reference "Direct link to resourceClaims-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#resourceClaims-patches-reference-apiVersion "Direct link to resourceClaims-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#resourceClaims-patches-reference-apiVersionPath "Direct link to resourceClaims-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#resourceClaims-patches-reference-kind "Direct link to resourceClaims-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#resourceClaims-patches-reference-kindPath "Direct link to resourceClaims-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#resourceClaims-patches-reference-namePath "Direct link to resourceClaims-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#resourceClaims-patches-reference-namespacePath "Direct link to resourceClaims-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#resourceClaims-patches-labels "Direct link to resourceClaims-patches-labels")Labels treats the path value as a labels selector. --- # ResourceClaimTemplates | | | Enterprise | | | | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [DRA Sync](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md#dynamic-resource-allocation-and-device-classes) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. When enabled, sync all [ResourceClaimTemplates](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#resourceclaims-templates) resources from the tenant cluster to the control plane cluster. Use this option to create resource claim templates that can be referenced in workloads created in the vCluster. When enabled, vCluster automatically tries to detect the presence of api-resource `resource.k8s.io/v1` on the control plane cluster and fails to start up if the api-resource is missing. Referenced Device Class The resource claim templates must reference **[device classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/device-classes.md)** synced from the host. ### Sync ResourceClaimTemplates from the tenant to control plane cluster[​](#sync-resourceclaimtemplates-from-the-tenant-to-control-plane-cluster "Direct link to Sync ResourceClaimTemplates from the tenant to control plane cluster") ``` sync: toHost: resourceClaimTemplates: enabled: true ``` ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.resourceClaimTemplates.patches` to transform ResourceClaimTemplate fields while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") ## `resourceClaimTemplates` object [​](#resourceClaimTemplates "Direct link to resourceClaimTemplates")ResourceClaimTemplates defines if resourceClaimTemplates created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean false [​](#resourceClaimTemplates-enabled "Direct link to resourceClaimTemplates-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#resourceClaimTemplates-patches "Direct link to resourceClaimTemplates-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#resourceClaimTemplates-patches-path "Direct link to resourceClaimTemplates-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#resourceClaimTemplates-patches-expression "Direct link to resourceClaimTemplates-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#resourceClaimTemplates-patches-reverseExpression "Direct link to resourceClaimTemplates-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#resourceClaimTemplates-patches-reference "Direct link to resourceClaimTemplates-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#resourceClaimTemplates-patches-reference-apiVersion "Direct link to resourceClaimTemplates-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#resourceClaimTemplates-patches-reference-apiVersionPath "Direct link to resourceClaimTemplates-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#resourceClaimTemplates-patches-reference-kind "Direct link to resourceClaimTemplates-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#resourceClaimTemplates-patches-reference-kindPath "Direct link to resourceClaimTemplates-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#resourceClaimTemplates-patches-reference-namePath "Direct link to resourceClaimTemplates-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#resourceClaimTemplates-patches-reference-namespacePath "Direct link to resourceClaimTemplates-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#resourceClaimTemplates-patches-labels "Direct link to resourceClaimTemplates-patches-labels")Labels treats the path value as a labels selector. --- # Endpoints Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is enabled. Sync all [Endpoint](https://kubernetes.io/docs/concepts/services-networking/service/#endpoints) resources from the tenant cluster to the control plane cluster. Service resources are also enabled by default and require Endpoint resources. ### Disable syncing Endpoints from the tenant to control plane cluster[​](#disable-syncing-endpoints-from-the-tenant-to-control-plane-cluster "Direct link to Disable syncing Endpoints from the tenant to control plane cluster") ``` sync: toHost: endpoints: enabled: false ``` ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.endpoints.patches` to transform Endpoints fields while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") ## `endpoints` object [​](#endpoints "Direct link to endpoints")Endpoints defines if endpoints created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean true [​](#endpoints-enabled "Direct link to endpoints-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#endpoints-patches "Direct link to endpoints-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#endpoints-patches-path "Direct link to endpoints-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#endpoints-patches-expression "Direct link to endpoints-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#endpoints-patches-reverseExpression "Direct link to endpoints-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#endpoints-patches-reference "Direct link to endpoints-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#endpoints-patches-reference-apiVersion "Direct link to endpoints-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#endpoints-patches-reference-apiVersionPath "Direct link to endpoints-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#endpoints-patches-reference-kind "Direct link to endpoints-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#endpoints-patches-reference-kindPath "Direct link to endpoints-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#endpoints-patches-reference-namePath "Direct link to endpoints-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#endpoints-patches-reference-namespacePath "Direct link to endpoints-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#endpoints-patches-labels "Direct link to endpoints-patches-labels")Labels treats the path value as a labels selector. --- # EndpointSlices Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is enabled. Sync all [EndpointSlices](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/) resources from the tenant cluster to the control plane cluster. Service resources are also enabled by default and require EndpointSlice resources. EndpointSlices provide a more scalable alternative to Endpoints for tracking network endpoints in Kubernetes clusters with many services or endpoints. ### Disable syncing EndpointSlices from the tenant to control plane cluster[​](#disable-syncing-endpointslices-from-the-tenant-to-control-plane-cluster "Direct link to Disable syncing EndpointSlices from the tenant to control plane cluster") ``` sync: toHost: endpointSlices: enabled: false ``` ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.endpointSlices.patches` to transform EndpointSlice fields while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") ## `endpointSlices` object [​](#endpointSlices "Direct link to endpointSlices")EndpointSlices defines if endpointslices created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean true [​](#endpointSlices-enabled "Direct link to endpointSlices-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#endpointSlices-patches "Direct link to endpointSlices-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#endpointSlices-patches-path "Direct link to endpointSlices-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#endpointSlices-patches-expression "Direct link to endpointSlices-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#endpointSlices-patches-reverseExpression "Direct link to endpointSlices-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#endpointSlices-patches-reference "Direct link to endpointSlices-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#endpointSlices-patches-reference-apiVersion "Direct link to endpointSlices-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#endpointSlices-patches-reference-apiVersionPath "Direct link to endpointSlices-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#endpointSlices-patches-reference-kind "Direct link to endpointSlices-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#endpointSlices-patches-reference-kindPath "Direct link to endpointSlices-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#endpointSlices-patches-reference-namePath "Direct link to endpointSlices-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#endpointSlices-patches-reference-namespacePath "Direct link to endpointSlices-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#endpointSlices-patches-labels "Direct link to endpointSlices-patches-labels")Labels treats the path value as a labels selector. --- # Gateway API Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, Gateway API sync is disabled. vCluster can sync the following [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/) resources from the tenant cluster to the control plane cluster: * `HTTPRoute` resources with `sync.toHost.gatewayApi.httpRoutes` * `TLSRoute` resources with `sync.toHost.gatewayApi.tlsRoutes` * `ReferenceGrant` resources with `sync.toHost.gatewayApi.referenceGrants` * `BackendTLSPolicy` resources with `sync.toHost.gatewayApi.backendTLSPolicies` vCluster can also import selected control plane `Gateway` resources into the tenant cluster with `sync.fromHost.gateways`, so tenants can attach routes to platform-managed infrastructure without creating Gateways directly. `sync.toHost.gatewayApi.gateways` exists for tenant-created Gateway sync, but the examples on this page use imported, control plane-owned Gateways. `GRPCRoute`, `TCPRoute`, and `UDPRoute` are not synced. vCluster also enforces Gateway API reference authorization before sync. Cross-namespace `allowedRoutes`, `ReferenceGrant`, and unsupported `parentRef`/`backendRef` group+kind combinations are validated in the tenant cluster, and rejected resources surface a `RefNotPermitted` or `SyncError` event on the tenant object. Prerequisites Before enabling Gateway API sync, review the [Gateway API prerequisites](https://vcluster.com/docs/vcluster/key-features/gateway-api.md#prerequisites). To import a control plane `GatewayClass` into the tenant cluster, also enable [`sync.fromHost.gatewayClasses`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/gateway-classes.md). Version requirement Gateway API sync requires a vCluster chart version that includes the resource-specific `sync.toHost.gatewayApi` fields and the `sync.fromHost.gatewayClasses` or `sync.fromHost.gateways` configuration fields. If Helm rejects any field as an additional property, upgrade vCluster before using these examples. Replaces the custom-resource workaround For native Gateway API sync, use this configuration instead of [custom-resource syncing](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/custom-resources.md). The custom-resource path is now only needed for Gateway API extensions outside the supported set (for example, Istio waypoint Gateways using the `HBONE` listener). ## Enable Gateway API sync[​](#enable-gateway-api-sync "Direct link to Enable Gateway API sync") Enable sync for tenant-created route and policy resources. Do not enable `sync.toHost.gatewayApi.gateways` for these examples: the platform-owned `Gateway` resources are created in the control plane cluster and imported into the tenant cluster with `sync.fromHost.gateways`. vcluster.yaml ``` sync: toHost: gatewayApi: httpRoutes: enabled: true tlsRoutes: enabled: true backendTLSPolicies: enabled: true referenceGrants: enabled: auto fromHost: gatewayClasses: enabled: true selector: matchLabels: gateway-demo.loft.sh/sync: "yes" gateways: enabled: true selector: matchLabels: gateway-demo.loft.sh/sync: "yes" mappings: byName: "gw-demo-01/edge": "gw-demo-01/edge" "gw-demo-02/edge": "gw-demo-02/edge" "gw-demo-03-edge/shared": "gw-demo-03-edge/shared" "gw-demo-05-routes/edge": "gw-demo-05-routes/edge" "gw-demo-07-edge/edge": "gw-demo-07-edge/edge" "gw-demo-09/edge": "gw-demo-09/edge" allowedRoutes: overrides: - hostNamespace: gw-demo-01 name: edge virtualNamespacePolicy: from: Same - hostNamespace: gw-demo-02 name: edge virtualNamespacePolicy: from: Same - hostNamespace: gw-demo-03-edge name: shared virtualNamespacePolicy: from: Selector selector: matchLabels: gateway-demo.loft.sh/route-access: demo-03 - hostNamespace: gw-demo-05-routes name: edge virtualNamespacePolicy: from: Same - hostNamespace: gw-demo-07-edge name: edge virtualNamespacePolicy: from: Same - hostNamespace: gw-demo-09 name: edge virtualNamespacePolicy: from: Same ``` The examples on this page use `GatewayClass` names such as `gw-demo-01` and `gw-demo-02`, and create platform-managed Gateways in matching namespaces in the control plane cluster. Create those control plane namespaces and `GatewayClass` resources before applying the example Gateways, or change each `gatewayClassName` to an existing imported `GatewayClass`. Replace `GATEWAY_CONTROLLER_NAME` with the `controllerName` for your Gateway API controller. Modify the following with your specific values to generate a copyable command: HOST\_CONTEXT$(kubectl config current-context) GATEWAY\_CONTROLLER\_NAMEgateway.envoyproxy.io/gatewayclass-controller GATEWAY\_NAMESPACESgw-demo-01 gw-demo-02 gw-demo-03-edge gw-demo-05-routes gw-demo-07-edge gw-demo-07-certs gw-demo-09 GATEWAY\_CLASSESgw-demo-01 gw-demo-02 gw-demo-03 gw-demo-05 gw-demo-07 gw-demo-09 Create the example GatewayClasses and namespaces in the control plane cluster ``` export HOST_CONTEXT="$(kubectl config current-context)" export GATEWAY_CONTROLLER_NAME="gateway.envoyproxy.io/gatewayclass-controller" for GATEWAY_NAMESPACE in gw-demo-01 gw-demo-02 gw-demo-03-edge gw-demo-05-routes gw-demo-07-edge gw-demo-07-certs gw-demo-09; do kubectl --context "$HOST_CONTEXT" create namespace "$GATEWAY_NAMESPACE" --dry-run=client -o yaml | kubectl --context "$HOST_CONTEXT" apply -f - done for GATEWAY_CLASS in gw-demo-01 gw-demo-02 gw-demo-03 gw-demo-05 gw-demo-07 gw-demo-09; do kubectl --context "$HOST_CONTEXT" apply -f - < -n ` | | Route references a Service in another namespace without a matching `ReferenceGrant` | `RefNotPermitted` | `kubectl describe httproute -n ` | | A tenant-owned route or policy references another namespace without a matching `ReferenceGrant` | `RefNotPermitted` | `kubectl describe httproute -n ` | | `BackendTLSPolicy` references a CA ConfigMap that has no synced control plane object | `SyncError` | `kubectl describe backendtlspolicy -n ` | | Route uses an unsupported `parentRef` or `backendRef` group+kind | `SyncError` | `kubectl describe httproute -n ` | For deeper recipes (including how to recover from each event), see [Gateway API sync troubleshooting](https://vcluster.com/docs/vcluster/troubleshoot/gateway-api-sync.md). ## Example: `BackendTLSPolicy`[​](#example-backendtlspolicy "Direct link to example-backendtlspolicy") `BackendTLSPolicy` tells the Gateway controller to originate TLS to a backend and validate it against a CA bundle. vCluster syncs the policy and translates the CA ConfigMap reference for the control plane cluster. The policy is only effective if the Gateway controller implements `BackendTLSPolicy`. Apply the namespace first: BackendTLSPolicy namespace ``` apiVersion: v1 kind: Namespace metadata: name: gw-demo-09 labels: gateway-demo.loft.sh/case: "09-backend-tls-policy" ``` Create a backend certificate and synced CA bundle: Create backend TLS material ``` openssl req -x509 -newkey rsa:2048 -nodes -days 365 \ -subj "/CN=backend.gw-demo.test" \ -addext "subjectAltName=DNS:backend.gw-demo.test" \ -keyout backend.key \ -out backend.crt kubectl -n gw-demo-09 create secret tls backend-cert \ --cert=backend.crt \ --key=backend.key kubectl -n gw-demo-09 annotate secret backend-cert vcluster.loft.sh/force-sync=true kubectl -n gw-demo-09 create configmap backend-ca --from-file=ca.crt=backend.crt kubectl -n gw-demo-09 annotate configmap backend-ca vcluster.loft.sh/force-sync=true ``` Then create the TLS-speaking backend Service: TLS backend resources ``` apiVersion: v1 kind: ConfigMap metadata: name: backend-caddy-config namespace: gw-demo-09 data: Caddyfile: | { auto_https off } :8443 { tls /etc/certs/tls.crt /etc/certs/tls.key respond "gateway demo 09 backend tls policy\n" } --- apiVersion: apps/v1 kind: Deployment metadata: name: secure-backend namespace: gw-demo-09 spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: secure-backend template: metadata: labels: app.kubernetes.io/name: secure-backend spec: containers: - name: caddy image: caddy:2.8-alpine ports: - name: https containerPort: 8443 volumeMounts: - name: caddy-config mountPath: /etc/caddy readOnly: true - name: backend-cert mountPath: /etc/certs readOnly: true volumes: - name: caddy-config configMap: name: backend-caddy-config - name: backend-cert secret: secretName: backend-cert --- apiVersion: v1 kind: Service metadata: name: secure-backend namespace: gw-demo-09 spec: selector: app.kubernetes.io/name: secure-backend ports: - name: https port: 443 targetPort: https appProtocol: https ``` Create the platform-managed Gateway in the control plane cluster. vCluster imports it into the tenant cluster as `gw-demo-09/edge`. HTTP Gateway in the control plane cluster ``` apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: edge namespace: gw-demo-09 labels: gateway-demo.loft.sh/sync: "yes" spec: gatewayClassName: gw-demo-09 listeners: - name: http protocol: HTTP port: 80 hostname: backend-tls.gw-demo.test allowedRoutes: kinds: - group: gateway.networking.k8s.io kind: HTTPRoute namespaces: from: Same ``` Then create the route and `BackendTLSPolicy` in the tenant cluster: HTTPRoute with BackendTLSPolicy in the tenant cluster ``` apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: backend-tls namespace: gw-demo-09 spec: parentRefs: - name: edge sectionName: http hostnames: - backend-tls.gw-demo.test rules: - backendRefs: - name: secure-backend port: 443 --- apiVersion: gateway.networking.k8s.io/v1 kind: BackendTLSPolicy metadata: name: secure-backend namespace: gw-demo-09 spec: targetRefs: - group: "" kind: Service name: secure-backend validation: hostname: backend.gw-demo.test caCertificateRefs: - group: "" kind: ConfigMap name: backend-ca ``` A successful policy reports an `ancestor` condition with `Accepted=True` against the route's parent `Gateway`. If the referenced CA ConfigMap is not synced to the control plane cluster, vCluster emits a `SyncError` event instead. See [`SyncError: referenced ConfigMap has no synced control plane object`](https://vcluster.com/docs/vcluster/troubleshoot/gateway-api-sync.md#syncerror-missing-ca-configmap). ## CLI guidance[​](#cli-guidance "Direct link to CLI guidance") ### Connect to the tenant cluster[​](#connect-to-the-tenant-cluster "Direct link to Connect to the tenant cluster") Tenant-owned Gateway API resources such as routes and policies live in the tenant cluster. Imported `Gateway` resources also appear in the tenant cluster, but their source of truth is the control plane cluster. Connect to the tenant with the vCluster CLI before issuing tenant-side `kubectl` commands: Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster VCLUSTER\_NAMESPACEmy-vcluster-host-ns Open a tenant kubeconfig context ``` vcluster connect my-vcluster --namespace my-vcluster-host-ns ``` `vcluster connect` writes a context entry to your kubeconfig and switches the current context. To run side-by-side checks against the control plane cluster, export both contexts explicitly: Modify the following with your specific values to generate a copyable command: TENANT\_CONTEXT$(kubectl config current-context) HOST\_CONTEXTyour-control-plane-kube-context Use both control plane and tenant contexts ``` export TENANT_CONTEXT="$(kubectl config current-context)" export HOST_CONTEXT="your-control-plane-kube-context" ``` There is no separate `vcluster` verb for Gateway API resources. Inspect imported Gateways and tenant-created routes with `kubectl` against the tenant context; the control plane data plane is reached using the address from `status.addresses` on the imported tenant `Gateway`. ### Inspect Gateway API resources[​](#inspect-gateway-api-resources "Direct link to Inspect Gateway API resources") List Gateway API resources across all tenant namespaces ``` kubectl --context "$TENANT_CONTEXT" get gateway,httproute,tlsroute,backendtlspolicy -A ``` Modify the following with your specific values to generate a copyable command: NAMESPACEgw-demo-01 GATEWAY\_NAMEedge HTTPROUTE\_NAMEbasic Show Gateway and route status ``` kubectl --context "$TENANT_CONTEXT" -n gw-demo-01 get gateway edge -o yaml kubectl --context "$TENANT_CONTEXT" -n gw-demo-01 describe httproute basic ``` Modify the following with your specific values to generate a copyable command: NAMESPACEgw-demo-01 See vCluster-emitted sync warnings ``` kubectl --context "$TENANT_CONTEXT" -n gw-demo-01 get events --field-selector type=Warning ``` `ReferenceGrant` has no status, so confirm it by checking that the dependent route, Gateway, or `BackendTLSPolicy` reaches `Accepted=True` and `ResolvedRefs=True`. ### Discover the control plane-assigned address[​](#discover-the-control-plane-assigned-address "Direct link to Discover the control plane-assigned address") Modify the following with your specific values to generate a copyable command: NAMESPACEgw-demo-01 GATEWAY\_NAMEedge Get the control plane address mirrored to the tenant Gateway ``` kubectl --context "$TENANT_CONTEXT" -n gw-demo-01 get gateway edge \ -o jsonpath='{.status.addresses[0].value}' ``` The address is whatever the Gateway controller assigned (LoadBalancer IP, NodePort, or controller-published hostname). Tenant clients curl that address directly; vCluster does not proxy data-plane traffic. ## Patches[​](#patches "Direct link to Patches") Patch lists are resource-specific. For these examples, use `sync.fromHost.gateways.patches` for imported control plane `Gateway` resources and `sync.toHost.gatewayApi.httpRoutes.patches`, `tlsRoutes.patches`, `backendTLSPolicies.patches`, or `referenceGrants.patches` for tenant-created resources. Each list accepts the same expression and reference patch shapes. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for full syntax and directionality. Patch imported Gateway hostnames during sync ``` sync: fromHost: gateways: enabled: true patches: - path: spec.listeners[*].hostname expression: '"tenant-" + value' reverseExpression: 'value.replace(/^tenant-/, "")' ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `gatewayApi` object [​](#gatewayApi "Direct link to gatewayApi")GatewayAPI defines Gateway API resources created within the tenant cluster that should get synced to the control plane cluster. Setting enabled: true turns on Gateway and HTTPRoute sync, imports control plane cluster GatewayClasses so tenant Gateways can resolve them, and serves tenant ReferenceGrants for validation; TLSRoutes and BackendTLSPolicies must be enabled individually. ### `enabled` boolean false [​](#gatewayApi-enabled "Direct link to gatewayApi-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#gatewayApi-patches "Direct link to gatewayApi-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#gatewayApi-patches-path "Direct link to gatewayApi-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#gatewayApi-patches-expression "Direct link to gatewayApi-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#gatewayApi-patches-reverseExpression "Direct link to gatewayApi-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#gatewayApi-patches-reference "Direct link to gatewayApi-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#gatewayApi-patches-reference-apiVersion "Direct link to gatewayApi-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#gatewayApi-patches-reference-apiVersionPath "Direct link to gatewayApi-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#gatewayApi-patches-reference-kind "Direct link to gatewayApi-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#gatewayApi-patches-reference-kindPath "Direct link to gatewayApi-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#gatewayApi-patches-reference-namePath "Direct link to gatewayApi-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#gatewayApi-patches-reference-namespacePath "Direct link to gatewayApi-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#gatewayApi-patches-labels "Direct link to gatewayApi-patches-labels")Labels treats the path value as a labels selector. ### `httpRoutes` object [​](#gatewayApi-httpRoutes "Direct link to gatewayApi-httpRoutes")HTTPRoutes configures HTTPRoute sync to the control plane cluster. #### `enabled` boolean false [​](#gatewayApi-httpRoutes-enabled "Direct link to gatewayApi-httpRoutes-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#gatewayApi-httpRoutes-patches "Direct link to gatewayApi-httpRoutes-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#gatewayApi-httpRoutes-patches-path "Direct link to gatewayApi-httpRoutes-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#gatewayApi-httpRoutes-patches-expression "Direct link to gatewayApi-httpRoutes-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#gatewayApi-httpRoutes-patches-reverseExpression "Direct link to gatewayApi-httpRoutes-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#gatewayApi-httpRoutes-patches-reference "Direct link to gatewayApi-httpRoutes-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#gatewayApi-httpRoutes-patches-reference-apiVersion "Direct link to gatewayApi-httpRoutes-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#gatewayApi-httpRoutes-patches-reference-apiVersionPath "Direct link to gatewayApi-httpRoutes-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#gatewayApi-httpRoutes-patches-reference-kind "Direct link to gatewayApi-httpRoutes-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#gatewayApi-httpRoutes-patches-reference-kindPath "Direct link to gatewayApi-httpRoutes-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#gatewayApi-httpRoutes-patches-reference-namePath "Direct link to gatewayApi-httpRoutes-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#gatewayApi-httpRoutes-patches-reference-namespacePath "Direct link to gatewayApi-httpRoutes-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#gatewayApi-httpRoutes-patches-labels "Direct link to gatewayApi-httpRoutes-patches-labels")Labels treats the path value as a labels selector. ### `gateways` object [​](#gatewayApi-gateways "Direct link to gatewayApi-gateways")Gateways configures tenant-created Gateway sync to the control plane cluster. #### `enabled` boolean false [​](#gatewayApi-gateways-enabled "Direct link to gatewayApi-gateways-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#gatewayApi-gateways-patches "Direct link to gatewayApi-gateways-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#gatewayApi-gateways-patches-path "Direct link to gatewayApi-gateways-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#gatewayApi-gateways-patches-expression "Direct link to gatewayApi-gateways-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#gatewayApi-gateways-patches-reverseExpression "Direct link to gatewayApi-gateways-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#gatewayApi-gateways-patches-reference "Direct link to gatewayApi-gateways-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#gatewayApi-gateways-patches-reference-apiVersion "Direct link to gatewayApi-gateways-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#gatewayApi-gateways-patches-reference-apiVersionPath "Direct link to gatewayApi-gateways-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#gatewayApi-gateways-patches-reference-kind "Direct link to gatewayApi-gateways-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#gatewayApi-gateways-patches-reference-kindPath "Direct link to gatewayApi-gateways-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#gatewayApi-gateways-patches-reference-namePath "Direct link to gatewayApi-gateways-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#gatewayApi-gateways-patches-reference-namespacePath "Direct link to gatewayApi-gateways-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#gatewayApi-gateways-patches-labels "Direct link to gatewayApi-gateways-patches-labels")Labels treats the path value as a labels selector. ### `tlsRoutes` object [​](#gatewayApi-tlsRoutes "Direct link to gatewayApi-tlsRoutes")TLSRoutes configures TLSRoute sync to the control plane cluster. #### `enabled` boolean false [​](#gatewayApi-tlsRoutes-enabled "Direct link to gatewayApi-tlsRoutes-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#gatewayApi-tlsRoutes-patches "Direct link to gatewayApi-tlsRoutes-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#gatewayApi-tlsRoutes-patches-path "Direct link to gatewayApi-tlsRoutes-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#gatewayApi-tlsRoutes-patches-expression "Direct link to gatewayApi-tlsRoutes-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#gatewayApi-tlsRoutes-patches-reverseExpression "Direct link to gatewayApi-tlsRoutes-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#gatewayApi-tlsRoutes-patches-reference "Direct link to gatewayApi-tlsRoutes-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#gatewayApi-tlsRoutes-patches-reference-apiVersion "Direct link to gatewayApi-tlsRoutes-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#gatewayApi-tlsRoutes-patches-reference-apiVersionPath "Direct link to gatewayApi-tlsRoutes-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#gatewayApi-tlsRoutes-patches-reference-kind "Direct link to gatewayApi-tlsRoutes-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#gatewayApi-tlsRoutes-patches-reference-kindPath "Direct link to gatewayApi-tlsRoutes-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#gatewayApi-tlsRoutes-patches-reference-namePath "Direct link to gatewayApi-tlsRoutes-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#gatewayApi-tlsRoutes-patches-reference-namespacePath "Direct link to gatewayApi-tlsRoutes-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#gatewayApi-tlsRoutes-patches-labels "Direct link to gatewayApi-tlsRoutes-patches-labels")Labels treats the path value as a labels selector. ### `backendTLSPolicies` object [​](#gatewayApi-backendTLSPolicies "Direct link to gatewayApi-backendTLSPolicies")BackendTLSPolicies configures BackendTLSPolicy sync to the control plane cluster. #### `enabled` boolean false [​](#gatewayApi-backendTLSPolicies-enabled "Direct link to gatewayApi-backendTLSPolicies-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#gatewayApi-backendTLSPolicies-patches "Direct link to gatewayApi-backendTLSPolicies-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#gatewayApi-backendTLSPolicies-patches-path "Direct link to gatewayApi-backendTLSPolicies-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#gatewayApi-backendTLSPolicies-patches-expression "Direct link to gatewayApi-backendTLSPolicies-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#gatewayApi-backendTLSPolicies-patches-reverseExpression "Direct link to gatewayApi-backendTLSPolicies-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#gatewayApi-backendTLSPolicies-patches-reference "Direct link to gatewayApi-backendTLSPolicies-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#gatewayApi-backendTLSPolicies-patches-reference-apiVersion "Direct link to gatewayApi-backendTLSPolicies-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#gatewayApi-backendTLSPolicies-patches-reference-apiVersionPath "Direct link to gatewayApi-backendTLSPolicies-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#gatewayApi-backendTLSPolicies-patches-reference-kind "Direct link to gatewayApi-backendTLSPolicies-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#gatewayApi-backendTLSPolicies-patches-reference-kindPath "Direct link to gatewayApi-backendTLSPolicies-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#gatewayApi-backendTLSPolicies-patches-reference-namePath "Direct link to gatewayApi-backendTLSPolicies-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#gatewayApi-backendTLSPolicies-patches-reference-namespacePath "Direct link to gatewayApi-backendTLSPolicies-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#gatewayApi-backendTLSPolicies-patches-labels "Direct link to gatewayApi-backendTLSPolicies-patches-labels")Labels treats the path value as a labels selector. ### `referenceGrants` object [​](#gatewayApi-referenceGrants "Direct link to gatewayApi-referenceGrants")ReferenceGrants configures ReferenceGrant sync to the control plane cluster. Enabled may be "auto", "true", or "false". In auto mode grants follow route sync and are validated within the tenant cluster; they sync to the control plane cluster only when namespace sync is also enabled. #### `enabled` string|boolean auto [​](#gatewayApi-referenceGrants-enabled "Direct link to gatewayApi-referenceGrants-enabled")Enabled defines if this option should be enabled. #### `patches` object\[] [​](#gatewayApi-referenceGrants-patches "Direct link to gatewayApi-referenceGrants-patches")Patches patch the resource according to the provided specification. ##### `path` required string [​](#gatewayApi-referenceGrants-patches-path "Direct link to gatewayApi-referenceGrants-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. ##### `expression` string [​](#gatewayApi-referenceGrants-patches-expression "Direct link to gatewayApi-referenceGrants-patches-expression")Expression transforms the value according to the given JavaScript expression. ##### `reverseExpression` string [​](#gatewayApi-referenceGrants-patches-reverseExpression "Direct link to gatewayApi-referenceGrants-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. ##### `reference` object [​](#gatewayApi-referenceGrants-patches-reference "Direct link to gatewayApi-referenceGrants-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#gatewayApi-referenceGrants-patches-reference-apiVersion "Direct link to gatewayApi-referenceGrants-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#gatewayApi-referenceGrants-patches-reference-apiVersionPath "Direct link to gatewayApi-referenceGrants-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#gatewayApi-referenceGrants-patches-reference-kind "Direct link to gatewayApi-referenceGrants-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#gatewayApi-referenceGrants-patches-reference-kindPath "Direct link to gatewayApi-referenceGrants-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#gatewayApi-referenceGrants-patches-reference-namePath "Direct link to gatewayApi-referenceGrants-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#gatewayApi-referenceGrants-patches-reference-namespacePath "Direct link to gatewayApi-referenceGrants-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. ##### `labels` object [​](#gatewayApi-referenceGrants-patches-labels "Direct link to gatewayApi-referenceGrants-patches-labels")Labels treats the path value as a labels selector. --- # Ingresses Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. When enabled, sync all [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) resources from the tenant cluster to the control plane cluster. Use this option to make a tenant cluster service available using a hostname/domain without having to configure DNS for each tenant cluster. This requires a compatible Ingress controller running on the control plane cluster, and a single shared controller can reconcile Ingress resources synced from multiple tenant clusters. vCluster preserves a tenant's `spec.ingressClassName` unchanged, so a dedicated per-tenant IngressClass alone doesn't isolate tenants. A tenant can still name another tenant's class. Isolate tenants one of two ways: * **Namespace-scoped controller.** Run a controller instance per tenant that only watches that tenant's own host namespace. This works regardless of which class a tenant names. * **Enforced class selection.** Use [`sync.fromHost.ingressClasses`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/ingress-classes.md) with a selector, paired with control plane cluster admission. The selector alone only helps partially, since it skips the check when `spec.ingressClassName` is empty or when a tenant uses the legacy `kubernetes.io/ingress.class` annotation. Admission needs to cover both paths, plus any classless Ingress that would fall through to a default class. Hostname claims aren't scoped to a tenant vCluster syncs a tenant's `spec.rules[].host` unchanged. It doesn't rewrite or scope the hostname to that tenant. If tenant clusters share a control plane cluster ingress controller, a tenant can claim a hostname belonging to another tenant or to platform infrastructure. The controller's own conflict-resolution behavior decides which rule wins, not vCluster. See [Ingress and Gateway API hostname claims](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#ingress-and-gateway-api-hostname-claims) in the shared-node hardening guide. Prefer Gateway API for new workloads For tenant traffic served by a Gateway controller on the control plane cluster, [Gateway API sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md) is the modern alternative. It syncs `Gateway`, `HTTPRoute`, `TLSRoute`, `ReferenceGrant`, and `BackendTLSPolicy`, and lets tenants import host `GatewayClass` resources via [`sync.fromHost.gatewayClasses`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/gateway-classes.md). Ingress sync remains supported for workloads that still depend on `networking.k8s.io/v1` Ingress resources. When enabled, vCluster automatically tries to detect the supported ingress version (`networking.k8s.io/v1` or `networking.k8s.io/v1beta1`). ### Sync Ingresses from the tenant to control plane cluster[​](#sync-ingresses-from-the-tenant-to-control-plane-cluster "Direct link to Sync Ingresses from the tenant to control plane cluster") ``` sync: toHost: ingresses: enabled: true ``` ### Admission webhook rejects synced Ingress[​](#admission-webhook-rejects-synced-ingress "Direct link to Admission webhook rejects synced Ingress") During an ingress-nginx Helm upgrade on the control plane cluster, the chart applies the ValidatingWebhookConfiguration with no `caBundle` set. A post-upgrade job then patches the `caBundle` onto it using the webhook's certificate secret. If the upgrade is interrupted before that job completes, the webhook config is left with an empty `caBundle`. When vCluster then creates a synced Ingress on the control plane cluster, the webhook rejects it, returning `tls: failed to verify certificate: x509: certificate signed by unknown authority`. From the tenant's side, the Ingress exists but never appears on the control plane cluster. The syncer log shows the TLS rejection against the translated Ingress name. Two ways to work around it: * For debugging only, set `controller.admissionWebhooks.enabled: false` on the ingress-nginx chart. This disables admission validation. The controller still routes traffic. * For a lasting fix, set `controller.admissionWebhooks.certManager.enabled: true` so cert-manager manages the `caBundle` instead of the chart's Helm hook jobs. This requires cert-manager already installed in the cluster, and removes the race entirely. This is an ingress-nginx webhook lifecycle issue, not a vCluster sync defect, and one more reason to plan a move to [Gateway API sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md). ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.ingresses.patches` to transform Ingress fields, such as `spec.rules[*].host`, while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") ## `ingresses` object [​](#ingresses "Direct link to ingresses")Ingresses defines if ingresses created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean false [​](#ingresses-enabled "Direct link to ingresses-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#ingresses-patches "Direct link to ingresses-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#ingresses-patches-path "Direct link to ingresses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#ingresses-patches-expression "Direct link to ingresses-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#ingresses-patches-reverseExpression "Direct link to ingresses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#ingresses-patches-reference "Direct link to ingresses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#ingresses-patches-reference-apiVersion "Direct link to ingresses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#ingresses-patches-reference-apiVersionPath "Direct link to ingresses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#ingresses-patches-reference-kind "Direct link to ingresses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#ingresses-patches-reference-kindPath "Direct link to ingresses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#ingresses-patches-reference-namePath "Direct link to ingresses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#ingresses-patches-reference-namespacePath "Direct link to ingresses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#ingresses-patches-labels "Direct link to ingresses-patches-labels")Labels treats the path value as a labels selector. --- # Network policies Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. Sync [NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) resources from the tenant cluster to the control plane cluster. This ensures correct policies are created in the control plane cluster to achieve the desired traffic behavior between pods. Additive policies can reopen a platform default-deny Kubernetes NetworkPolicies are additive. Enabling this lets a tenant's own permissive NetworkPolicy reopen traffic that a platform-level default-deny was meant to close for the same pods. vCluster preserves a tenant's `ipBlock` peers unchanged, and it preserves a rule with no peers at all the same way. Neither carries tenant identity for vCluster to constrain, and Kubernetes treats a missing or empty peer list as matching every source or destination. Enable this only where the platform boundary uses a control tenants can't override, such as an AdminNetworkPolicy or a CNI-specific tiered policy. Otherwise, add control plane cluster admission that constrains synced `ipBlock` peers and rejects rules that omit their peer list entirely. See [network and endpoint isolation](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#primary-pod-network). ### Sync NetworkPolicies from the tenant to control plane cluster[​](#sync-networkpolicies-from-the-tenant-to-control-plane-cluster "Direct link to Sync NetworkPolicies from the tenant to control plane cluster") ``` sync: toHost: networkPolicies: enabled: true ``` info NetworkPolicy resources inside tenant clusters rely on the control plane cluster's support for this feature. Make sure that your control plane cluster satisfies the [Network Policy prerequisites](https://kubernetes.io/docs/concepts/services-networking/network-policies/#prerequisites). ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.networkPolicies.patches` to transform NetworkPolicy fields while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") ## `networkPolicies` object [​](#networkPolicies "Direct link to networkPolicies")NetworkPolicies defines if network policies created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean false [​](#networkPolicies-enabled "Direct link to networkPolicies-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#networkPolicies-patches "Direct link to networkPolicies-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#networkPolicies-patches-path "Direct link to networkPolicies-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#networkPolicies-patches-expression "Direct link to networkPolicies-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#networkPolicies-patches-reverseExpression "Direct link to networkPolicies-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#networkPolicies-patches-reference "Direct link to networkPolicies-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#networkPolicies-patches-reference-apiVersion "Direct link to networkPolicies-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#networkPolicies-patches-reference-apiVersionPath "Direct link to networkPolicies-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#networkPolicies-patches-reference-kind "Direct link to networkPolicies-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#networkPolicies-patches-reference-kindPath "Direct link to networkPolicies-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#networkPolicies-patches-reference-namePath "Direct link to networkPolicies-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#networkPolicies-patches-reference-namespacePath "Direct link to networkPolicies-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#networkPolicies-patches-labels "Direct link to networkPolicies-patches-labels")Labels treats the path value as a labels selector. --- # Services Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is enabled. Sync all [Service](https://kubernetes.io/docs/concepts/services-networking/service/) resources from the tenant cluster to the control plane cluster. LoadBalancer address pool governance on shared nodes A synced Service of type `LoadBalancer` draws its address from whatever pool your load balancer controller (for example, MetalLB) makes available on the control plane cluster. Without pool scoping, a tenant can claim an address from any pool, including a management-network or public pool. Scope address pools with namespace or Service selectors, or gate pool and `loadBalancerIP` selection with admission policy. Treat public IP exposure as an explicit opt-in, not an automatic default. See [LoadBalancer address pools](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#loadbalancer-address-pools). ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.services.patches` to transform Service fields, such as `spec.ports[*].name`, while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") Do Not Disable Disabling the syncing of this resource could cause the vCluster to not work properly. ## `services` object [​](#services "Direct link to services")Services defines if services created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean true [​](#services-enabled "Direct link to services-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#services-patches "Direct link to services-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#services-patches-path "Direct link to services-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#services-patches-expression "Direct link to services-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#services-patches-reverseExpression "Direct link to services-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#services-patches-reference "Direct link to services-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#services-patches-reference-apiVersion "Direct link to services-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#services-patches-reference-apiVersionPath "Direct link to services-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#services-patches-reference-kind "Direct link to services-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#services-patches-reference-kindPath "Direct link to services-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#services-patches-reference-namePath "Direct link to services-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#services-patches-reference-namespacePath "Direct link to services-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#services-patches-labels "Direct link to services-patches-labels")Labels treats the path value as a labels selector. --- # Persistent volume claims Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is enabled. Sync [PersistentVolumeClaim](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) resources from the tenant cluster to the control plane cluster. ### Disable syncing PersistentVolumeClaims from the tenant to control plane cluster[​](#disable-syncing-persistentvolumeclaims-from-the-tenant-to-control-plane-cluster "Direct link to Disable syncing PersistentVolumeClaims from the tenant to control plane cluster") ``` sync: toHost: persistentVolumeClaims: enabled: false ``` ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.persistentVolumeClaims.patches` to transform PersistentVolumeClaim fields while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") ## `persistentVolumeClaims` object [​](#persistentVolumeClaims "Direct link to persistentVolumeClaims")PersistentVolumeClaims defines if persistent volume claims created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean true [​](#persistentVolumeClaims-enabled "Direct link to persistentVolumeClaims-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#persistentVolumeClaims-patches "Direct link to persistentVolumeClaims-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#persistentVolumeClaims-patches-path "Direct link to persistentVolumeClaims-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#persistentVolumeClaims-patches-expression "Direct link to persistentVolumeClaims-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#persistentVolumeClaims-patches-reverseExpression "Direct link to persistentVolumeClaims-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#persistentVolumeClaims-patches-reference "Direct link to persistentVolumeClaims-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#persistentVolumeClaims-patches-reference-apiVersion "Direct link to persistentVolumeClaims-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#persistentVolumeClaims-patches-reference-apiVersionPath "Direct link to persistentVolumeClaims-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#persistentVolumeClaims-patches-reference-kind "Direct link to persistentVolumeClaims-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#persistentVolumeClaims-patches-reference-kindPath "Direct link to persistentVolumeClaims-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#persistentVolumeClaims-patches-reference-namePath "Direct link to persistentVolumeClaims-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#persistentVolumeClaims-patches-reference-namespacePath "Direct link to persistentVolumeClaims-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#persistentVolumeClaims-patches-labels "Direct link to persistentVolumeClaims-patches-labels")Labels treats the path value as a labels selector. --- # Persistent volumes Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. Sync [PersistentVolume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) resources from the tenant cluster to the control plane cluster. When disabled, vCluster creates pseudo PersistentVolume resources instead. Enabling PersistentVolume syncing is not necessary to use PersistentVolumeClaim syncing. Dynamic provisioning through a StorageClass and PersistentVolumeClaim sync doesn't require this setting. Enable this feature if you have a scenario that require details about the PersistentVolume and its IDs, you want to create PersistentVolume's directly in the tenant cluster, such as when you want to use Velero or specific CSI drivers. hostPath PersistentVolumes on shared nodes On shared nodes, enabling this lets a tenant create a PersistentVolume with `spec.hostPath` set to any path on the node, then mount it through an ordinary PersistentVolumeClaim. Pod Security Standards don't catch this, since the Pod's volume is a PVC, not a `hostPath` volume. If you enable PersistentVolume sync on shared nodes, add control plane cluster admission that denies synced PersistentVolumes with `spec.hostPath`, or restricts them to an explicit safe-path allowlist. See [storage breakout prevention](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#storage-breakout-prevention). ### Sync PersistentVolumes from the tenant to control plane cluster[​](#sync-persistentvolumes-from-the-tenant-to-control-plane-cluster "Direct link to Sync PersistentVolumes from the tenant to control plane cluster") ``` sync: toHost: persistentVolumes: enabled: true ``` ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.persistentVolumes.patches` to transform PersistentVolume fields while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") ## `persistentVolumes` object [​](#persistentVolumes "Direct link to persistentVolumes")PersistentVolumes defines if persistent volumes created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean false [​](#persistentVolumes-enabled "Direct link to persistentVolumes-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#persistentVolumes-patches "Direct link to persistentVolumes-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#persistentVolumes-patches-path "Direct link to persistentVolumes-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#persistentVolumes-patches-expression "Direct link to persistentVolumes-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#persistentVolumes-patches-reverseExpression "Direct link to persistentVolumes-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#persistentVolumes-patches-reference "Direct link to persistentVolumes-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#persistentVolumes-patches-reference-apiVersion "Direct link to persistentVolumes-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#persistentVolumes-patches-reference-apiVersionPath "Direct link to persistentVolumes-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#persistentVolumes-patches-reference-kind "Direct link to persistentVolumes-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#persistentVolumes-patches-reference-kindPath "Direct link to persistentVolumes-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#persistentVolumes-patches-reference-namePath "Direct link to persistentVolumes-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#persistentVolumes-patches-reference-namespacePath "Direct link to persistentVolumes-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#persistentVolumes-patches-labels "Direct link to persistentVolumes-patches-labels")Labels treats the path value as a labels selector. --- # Storage Classes Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. Sync [StorageClass](https://kubernetes.io/docs/concepts/storage/storage-classes/) resources from the tenant cluster to the control plane cluster. When enabled and when the necessary CSI driver is installed within the control plane cluster, vCluster updates certain parameters for the CSI driver through synced StorageClass resources. ### Sync StorageClasses from the tenant to control plane cluster[​](#sync-storageclasses-from-the-tenant-to-control-plane-cluster "Direct link to Sync StorageClasses from the tenant to control plane cluster") ``` sync: toHost: storageClasses: enabled: true ``` ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.storageClasses.patches` to transform StorageClass fields while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") ## `storageClasses` object [​](#storageClasses "Direct link to storageClasses")StorageClasses defines if storage classes created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean false [​](#storageClasses-enabled "Direct link to storageClasses-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#storageClasses-patches "Direct link to storageClasses-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#storageClasses-patches-path "Direct link to storageClasses-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#storageClasses-patches-expression "Direct link to storageClasses-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#storageClasses-patches-reverseExpression "Direct link to storageClasses-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#storageClasses-patches-reference "Direct link to storageClasses-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#storageClasses-patches-reference-apiVersion "Direct link to storageClasses-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#storageClasses-patches-reference-apiVersionPath "Direct link to storageClasses-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#storageClasses-patches-reference-kind "Direct link to storageClasses-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#storageClasses-patches-reference-kindPath "Direct link to storageClasses-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#storageClasses-patches-reference-namePath "Direct link to storageClasses-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#storageClasses-patches-reference-namespacePath "Direct link to storageClasses-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#storageClasses-patches-labels "Direct link to storageClasses-patches-labels")Labels treats the path value as a labels selector. --- # Volume snapshots Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** By default, this is disabled. Sync [VolumeSnapshot](https://kubernetes.io/docs/concepts/storage/volume-snapshots/) and [VolumeSnapshotContent](https://kubernetes.io/docs/concepts/storage/volume-snapshots/#volume-snapshot-contents) resources from the tenant cluster to the control plane cluster. These two resources work together. A VolumeSnapshot is the tenant-facing request for a snapshot of a PersistentVolumeClaim. A VolumeSnapshotContent is the underlying snapshot resource that a CSI driver provisions in response. ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * A CSI driver with snapshot support, its CSI snapshotter sidecar, and the common snapshot controller installed in the control plane cluster. See the [external-snapshotter project](https://github.com/kubernetes-csi/external-snapshotter) for these components. * The VolumeSnapshot, VolumeSnapshotContent, and VolumeSnapshotClass CRDs installed in the control plane cluster. When enabled, vCluster tries to detect these CRDs on the control plane cluster and fails to start up if they're missing. * A VolumeSnapshotClass available in the control plane cluster for the CSI driver you're using. Sync it into the tenant cluster with [`sync.fromHost.volumeSnapshotClasses`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/volume-snapshot-classes.md), since a VolumeSnapshot normally references a VolumeSnapshotClass by name to resolve which driver takes the snapshot. ### Enable syncing VolumeSnapshots and VolumeSnapshotContents to the control plane cluster[​](#enable-syncing-volumesnapshots-and-volumesnapshotcontents-to-the-control-plane-cluster "Direct link to Enable syncing VolumeSnapshots and VolumeSnapshotContents to the control plane cluster") ``` sync: fromHost: # Make the control plane cluster's snapshot classes available to users. volumeSnapshotClasses: enabled: true toHost: volumeSnapshots: enabled: true volumeSnapshotContents: enabled: true ``` You must enable `volumeSnapshots` and `volumeSnapshotContents` together. vCluster rejects a configuration that enables only one of these resources because each VolumeSnapshot depends on a corresponding VolumeSnapshotContent. ### RBAC[​](#rbac "Direct link to RBAC") Enabling volume snapshot syncing requires additional RBAC permissions for the vCluster ServiceAccount to manage VolumeSnapshot and VolumeSnapshotContent resources in the control plane cluster. [`rbac.enableVolumeSnapshotRules`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/rbac.md) defaults to `auto` and adds these permissions automatically when volume snapshot syncing is enabled. ## Patches[​](#patches "Direct link to Patches") Use `sync.toHost.volumeSnapshots.patches` and `sync.toHost.volumeSnapshotContents.patches` to transform these resources' fields while syncing to the control plane cluster. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for syntax, directionality, and examples. ## Config reference[​](#config-reference "Direct link to Config reference") ### `volumeSnapshots`[​](#volumesnapshots "Direct link to volumesnapshots") ## `volumeSnapshots` object [​](#volumeSnapshots "Direct link to volumeSnapshots")VolumeSnapshots defines if volume snapshots created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean false [​](#volumeSnapshots-enabled "Direct link to volumeSnapshots-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#volumeSnapshots-patches "Direct link to volumeSnapshots-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#volumeSnapshots-patches-path "Direct link to volumeSnapshots-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#volumeSnapshots-patches-expression "Direct link to volumeSnapshots-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#volumeSnapshots-patches-reverseExpression "Direct link to volumeSnapshots-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#volumeSnapshots-patches-reference "Direct link to volumeSnapshots-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#volumeSnapshots-patches-reference-apiVersion "Direct link to volumeSnapshots-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#volumeSnapshots-patches-reference-apiVersionPath "Direct link to volumeSnapshots-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#volumeSnapshots-patches-reference-kind "Direct link to volumeSnapshots-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#volumeSnapshots-patches-reference-kindPath "Direct link to volumeSnapshots-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#volumeSnapshots-patches-reference-namePath "Direct link to volumeSnapshots-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#volumeSnapshots-patches-reference-namespacePath "Direct link to volumeSnapshots-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#volumeSnapshots-patches-labels "Direct link to volumeSnapshots-patches-labels")Labels treats the path value as a labels selector. ### `volumeSnapshotContents`[​](#volumesnapshotcontents "Direct link to volumesnapshotcontents") ## `volumeSnapshotContents` object [​](#volumeSnapshotContents "Direct link to volumeSnapshotContents")VolumeSnapshotContents defines if volume snapshot contents created within the virtual cluster should get synced to the host cluster. ### `enabled` boolean false [​](#volumeSnapshotContents-enabled "Direct link to volumeSnapshotContents-enabled")Enabled defines if this option should be enabled. ### `patches` object\[] [​](#volumeSnapshotContents-patches "Direct link to volumeSnapshotContents-patches")Patches patch the resource according to the provided specification. #### `path` required string [​](#volumeSnapshotContents-patches-path "Direct link to volumeSnapshotContents-patches-path")Path is the path within the patch to target. If the path is not found within the patch, the patch is not applied. #### `expression` string [​](#volumeSnapshotContents-patches-expression "Direct link to volumeSnapshotContents-patches-expression")Expression transforms the value according to the given JavaScript expression. #### `reverseExpression` string [​](#volumeSnapshotContents-patches-reverseExpression "Direct link to volumeSnapshotContents-patches-reverseExpression")ReverseExpression transforms the value according to the given JavaScript expression. #### `reference` object [​](#volumeSnapshotContents-patches-reference "Direct link to volumeSnapshotContents-patches-reference")Reference treats the path value as a reference to another object and will rewrite it based on the chosen mode automatically. In single-namespace mode this will translate the name to "vxxxxxxxxx" to avoid conflicts with other names, in multi-namespace mode this will not translate the name. ##### `apiVersion` required string [​](#volumeSnapshotContents-patches-reference-apiVersion "Direct link to volumeSnapshotContents-patches-reference-apiVersion")APIVersion is the apiVersion of the referenced object. ##### `apiVersionPath` string [​](#volumeSnapshotContents-patches-reference-apiVersionPath "Direct link to volumeSnapshotContents-patches-reference-apiVersionPath")APIVersionPath is optional relative path to use to determine the kind. If APIVersionPath is not found, will fallback to apiVersion. ##### `kind` required string [​](#volumeSnapshotContents-patches-reference-kind "Direct link to volumeSnapshotContents-patches-reference-kind")Kind is the kind of the referenced object. ##### `kindPath` string [​](#volumeSnapshotContents-patches-reference-kindPath "Direct link to volumeSnapshotContents-patches-reference-kindPath")KindPath is the optional relative path to use to determine the kind. If KindPath is not found, will fallback to kind. ##### `namePath` string [​](#volumeSnapshotContents-patches-reference-namePath "Direct link to volumeSnapshotContents-patches-reference-namePath")NamePath is the optional relative path to the reference name within the object. ##### `namespacePath` string [​](#volumeSnapshotContents-patches-reference-namespacePath "Direct link to volumeSnapshotContents-patches-reference-namespacePath")NamespacePath is the optional relative path to the reference namespace within the object. If omitted or not found, namespacePath equals to the metadata.namespace path of the object. #### `labels` object [​](#volumeSnapshotContents-patches-labels "Direct link to volumeSnapshotContents-patches-labels")Labels treats the path value as a labels selector. --- # Telemetry Set your vCluster telemetry preferences. ### Why vCluster Labs collects telemetry[​](#why-vcluster-labs-collects-telemetry "Direct link to Why vCluster Labs collects telemetry") Because vCluster is a freely available open source project, vCluster Labs has a very limited idea of how the project is being used, and very limited possibilities to gather this information from the users. Without reliable information, it is difficult to make decisions about the prioritization of features, test automation or bug fixes. Deprecation of the flags and features turns into guesswork, and removal becomes nearly impossible. To advance the project and ensure long-term maintainability, decisions are made about feature deprecation, test coverage is prioritized, and all choices are guided by data. ### What vCluster Labs collects[​](#what-vcluster-labs-collects "Direct link to What vCluster Labs collects") vCluster Labs is not interested in collecting data about individuals that are using vCluster. Rather, vCluster Labs collects data about how vCluster is being used. This entails information about the configuration of vCluster, and the environment where it is deployed (for example, Kubernetes version and CPU architecture). Each vCluster is deployed with a syncer component, which contains all controllers that make a tenant cluster function. This component collects the data and uploads it to vCluster Labs at regular intervals (once every 1-5 minutes). vCluster Labs provides a documented example of the telemetry payload. The source code responsible for this is fully available in the vCluster [repo](https://github.com/loft-sh/vcluster). The telemetry backend is hosted at "". The ingestion service is written and maintained by the vCluster maintainers. The data is saved into a relational database with strict access control. ### Telemetry payload example[​](#telemetry-payload-example "Direct link to Telemetry payload example") This is an example of the payload that the vCluster syncer component would send to the telemetry backend. Some fields are self-explanatory, and some are explained after the example. ``` { "instanceProperties":{ "uid":"7629e852-34d9-4fa6-8fc7-4c0423b5c44b", "instanceCreator":"vclusterctl", "instanceCreatorUID":"2c98c17cb7b02912d68eed65398d136ea4ff42e77d14f48a4d383420f8eada44", "arch":"amd64", "os":"linux", "syncerVersion":"v0.15.0", "syncerFlags":"{setFlags:{\"node-selector\":true},controllers:[\"hoststorageclasses\"]}", "vclusterServiceType":"NodePort", "virtualKubernetesVersion":{ "major":"1", "minor":"26", "gitVersion":"v1.26.0+k3s1" }, "hostKubernetesVersion":{ "major":"1", "minor":"25", "gitVersion":"v1.25.3" } }, "events":[ { "type":"SyncerStarted", "time":1681418584841756 }, { "type":"LeadershipStarted", "time":1681418585169754 }, { "type":"ResourceSync", "time":1681418812947320, "success":true, "group":"core", "version":"v1", "kind":"ConfigMap" }, { "type":"ResourceSync", "time":1681418812968426, "success":true, "processingTime":21, "group":"core", "version":"v1", "kind":"Pod" } ], "token":"eyJhbGciOiJSUzI1NiIsImtpZCI6IjRnektQWDRLLTdNZXlYWVFRVjdKM1BVUnhHRTdOVW1UazIyeGphcVRmdlUifQ.eyJpc3MiOiJ2Y2x1c3Rlci10ZWxlbWV0cnkifQ.WEh-8EfCpB1mW24s-jLW1vkcgLzTo6UArxgL03A650BnHv5aUY-C-BcR_sdOoMJIGZifdTImYwWIZYBPD232odlOiVijKGH3CMRPH9gblzUeRkBO1Qdl_qq7pIjEIWiQ4y6-H02u3nFtXBWmdq77rF32pxtzonvknFPKCvdoICP773wiJk_Tvofno0GnMxWLuojqahZdTjo15ww4kB5BnZZoT7UrhyPRFYD6fAL1YJq8WLi0GSKN2hoeC5EpcvLzmCfV3qUd_3PmHTIvvUJdTvjDfTUjI7aQfn1XsPz_GfRCsv18LNmTd2PqYbOaIZP7GeJ5l63pNp0IDKNgjtnjdl6bPgtoV21cZjZ4iVd2ODjUiTBUtFTT8QxXPA6_fuX1ZCpnAEppfELopkrFKD0F9RQFZW8BVWiaKLiULy3AgfaOUYOTlt0-j52JXW7gbNXhdIgWvt5r99a7DUAd8yo76SJBxMA3WQfSs43SeppM_iegVAa13ZUSKsiyNWUZ84SXtT1nlma93JZPchtdNKpgQIWiKC9MrNf3DiibBLHln3DE21XMVzn7CEuFiNLy4zl088Kg3Ue3KJiJBvWWFBN1kf-5xq1BN9lfVzwbaRW_mF2zcKz5sXe8J5wbiFzWOuCTWu24XPUUVj4aW_SkTeag80iMqFtrSAt63diQ6s9gm9g" } ``` * `instanceProperties.uid` - is a unique identifier of a particular instance. It is used to deduplicate data sent over time. The `.metadata.uid` value of the vCluster PVC or Service resource is used as value. * `instanceProperties.instanceCreatorUID` - is a machine identifier attached by the vCluster CLI to an instance during creation. We use [machineid](https://github.com/denisbrodbeck/machineid) library to get the identifier, and then hash it for privacy. * `instanceProperties.syncerFlags` - contains a JSON payload that can have two fields: `setFlags` - list of the non-default syncer flags that were set, but the values are not collected, we only set `true` as value for each key; `controllers` - list of the resource sync controllers that have been enabled in addition to the default one. * `events` - an array of events for which we want to track duration and outcome (success/failure). We are sending just the GVK of the resource, but never any content. * `token` - this is a token generated in memory from a static key that is part of the vCluster binary. It is used to validate that the payload is being received from a real vCluster binary. The CLI may also send telemetry data. For example: ``` { "data": [ { "event": { "machine_id": "randomID", "platform_instance_id": "randomID", "platform_user_id": "randomID", "properties": "{\"command\":[\"vcluster\",\"completion\",\"bash\"],\"version\":\"0.18.1\"}", "timestamp": 17071593829, "type": "vcluster_cli" }, "user": { "machine_id": "randomID", "platform_instance_id": "randomID", "platform_user_id": "randomID", "properties": "{\"os_arch\":\"arm64\",\"os_name\":\"darwin\",\"timezone\":\"GMT\"}", "timestamp": 17071593829 } } ] } ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `telemetry` object [​](#telemetry "Direct link to telemetry")Configuration related to telemetry gathered about vCluster usage. ### `enabled` boolean true [​](#telemetry-enabled "Direct link to telemetry-enabled")Enabled specifies that the telemetry for the vCluster control plane should be enabled. ### `instanceCreator` string [​](#telemetry-instanceCreator "Direct link to telemetry-instanceCreator") ### `machineID` string [​](#telemetry-machineID "Direct link to telemetry-machineID") ### `platformUserID` string [​](#telemetry-platformUserID "Direct link to telemetry-platformUserID") ### `platformInstanceID` string [​](#telemetry-platformInstanceID "Direct link to telemetry-platformInstanceID") --- # What is vcluster.yaml? The `vcluster.yaml` file is the central configuration source for a tenant cluster. It controls the control plane architecture, worker node pool, resource sync rules, networking, and access settings. Some choices, such as the worker node type and backing store, must be made at deployment time and can't be changed later. The file is optional. If you don't provide one, the tenant cluster runs on shared nodes (the control plane cluster's node pool) with all resources syncing to a single namespace. Shared nodes are the vCluster OSS default and the only worker node model OSS provides. They suit internal, trusted tenants. For externally facing, resold, or regulated tenant offerings, [configure private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md), which require a vCluster license. Most production deployments benefit from explicit configuration. New to vCluster? If you haven't chosen a deployment path yet, start with [What is vCluster?](https://vcluster.com/docs/vcluster/introduction/what-is-vcluster.md) or the [architecture overview](https://vcluster.com/docs/vcluster/introduction/architecture.md) before configuring `vcluster.yaml`. ## Configuration structure for vcluster.yaml[​](#configuration-structure-for-vclusteryaml "Direct link to Configuration structure for vcluster.yaml") The `vcluster.yaml` file organizes configuration into logical sections. Each section controls a specific aspect of tenant cluster behavior. ### Control plane settings[​](#control-plane-settings "Direct link to Control plane settings") The control plane section defines how vCluster runs its Kubernetes API server, controller manager, and data storage. Key configuration areas include: * **Deployment settings**: * **Run as a container** - Configure how the control plane pods deploy, including resource limits, node selectors, and affinity rules. * **Run as a binary on a node** - Configure if you want the control plane to be deployed as a binary. Note: Worker node pool can only be private nodes. * **Backing store selection**: Choose between different storage backends for the vCluster control plane's [backing store](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store.md). Options include embedded databases, managed etcd or external databases. * **Networking options**: Define how external traffic reaches the tenant cluster's API server, including ingress configuration and service exposure methods. ### Worker node pool settings[​](#worker-node-pool-settings "Direct link to Worker node pool settings") You can configure which type of worker nodes to use for the tenant cluster. * **Shared nodes**: Use all or a labeled subset of the underlying cluster's worker nodes. * **Private nodes**: Dedicated nodes that join exclusively through a token-based process. Nodes can be manually provisioned or automatically managed by vCluster Platform. ### Access configuration[​](#access-configuration "Direct link to Access configuration") You can configure access settings to control how users and applications connect to the tenant cluster: * **The kubeconfig file**: Configure how vCluster generates and stores [the kubeconfig file](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/export-kube-config.md) that clients use to connect. This file contains API server endpoints, authentication credentials, and cluster context. You specify the Kubernetes secret name for storage, the context name for identification, and the server URL for connection. * **Authentication methods**: vCluster supports certificate-based and service account token authentication for secure access to the tenant cluster. When exposing vCluster through an ingress or load balancer, you can use service account tokens if client certificate validation is not supported. Authentication methods can be customized to fit your access requirements. ### Resource sync rules[​](#resource-sync-rules "Direct link to Resource sync rules") Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** When using shared nodes, the tenant cluster shares certain resources with the underlying cluster to function properly. The sync configuration controls this in both directions: * **From underlying cluster to tenant**: Configure which resources from the underlying cluster appear inside the tenant cluster through [host-to-virtual synchronization](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host.md). Common examples include nodes, storage classes, and priority classes. * **From tenant to underlying cluster**: Define which tenant cluster resources sync to the underlying cluster through [virtual-to-host synchronization](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host.md). This typically includes pods, services, and persistent volume claims. ## Example vcluster.yaml configuration[​](#example-vclusteryaml-configuration "Direct link to Example vcluster.yaml configuration") The following examples demonstrates common configurations and explains each section's purpose: Each configuration section has a specific purpose in the tenant cluster setup: * **Control plane with embedded storage**: The configuration uses [embedded etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/embedded.md) as the data store for the tenant cluster's control plane. This approach reduces operational complexity because etcd runs inside the vCluster pod rather than requiring a separate database. * **API server exposure**: An [Ingress](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/deployment/ingress.md) resource exposes the tenant cluster's API server through a hostname. This allows external tools and users to access the tenant cluster without port-forwarding or `NodePort` services. * **ServiceAccount synchronization**: The configuration enables [`ServiceAccount`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/service-accounts.md) syncing from the tenant cluster to the control plane cluster. This synchronization supports workloads that need specific permissions on the control plane cluster, such as operators or controllers that manage control plane cluster resources. * **Real node synchronization**: Instead of using vCluster's default "pseudo" nodes, this configuration syncs real [node](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md) information from the control plane cluster. The `clearImageStatus` option removes image information from synced nodes to reduce data transfer and storage requirements. * **Kubeconfig access**: The configuration defines how vCluster exports the [kubeconfig](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/export-kube-config.md) for accessing the tenant cluster. The exported kubeconfig gets stored in a Kubernetes secret, making it available for CI/CD pipelines, ArgoCD apps, or Terraform configurations. ### Control plane as a container and shared nodes[​](#control-plane-as-a-container-and-shared-nodes "Direct link to Control plane as a container and shared nodes") Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** vcluster.yaml ``` # Control plane configuration controlPlane: # Configure the backing store for vCluster's data backingStore: # Enterprise feature - requires license etcd: embedded: enabled: true # Run etcd inside the vCluster pod # Expose the API server via ingress ingress: enabled: true host: vcluster-k8s-api.example.com # Your API server hostname # Resource synchronization settings when using host cluster's worker nodes sync: # Resources that sync from virtual cluster to host toHost: serviceAccounts: enabled: true # Sync ServiceAccounts for operators/controllers # Resources that sync from host to virtual cluster fromHost: nodes: enabled: true # Sync real nodes instead of fake nodes clearImageStatus: true # Remove image data to save resources # Kubeconfig export configuration exportKubeConfig: context: my-vcluster-context # Name in the kubeconfig file server: https://vcluster-k8s-api.example.com # API server URL additionalSecrets: - name: my-vcluster-kubeconfig # Secret name for storing kubeconfig ``` ### Control plane as a container and private nodes[​](#control-plane-as-a-container-and-private-nodes "Direct link to Control plane as a container and private nodes") Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** vcluster.yaml ``` # Control plane configuration controlPlane: # Configure the backing store for vCluster's data backingStore: # Enterprise feature - requires license etcd: embedded: enabled: true # Run etcd inside the vCluster pod # Expose the API server via ingress ingress: enabled: true host: vcluster-k8s-api.example.com # Your API server hostname # Enabling private nodes as worker nodes privateNodes: enabled: true # Kubeconfig export configuration exportKubeConfig: context: my-vcluster-context # Name in the kubeconfig file server: https://vcluster-k8s-api.example.com # API server URL additionalSecrets: - name: my-vcluster-kubeconfig # Secret name for storing kubeconfig ``` ### Control plane as a binary and private nodes[​](#control-plane-as-a-binary-and-private-nodes "Direct link to Control plane as a binary and private nodes") Supported Configurations Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. vcluster.yaml ``` # Control plane configuration controlPlane: # Enabling control plane to be installed as a binary on a node standalone: enabled: true joinNode: enabled: true # Configure the backing store for vCluster's data backingStore: # Enterprise feature - requires license etcd: embedded: enabled: true # Run etcd inside the vCluster pod # Enabling private nodes as worker nodes privateNodes: enabled: true # Kubeconfig export configuration exportKubeConfig: context: my-vcluster-context # Name in the kubeconfig file server: https://vcluster-k8s-api.example.com # API server URL additionalSecrets: - name: my-vcluster-kubeconfig # Secret name for storing kubeconfig ``` ## Deploy a tenant cluster[​](#deploy-a-tenant-cluster "Direct link to Deploy a tenant cluster") Before you deploy, review the [worker node deployment options](https://vcluster.com/docs/vcluster/introduction/architecture.md#worker-nodes) to determine how the infrastructure of the tenant cluster will be configured. Once you've chosen your deployment path, read the different ways to deploy: ### Control plane as a container[​](#control-plane-as-a-container "Direct link to Control plane as a container") * [Basics](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/basics.md) * [High Availability](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/high-availability.md) * In Different environments * [EKS](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/eks.md) * [AKS](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/aks.md) * [GKE](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/gke.md) * [OpenShift](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/openshift.md) ### Control plane as a binary[​](#control-plane-as-a-binary "Direct link to Control plane as a binary") * [Basics](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) * [High Availability](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md) ## Frequently asked questions[​](#frequently-asked-questions "Direct link to Frequently asked questions") **Is a `vcluster.yaml` file required?** The `vcluster.yaml` file is optional. If not provided, vCluster applies default settings that: * Use an embedded database (SQLite). * Sync essential resources such as pods and services. * Configure networking with basic `ClusterIP` services. * Enable default authentication and access controls. note The `vcluster.yaml` configuration defaults are for development and simple test scenarios. For production use, defining a `vcluster.yaml` file is strongly recommended to meet specific security, performance, and integration needs. **Can an existing tenant cluster be updated using `vcluster.yaml`?** Yes. You can update an existing tenant cluster by applying a modified `vcluster.yaml` file. This allows you to change the configuration without deleting or recreating the cluster. To apply changes, use the `--upgrade` flag with the `vcluster create` command: ``` vcluster create VCLUSTER_NAME --upgrade -f vcluster.yaml ``` The command performs the following actions: 1. Reads the updated `vcluster.yaml` file. 2. Compares it to the current configuration of the tenant cluster. 3. Applies only the differences between the new and existing configuration. 4. Triggers a rolling update if required to apply the changes. Some configuration changes, such as updates to sync settings, take effect immediately. Others, such as control plane modifications, may require pod restarts. The vCluster CLI handles these updates automatically and ensures minimal disruption to workloads. For more details on updating tenant clusters, see the [deployment changes documentation](https://vcluster.com/docs/vcluster/manage/deploy-changes.md). --- # Deploy vCluster Your infrastructure determines your deployment path. Choose the section that matches what you have today. * [You have a Kubernetes cluster](#you-have-a-kubernetes-cluster) * [You have bare metal or VMs](#you-have-bare-metal-or-vms) * [You're using Docker](#youre-using-docker) Choose your worker node model before deployment For a customer-facing or production offering, decide between shared and private nodes before you deploy. It's locked at creation and can't change for an existing tenant cluster. See [Choose a worker node model](https://vcluster.com/docs/vcluster/production-guide/choose-worker-node-model.md). ## You have a Kubernetes cluster[​](#you-have-a-kubernetes-cluster "Direct link to You have a Kubernetes cluster") The control plane runs as a pod on any conformant Kubernetes cluster, including EKS, GKE, AKS, OpenShift, and self-managed clusters. You choose how to provision worker nodes for each tenant cluster: * **[Shared nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes.md)** — tenant workloads run on the Control Plane Cluster's existing node pool. For internal, trusted tenants such as internal platforms, CI/CD, testing, and development. Not for externally facing or resold tenants. * **[Private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md)** — dedicated nodes join each tenant cluster exclusively. Ideal for GPU workloads, regulated environments, and full isolation. You can't mix shared and private nodes in a single vCluster, and you can't change the node type after deployment. warning **Decide before you deploy.** vCluster locks the worker node type, backing store (SQLite, etcd, MySQL, or PostgreSQL), and control plane distro at creation time. You can't change them by upgrading an existing vCluster. [Deploy on a Kubernetes cluster →](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/basics.md) ## You have bare metal or VMs[​](#you-have-bare-metal-or-vms "Direct link to You have bare metal or VMs") [vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md) deploys a full Kubernetes cluster directly on your nodes. No existing cluster required, minimal setup, no heavyweight distribution overhead. Add nodes as needed and have a working cluster in minutes. Once the cluster is up, join worker nodes from any infrastructure. Nodes connect to the exposed cluster endpoint using a bootstrap token, so workers don't need to be co-located. Bring in cloud provider instances, VMs from a different site, or additional bare metal machines. See [Private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for requirements and steps. To add vCluster tenant clusters on top, follow the [Kubernetes path](#you-have-a-kubernetes-cluster) above. info **Decide before you deploy.** vCluster sets the backing store at install time. You can't change it without redeploying. [Deploy Standalone →](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md) ## You're using Docker[​](#youre-using-docker "Direct link to You're using Docker") [vind](https://vcluster.com/docs/vcluster/deploy/control-plane/docker-container/basics.md) runs a full Kubernetes cluster entirely in Docker. The control plane and worker nodes all run as containers. Docker is the only requirement. Use vind for local development, offline CI pipelines, and testing configurations before committing to a cluster deployment. warning **Decide before you deploy.** vCluster sets the backing store and control plane distro at creation time. You can't change them without redeploying. [Deploy with Docker →](https://vcluster.com/docs/vcluster/deploy/control-plane/docker-container/basics.md) --- # Control plane benchmark methodology This page documents the test harness behind the [Sizing and performance](https://vcluster.com/docs/vcluster/deploy/control-plane/sizing-and-performance.md) page: workload profiles, service level indicators, and the search procedure used to find each profile's stable operating point. ## Test setup[​](#test-setup "Direct link to Test setup") Control plane only Workloads are simulated. KWOK fake nodes do not run real container runtimes, image pulls, kubelets, or CNI plumbing. The synthetic workloads here pressure the Virtual Control Plane (syncer, embedded kube-apiserver, kube-controller-manager, kube-scheduler, and etcd) — not pod startup, network programming, DNS, or storage paths. Results may differ from production workloads with real container runtimes and operators. Six resource profiles (P1–P6) vary CPU and memory per control plane replica, mapping directly to the [sizing table](https://vcluster.com/docs/vcluster/deploy/control-plane/sizing-and-performance.md#choose-a-resource-profile). Every profile runs three replicas (HA) with embedded etcd. ### Workload profiles[​](#workload-profiles "Direct link to Workload profiles") Four workload profiles vary object count, pod density, and controller density inside the Tenant Cluster: | Profile | Description | Churn model | | ------- | ---------------------------------------------------------------- | -------------------------------------------------------- | | K0 | Baseline — KWOK nodes only, no synthetic workload | 10-minute stability window | | W7 | Heavy realistic — mixed Deployment shapes across many namespaces | Pod lifecycle, namespace churn, controller PATCH traffic | | W8 | Extreme realistic — 3× W7 scale | Same as W7, scaled up to surface etcd write-path effects | | W9 | Controller density — W8 plus persistent watch streams | W8 churn + synthetic operator watches | W9 closes the gap between synthetic load and a real Tenant Cluster with a full operator stack: cert-manager, external-secrets, Argo, Kubeflow, GPU Operator, KServe, Prometheus Operator, KEDA, and similar. ### Service level indicators[​](#service-level-indicators "Direct link to Service level indicators") We declare a Stable Node Count (SN) result only when all 13 conditions hold across the evaluation window. Each is a PromQL query against the vCluster namespace. | # | Condition | Threshold | | -- | ------------------------------ | -------------------------------------------------------------------- | | 1 | Control plane replica Ready | Any pod not Ready → fail | | 2a | Container restart count | > 2 → fail (the 1→3 HA bootstrap causes 1–2 expected pod-0 restarts) | | 2b | OOMKilled | Any event → fail | | 3a | Sustained 429 responses | > 0.1 req/s → fail | | 3b | Sustained 5xx responses | > 0.1 req/s → fail | | 3c | Sustained request terminations | > 0.1 req/s → fail | | 4 | API P99 latency (non-watch) | > 2 s sustained → fail | | 5a | etcd leader lost | Any interval → fail | | 5b | etcd leader changes | Any → fail | | 5c | etcd proposals failed | Any → fail | | 5d | etcd WAL fsync P99 | > 25 ms → warn | | 5e | etcd DB size | > 4 GB → warn; > 12 GB → fail | | 6 | Node convergence | < 98% of target N → fail | Crossing any condition invalidates the stage result for that profile but doesn't necessarily mean total cluster failure. The thresholds define a conservative, reproducible operating envelope, not a cliff edge. Each test stage emits two artifacts alongside full control plane logs and a configuration manifest: * `fail-conditions-.{json,txt}` — pass/fail evaluation of every condition. * `collected-metrics-/` — raw time series for 41 metrics across eight categories (API latency, API errors, etcd health, container resources, pod state, scheduler, workqueue, and Control Plane Cluster metrics). ### Search procedure[​](#search-procedure "Direct link to Search procedure") For each `Profile × Workload` pair, the test runner finds SN in two phases: 1. **Doubling ladder** — starting from N = 100 nodes, double node count at each stage (100 → 200 → 400 → 800, and so on) until at least one SLI fails. 2. **Binary search** — bisect between the last passing node count and the first failing count until the gap is ≤ 15%. The midpoint, rounded down, is reported as SN. Each stage provisions a fresh Tenant Cluster and registers N KWOK nodes. It then runs the workload generator, evaluates all SLIs, and records artifacts before tearing down the Tenant Cluster and moving to the next stage. ### Node simulation[​](#node-simulation "Direct link to Node simulation") KWOK simulates worker nodes, emulating node and pod lifecycle events without running real container runtimes. KWOK nodes register directly to the Virtual Control Plane's API server, matching the [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) deployment topology these benchmarks target. KWOK doesn't exercise real kubelet behavior: image pulls, CSI mounts, and device plugins. The vCluster control-plane code paths for node registration, heartbeats, pod scheduling, and status updates work the same as with real nodes. ### Test environment[​](#test-environment "Direct link to Test environment") Terraform provisions each resource profile's isolated EKS cluster (using `make apply-p1` through `make apply-p6`). Every cluster ships with Karpenter for Control Plane Cluster autoscaling, a monitoring stack (Prometheus, Grafana, Loki, OpenTelemetry), and vCluster Platform managing the Tenant Cluster under test. Prometheus scrapes the vCluster API server and etcd endpoints through a `ServiceMonitor` enabled with `controlPlane.serviceMonitor.enabled`. ## Analysis[​](#analysis "Direct link to Analysis") ### CPU is the bottleneck below 4 CPU per replica[​](#cpu-is-the-bottleneck-below-4-cpu-per-replica "Direct link to CPU is the bottleneck below 4 CPU per replica") Under earlier, lighter workload profiles now superseded by W7/W8/W9, capacity scaled linearly with CPU across P1, P2, and P3. At P1, CPU saturation caused a pod restart and a subsequent etcd leader change, confirming CPU as the binding resource. Resource profile drives capacity in this regime, and linear sizing is a reasonable planning model. ### The etcd write path is the bottleneck above 4 CPU[​](#the-etcd-write-path-is-the-bottleneck-above-4-cpu "Direct link to The etcd write path is the bottleneck above 4 CPU") From P3 onward, adding CPU doesn't raise the node ceiling. The bottleneck shifts to the etcd write path: WAL fsync latency, proposal commit throughput, and apply throughput. At high-CPU profiles, API p99 latency differences between adjacent profiles are within single-digit percentages at the same node count, with no change to the ceiling. Raising this ceiling requires moving etcd out-of-process, tuning batch apply behavior, or sharding the write path. Adding more CPU to the control plane replicas won't help. ### Memory scales with object count[​](#memory-scales-with-object-count "Direct link to Memory scales with object count") In W9 tests, pod, Deployment, and Service object count drove etcd DB size, not node count. When sizing memory, plan for object density (pods and controllers per namespace) rather than raw node count. ## Limitations[​](#limitations "Direct link to Limitations") These benchmarks cover one specific vCluster control-plane configuration. Keep these constraints in mind when applying the results: * **KWOK simulates nodes, not kubelets.** Workloads sensitive to kubelet behavior are out of scope. These include image pulls, CSI mounts, device plugins, and kubelet metrics. * **Fixed infrastructure.** Tests hold EKS, a specific Kubernetes version, storage class, and CNI constant across profiles. Results vary on other clouds, Kubernetes distributions, and disk classes. Anything that changes WAL fsync latency is especially significant. * **Fixed admission and operator surface.** Tests don't vary admission webhooks, custom resource definitions, or operator density on the Tenant Cluster. CRD-heavy workloads add separate watch caches and informers per Kind, raising memory and watch overhead linearly with Kind count. Examples include PyTorchJobs, KServe Inferences, and Knative Revisions. * **Short-duration runs.** Individual test runs are 30–60 minutes. Long-tail issues such as gradual memory growth over days are out of scope. * **One Tenant Cluster at a time.** Tests don't measure co-tenancy effects from multiple Tenant Clusters sharing a Control Plane Cluster. ## Reproduce the benchmarks[​](#reproduce-the-benchmarks "Direct link to Reproduce the benchmarks") Reproduce these benchmarks against your own configuration using the open source framework: [loft-sh/vCluster-Control-Plane-Load-Testing](https://github.com/loft-sh/vcluster-control-plane-load-testing). The repo includes Terraform modules for per-profile EKS clusters, kube-burner workload configurations for K0/W7/W8/W9, the 13-SLI fail-condition checks, and the doubling-ladder and binary-search scripts. --- # vCluster Standalone Supported Configurations Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. vCluster Standalone is a different architecture model for vCluster where the control plane runs directly on your infrastructure with no dependency on an existing Control Plane Cluster. vCluster is deployed directly onto nodes like other Kubernetes distributions. vCluster Standalone can run on any type of node, whether that is a bare-metal node or VM. It provides the strictest tenant isolation as there is no shared Control Plane Cluster for the control plane or worker nodes. When using vCluster Standalone, the control plane runs on its own independent node rather than as a pod on a shared cluster. Worker nodes must be private nodes.
![Comparison of traditional vCluster (control plane as pod) versus vCluster Standalone (control plane as binary on dedicated nodes)](/docs/media/private-nodes/architecture.png) ## Resource requirements[​](#resource-requirements "Direct link to Resource requirements") vCluster Standalone can be deployed on bare metal, virtual machines, or containers. The minimal recommended resource requirements for vCluster Standalone are as follows: * 0.2 vCPU * 512 MiB of RAM * 1 GiB of ephemeral storage info These are minimal requirements suitable for testing and development. Production deployments should be sized based on your specific workload, cluster size, and the number of tenant clusters you plan to run. ## Installation overview[​](#installation-overview "Direct link to Installation overview") vCluster Standalone requires its own node for the control plane. You can add additional nodes later as workers. * Self-managed * Platform-managed | Step | Summary | | ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | | 1. [Install control plane node](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) | Install the initial control plane node on your infrastructure. | | 2. [Join control plane nodes](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md) (optional) | Join additional control plane nodes for HA. | | 3. [Join worker nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) (optional) | Join worker nodes as private nodes. | info Self-managed Standalone clusters can later be connected to **[vCluster Platform](https://vcluster.com/docs/platform/.md)**. Once connected, configuration updates and version upgrades can be coordinated from the platform with rolling upgrades, while node management remains self-managed. | Step | Summary | | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | | 1. [Install control plane node](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) | Use vCluster Platform to provision the initial control plane node. | | 2. Add control plane nodes (optional) | Scale control plane nodes from the Platform UI. | | 3. Add worker nodes (optional) | Add private worker nodes from the Platform UI. | [Install the control plane →](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) --- # Create control plane Supported Configurations Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. ## Overview[​](#overview "Direct link to Overview") * Self-managed * Platform-managed When deploying a Platform-managed vCluster Standalone cluster, [vCluster Platform](https://vcluster.com/docs/platform/.md) handles automated provisioning and lifecycle management of the control plane node, including rolling upgrades. When deploying a self-managed vCluster Standalone cluster, the assets required to install the control plane are located in the [GitHub releases](https://github.com/loft-sh/vcluster/releases) of vCluster. Why is the configuration so minimal? vCluster Standalone eliminates the dependency on an external Control Plane Cluster by running directly on your infrastructure. This architectural difference means: * **`controlPlane.standalone.enabled`** is automatically set when vCluster detects it's running as a binary rather than as a pod * **`controlPlane.privateNodes.enabled`** is automatically set because standalone mode has no Control Plane Cluster, so all worker nodes are private nodes that join directly If you're familiar with traditional vCluster deployments, you'll notice this configuration has fewer required settings because there's no Control Plane Cluster relationship to configure. For optional settings like dedicating the control plane node (not running workloads on it), configuring containerd, or customizing node registration, see the [standalone configuration reference](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/standalone.md). For vcluster.yaml options that are not supported in Standalone mode, see [Unsupported configuration options](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/troubleshooting.md#unsupported-configuration-options). ## Predeployment configuration options[​](#predeployment-configuration-options "Direct link to Predeployment configuration options") Before deploying, review the configuration options that can't be updated after deployment. These options require you to deploy a new vCluster instance rather than upgrading your existing one. ### Control-plane options[​](#control-plane-options "Direct link to Control-plane options") * [High availability](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md) - Run multiple control plane nodes * [CoreDNS](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/coredns.md) - Currently only CoreDNS deployed by vCluster during startup is supported. * Backing Store - Decide how the data of your cluster is stored, must be one of either [embedded SQLite](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/sqlite.md) (the default) or [embedded etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/embedded.md). ### Node roles[​](#node-roles "Direct link to Node roles") Decide if the control plane node will also be a worker node or not. Once a node joins the cluster, the roles of the node cannot change. By default, the control plane node also acts as a worker node. To deploy a dedicated control plane that does not run workloads, set [`controlPlane.standalone.joinNode.enabled`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/standalone.md#standalone-joinNode-enabled) to `false`. warning When the control plane node also acts as a worker, tenant workloads share the machine with the vCluster control-plane process and its credentials. For production deployments, set [`controlPlane.standalone.joinNode.enabled`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/standalone.md#standalone-joinNode-enabled) to `false` and use dedicated worker nodes. ### Worker nodes[​](#worker-nodes "Direct link to Worker nodes") With vCluster Standalone, worker node pools can only be [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md). Since there is no Control Plane Cluster, there is no concept of Shared Nodes. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Access to a node that satisfies the [node requirements](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/node-requirements.md) ## Install control plane node[​](#install-control-plane-node "Direct link to Install control plane node") ### Self-managed[​](#self-managed "Direct link to Self-managed") Control Plane Node Perform all steps on the control plane node as root. 1. Switch to root and set `CONFIG_DIRECTORY` to the directory where you want to store the `vcluster.yaml` configuration file. ``` sudo su - ``` Modify the following with your specific values to replace on the whole page and generate copyable commands: CONFIG\_DIRECTORY/etc/vcluster 2. Create directory for storing vCluster Standalone configuration. Create config directory ``` mkdir -p /etc/vcluster ``` Save a `vcluster.yaml` configuration file for vCluster Standalone on the control plane node. The configuration file is optional. Provide one when you need to pin the Kubernetes version or customize other settings. note vCluster deploys Kubernetes **v1.36.0** by default. To use a different version, set `controlPlane.distro.k8s.image.tag` in your `vcluster.yaml`. Create vCluster config file ``` cat < /etc/vcluster/vcluster.yaml controlPlane: distro: k8s: version: v1.36.0 EOF ``` warning Adding additional control plane nodes is not supported unless you follow the [high availability](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md) steps for configuration. 3. Run the installation script on the control plane node: Install vCluster Standalone on control plane node ``` curl -sfL https://github.com/loft-sh/vcluster/releases/download/v0.37.0/install-standalone.sh | sh -s -- --vcluster-name standalone ``` 4. Check that the control plane node is ready by running these commands: Check node status ``` kubectl get nodes ``` Expected output: ``` NAME STATUS ROLES AGE VERSION ip-192-168-3-131 Ready control-plane,master 11m v1.36.0 ``` Verify cluster components are running ``` kubectl get pods -A ``` Pods should include: * **Flannel**: CNI for container networking * **CoreDNS**: DNS service for the cluster * **KubeProxy**: Network traffic routing and load balancing * **Konnectivity**: Secure control plane to worker node communication * **Local Path Provisioner**: Dynamic storage provisioning #### Available flags to use in the install script[​](#available-flags-to-use-in-the-install-script "Direct link to Available flags to use in the install script") There are several flags available that can be added to the script. | Flag | Description | | ------------------------------ | ------------------------------------------------------------------ | | `--vcluster-name` | Name of the vCluster instance | | `--vcluster-version` | Specific vCluster version to install | | `--config` | Path to the vcluster.yaml configuration file | | `--binary` | Path to an existing vCluster binary (use with `--skip-download`) | | `--skip-download` | Skip downloading vCluster binary (use existing) | | `--skip-wait` | Exit without waiting for vCluster to be ready | | `--extra-env` | Additional environment variables for vCluster | | `--join-token` | Token for joining additional nodes to the cluster | | `--join-endpoint` | Endpoint address for joining additional nodes | | `--vcluster-kubernetes-bundle` | Path to an air-gapped Kubernetes bundle | | `--reset-only` | Uninstall and reset the vCluster installation without reinstalling | | `--fips` | Enable FIPS-compliant mode | | `--platform-access-key` | Access key for vCluster Platform integration | | `--platform-host` | vCluster Platform host URL | | `--platform-insecure` | Skip TLS verification for Platform connection | | `--platform-instance-name` | Instance name in vCluster Platform | | `--platform-project` | Project name in vCluster Platform | #### Access your cluster[​](#access-your-cluster "Direct link to Access your cluster") After installation, vCluster automatically configures the kubeconfig on the control plane node and sets the kubectl context to your new vCluster Standalone instance. To use vCluster Standalone as a Control Plane Cluster for tenant clusters, set your kubectl context to the vCluster Standalone instance. You can then create and manage tenant clusters using the vCluster CLI. To access the cluster from other machines, copy the kubeconfig from `/var/lib/vcluster/kubeconfig.yaml` on the control plane node, then replace the server field with relevant IP or DNS to access the cluster. You can also configure vCluster to make [external access easier](https://vcluster.com/docs/vcluster/manage/accessing-vcluster.md#access-vcluster-externally). The vCluster CLI is installed at `/var/lib/vcluster/bin/vcluster-cli`. ### Platform-managed[​](#platform-managed "Direct link to Platform-managed") When managing a standalone cluster through [vCluster Platform](https://vcluster.com/docs/platform/.md), the initial control plane node is provisioned through the platform using [Auto Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/auto-nodes.md). Platform connects to your node provider (such as AWS, GCP, or a bare metal provisioner), provisions a VM or physical server, installs the vCluster binary, and joins the node as the control plane. Once the cluster is running, Platform manages its full lifecycle. Platform coordinates configuration updates and version upgrades as rolling operations, eliminating the need for manual SSH access. Use [vCluster Platform](https://vcluster.com/docs/platform/.md) to: 1. Add a [Node Provider](https://vcluster.com/docs/platform/administer/node-providers/overview.md). 2. Add the vCluster configuration (example below). 3. Provision the cluster from the platform UI. Modify the following with your specific values to replace on the whole page and generate copyable commands: K8S\_VERSIONv1.36.0 NODE\_PROVIDERaws vcluster.yaml for a standalone control plane managed by vCluster Platform ``` controlPlane: standalone: enabled: true autoNodes: provider: aws # Node provider you want to use quantity: 1 # Number of nodes (HA requires embedded etcd or external DB) distro: k8s: image: tag: v1.36.0 # Kubernetes version you want to use # Worker nodes privateNodes: enabled: true autoNodes: # (optional) Add worker nodes with Auto Nodes - provider: aws dynamic: - name: aws-pool-1 ``` After provisioning completes, [vCluster Platform](https://vcluster.com/docs/platform/.md) manages the control plane node lifecycle. Worker node lifecycle also remains managed through the platform UI when [Auto Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/auto-nodes.md) are used. #### Access your cluster[​](#access-your-cluster-1 "Direct link to Access your cluster") To access a Standalone cluster managed by [vCluster Platform](https://vcluster.com/docs/platform/.md), open the vCluster in the platform UI and click `Connect`. As an alternative, use the [`vcluster platform connect vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_connect_vcluster) command. --- # Deploy in high availability Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. Supported Configurations Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. ## Overview[​](#overview "Direct link to Overview") By default, vCluster Standalone installs on a single control plane node. For production deployments, run multiple control plane nodes in high availability (HA) to improve resilience. Each additional control plane node joins the cluster one at a time, starting after the initial node is running. ## Predeployment configuration options[​](#predeployment-configuration-options "Direct link to Predeployment configuration options") ### Backing store must be embedded etcd or external database[​](#backing-store-must-be-embedded-etcd-or-external-database "Direct link to Backing store must be embedded etcd or external database") HA requires either embedded etcd or an external database as the backing store. This must be configured on the initial node before adding additional control plane nodes. ### Control plane node roles[​](#control-plane-node-roles "Direct link to Control plane node roles") Decide if each control plane node will also act as a worker node. Once a node joins the cluster, its roles cannot change. ### Worker nodes[​](#worker-nodes "Direct link to Worker nodes") With vCluster Standalone, worker nodes can only be [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md). Since there is no Control Plane Cluster, there is no concept of Shared Nodes. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Access to nodes that satisfy the [node requirements](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/node-requirements.md) ## Install initial control plane node[​](#install-initial-control-plane-node "Direct link to Install initial control plane node") * Self-managed * Platform-managed Control Plane Node Perform all steps on the initial control plane node as root. 1. Switch to root, then create a `vcluster.yaml` with HA enabled. Switch to root ``` sudo su - ``` Create vCluster config file with HA enabled ``` mkdir -p /etc/vcluster cat < /etc/vcluster/vcluster.yaml controlPlane: distro: k8s: version: v1.36.0 backingStore: etcd: embedded: enabled: true # Required for HA (or use external DB) EOF ``` 2. Run the installation script on the control plane node: Install vCluster Standalone on control plane node ``` curl -sfL https://github.com/loft-sh/vcluster/releases/download/v0.37.0/install-standalone.sh | sh -s -- --vcluster-name standalone ``` 3. Check that the control plane node is ready by running these commands: Check node status ``` kubectl get nodes ``` Expected output: ``` NAME STATUS ROLES AGE VERSION ip-192-168-3-131 Ready control-plane,master 11m v1.36.0 ``` Verify cluster components are running ``` kubectl get pods -A ``` Pods should include: * **Flannel**: CNI for container networking * **CoreDNS**: DNS service for the cluster * **KubeProxy**: Network traffic routing and load balancing * **Konnectivity**: Secure control plane to worker node communication * **Local Path Provisioner**: Dynamic storage provisioning #### Available flags to use in the install script[​](#available-flags-to-use-in-the-install-script "Direct link to Available flags to use in the install script") There are several flags available that can be added to the script. | Flag | Description | | ------------------------------ | ------------------------------------------------------------------ | | `--vcluster-name` | Name of the vCluster instance | | `--vcluster-version` | Specific vCluster version to install | | `--config` | Path to the vcluster.yaml configuration file | | `--binary` | Path to an existing vCluster binary (use with `--skip-download`) | | `--skip-download` | Skip downloading vCluster binary (use existing) | | `--skip-wait` | Exit without waiting for vCluster to be ready | | `--extra-env` | Additional environment variables for vCluster | | `--join-token` | Token for joining additional nodes to the cluster | | `--join-endpoint` | Endpoint address for joining additional nodes | | `--vcluster-kubernetes-bundle` | Path to an air-gapped Kubernetes bundle | | `--reset-only` | Uninstall and reset the vCluster installation without reinstalling | | `--fips` | Enable FIPS-compliant mode | | `--platform-access-key` | Access key for vCluster Platform integration | | `--platform-host` | vCluster Platform host URL | | `--platform-insecure` | Skip TLS verification for Platform connection | | `--platform-instance-name` | Instance name in vCluster Platform | | `--platform-project` | Project name in vCluster Platform | #### Access your cluster[​](#access-your-cluster "Direct link to Access your cluster") After installation, vCluster automatically configures the kubeconfig on the control plane node and sets the kubectl context to your new vCluster Standalone instance. To access the cluster from other machines, copy the kubeconfig from `/var/lib/vcluster/kubeconfig.yaml` on the control plane node or use the vCluster CLI to [generate access credentials](https://vcluster.com/docs/vcluster/manage/accessing-vcluster.md). #### Platform-managed[​](#platform-managed "Direct link to Platform-managed") When managing a Standalone HA cluster through [vCluster Platform](https://vcluster.com/docs/platform/.md), the initial control plane node is provisioned through the Platform using [Auto Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/auto-nodes.md). Platform coordinates configuration updates and version upgrades as rolling operations, eliminating the need for manual SSH access. Use [vCluster Platform](https://vcluster.com/docs/platform/.md) to: 1. Add a [Node Provider](https://vcluster.com/docs/platform/administer/node-providers/overview.md). 2. Add the vCluster configuration (example below). 3. Provision the cluster from the platform UI. Modify the following with your specific values to replace on the whole page and generate copyable commands: K8S\_VERSIONv1.36.0 NODE\_PROVIDERaws CONTROL\_PLANE\_NODES3 vcluster.yaml for an HA standalone control plane managed by vCluster Platform ``` controlPlane: standalone: enabled: true autoNodes: provider: aws # Node provider you want to use quantity: 3 # Number of nodes distro: k8s: image: tag: v1.36.0 # Kubernetes version you want to use backingStore: etcd: embedded: enabled: true # Required for HA (or use external DB) # Worker nodes privateNodes: enabled: true autoNodes: # (optional) Add worker nodes with Auto Nodes - provider: aws dynamic: - name: aws-pool-1 ``` After provisioning completes, [vCluster Platform](https://vcluster.com/docs/platform/.md) manages the control plane node lifecycle. Worker node lifecycle also remains managed through the platform UI when [Auto Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/auto-nodes.md) are used. #### Access your cluster[​](#access-your-cluster-1 "Direct link to Access your cluster") To access a Standalone cluster managed by [vCluster Platform](https://vcluster.com/docs/platform/.md), open the vCluster in the Platform UI and click `Connect`. As an alternative, use the [`vcluster platform connect vcluster`](https://vcluster.com/docs/vcluster/cli/vcluster_platform_connect_vcluster) command. ## Add additional control plane nodes[​](#add-additional-control-plane-nodes "Direct link to Add additional control plane nodes") After the initial control plane node is running, additional nodes join the cluster using a bootstrap token and the install script. ### Create a join token[​](#create-a-join-token "Direct link to Create a join token") Generate a token on the initial control plane node. A single token can be used for multiple nodes, or you can create one per node. Create a token for control plane nodes ``` /var/lib/vcluster/bin/vcluster-cli token create --control-plane --expires=1h ``` The command returns the join endpoint and token: Example output ``` Join endpoint: https:// Join token: ``` By default the token expires after 1 hour. The token is stored as a Secret prefixed with `bootstrap-token-` in the `kube-system` namespace. ### Join each control plane node[​](#join-each-control-plane-node "Direct link to Join each control plane node") On each additional control plane node, run the install script with the `--join-token` and `--join-endpoint` flags: Modify the following with your specific values to generate a copyable command: JOIN\_TOKENyour-token JOIN\_ENDPOINThttps\://your-vcluster-endpoint Join an additional control plane node ``` curl -sfL https://github.com/loft-sh/vcluster/releases/download/v0.37.0/install-standalone.sh | sh -s -- \ --vcluster-name standalone \ --join-token your-token \ --join-endpoint https://your-vcluster-endpoint ``` The node automatically downloads the necessary binaries and configuration, and joins as an additional control plane node. ## Add worker nodes[​](#add-worker-nodes "Direct link to Add worker nodes") After the vCluster control plane is up and running, add [dedicated worker nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md). The API server endpoint must be reachable from the worker nodes. Configure `controlPlane.endpoint` and `controlPlane.proxy.extraSANs` in your vCluster configuration to expose the API server. --- # Manage vCluster Standalone Control Plane Nodes Supported Configurations Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. ## Managed by vCluster Platform[​](#managed-by-vcluster-platform "Direct link to Managed by vCluster Platform") ### Update configuration[​](#update-configuration "Direct link to Update configuration") When a standalone cluster is connected to [vCluster Platform](https://vcluster.com/docs/platform/.md), configuration updates and version upgrades can be coordinated from the Platform UI with rolling upgrades: 1. Go to `Virtual Clusters` and select the vCluster. 2. Click `Edit`. 3. Change the configuration. 4. Click `Save`. For the full configuration reference, see `config.yaml` in the [standalone configuration reference](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/standalone.md). Review the [standalone limitations](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/troubleshooting.md#unsupported-configuration-options) before changing settings. ### Add and remove nodes[​](#add-and-remove-nodes "Direct link to Add and remove nodes") When a standalone cluster is connected to [vCluster Platform](https://vcluster.com/docs/platform/.md), add or remove nodes by updating the `quantity` in your configuration and saving the changes in the Platform UI. Worker node lifecycle is managed through the Platform UI when [Auto Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/auto-nodes.md) are used, and provisioning is backed by [Node Providers](https://vcluster.com/docs/platform/administer/node-providers/overview.md). Example: scale control plane nodes ``` controlPlane: standalone: autoNodes: quantity: 3 ``` ## Self-managed[​](#self-managed "Direct link to Self-managed") ### Connect to vCluster Platform[​](#connect-to-vcluster-platform "Direct link to Connect to vCluster Platform") Self-managed standalone vCluster can be connected to [vCluster Platform](https://vcluster.com/docs/platform/.md). Once connected, configuration updates and version upgrades can be coordinated from the platform, but this does not make the nodes platform-managed. Use the following command to connect a standalone cluster: Connect a vCluster Standalone to vCluster Platform ``` vcluster platform add standalone my-cluster --project my-project --access-key my-access-key --host https://my-vcluster-platform.com ``` ### Update configuration[​](#update-configuration-1 "Direct link to Update configuration") Changing the `vcluster.yaml` configuration has to be done on all control plane nodes. 1. Modify the `vcluster.yaml`. The `vcluster.yaml` is located at: `/etc/vcluster/vcluster.yaml`. Edit and save your changes. 2. Restart vCluster systemd service to deploy your changes. Restart vCluster ``` systemctl restart vcluster.service ``` ### Add nodes[​](#add-nodes "Direct link to Add nodes") To add control plane nodes to a self-managed standalone cluster, create a join token on an existing control plane node, then run the join command on each new node: 1. Create a control plane join token on an existing control plane node. Create a token for control plane nodes ``` /var/lib/vcluster/bin/vcluster-cli token create --control-plane --expires=1h ``` 2. Run the returned join command on each new control plane node. Join a control plane node ``` curl -sfLk https:///node/join?token= | sh - ``` ### Remove nodes[​](#remove-nodes "Direct link to Remove nodes") If you have not enabled joining the control plane nodes as worker nodes, then removing vCluster requires only stopping the service and removing files from the nodes. For HA clusters, there are additional steps detailed below. For control plane nodes that are joined in the cluster (`controlPlane.standalone.joinNode.enabled: true`), first follow the [worker node removal steps](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/manage.md#remove-worker-node) and [node cleanup](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/manage.md#clean-up-orphaned-node).

* Single Control Plane Node * Multiple Control Plane Nodes (HA) 1. Stop the vCluster service on the control plane node. Stop vCluster service ``` systemctl stop vcluster.service ``` 2. Remove vCluster files on the control plane node. Clean up vCluster files ``` rm -rf /var/lib/vcluster && rm /etc/systemd/system/vcluster.service ``` 1) Save the IP Address of the node you want to remove, it will be needed later. There are multiple ways to check it, but if you're unsure, you can get it from the vcluster logs: Get etcd peer ip ``` journalctl -u vcluster.service | grep etcd | grep "Adding peer" INFO etcd/add.go:48 Adding peer https://10.244.0.28:2380 ``` Stop the vCluster service on the control plane node. Stop vCluster service ``` systemctl stop vcluster.service ``` 2) Remove vCluster files on the control plane node. Clean up vCluster files ``` rm -rf /var/lib/vcluster && rm /etc/systemd/system/vcluster.service ``` 3) On the different node, list existing etcd cluster members: List ETCD members ``` /var/lib/vcluster/bin/etcdctl --endpoints=127.0.0.1:2379 --cert=/var/lib/vcluster/pki/apiserver-etcd-client.crt --key=/var/lib/vcluster/pki/apiserver-etcd-client.key --cacert=/var/lib/vcluster/pki/etcd/ca.crt member list ``` find a peer ID (first column) in the output that matches your removed node IP saved earlier (`https://10.244.0.28:2380`): List ETCD members output ``` 8f2bcb8fdb98bd92, started, 10.244.0.26, https://10.244.0.26:2380, https://10.244.0.26:2379, false d12398ad344dc8cf, started, 10.244.0.29, https://10.244.0.29:2380, https://10.244.0.29:2379, false ee33f76261ecb3b3, started, 10.244.0.28, https://10.244.0.28:2380, https://10.244.0.28:2379, false ``` matching peer ID is `ee33f76261ecb3b3`. then, remove etcd peer: Remove ETCD peer ``` /var/lib/vcluster/bin/etcdctl --endpoints=127.0.0.1:2379 --cert=/var/lib/vcluster/pki/apiserver-etcd-client.crt --key=/var/lib/vcluster/pki/apiserver-etcd-client.key --cacert=/var/lib/vcluster/pki/etcd/ca.crt member remove ee33f76261ecb3b3 ``` ### Reuse nodes[​](#reuse-nodes "Direct link to Reuse nodes") If you want to reuse a node that was already running vCluster standalone, you can do so by resetting the node and running install script again. * Control Plane to Control Plane * Control Plane to Worker Node * Worker Node to Control Plane 1. To reset the node, run install script using `--reset-only` flag. Reset the node ``` curl -sfL https://github.com/loft-sh/vcluster/releases/download/${VCLUSTER_VERSION}/install-standalone.sh | sh -s -- --reset-only ``` You should see the following output: ``` 🧹 Resetting vCluster installation... ✅ Reset complete. ``` 2. Run the install script again. Run the install script again ``` curl -sfL https://github.com/loft-sh/vcluster/releases/download/${VCLUSTER_VERSION}/install-standalone.sh | sh -s -- --vcluster-name my-new-vcluster --config ${PWD}/vcluster.yaml ``` 1) To reset the node, run install script using `--reset-only` flag. Reset the node ``` curl -sfL https://github.com/loft-sh/vcluster/releases/download/${VCLUSTER_VERSION}/install-standalone.sh | sh -s -- --reset-only ``` You should see the following output: Output ``` 🧹 Resetting vCluster installation... ✅ Reset complete. ``` 2) From the primary node for the tenant cluster, create a node join token. Create node join token ``` vcluster token create ``` You should see similar output: Output ``` curl -fsSLk "https://10.244.0.16:8443/node/join?token=hzndlx.vhf08ob7pg97r2hb" | sh - ``` 3) Back on your node, in order to join you'll need to add `--force-join` flag to the command. Join the node ``` curl -fsSLk "https://10.244.0.16:8443/node/join?token=hzndlx.vhf08ob7pg97r2hb" | sh -s -- --force-join ``` Run it on the node you want to join. You should see output similar to this: Output ``` Detected OS: ubuntu Preparing node for Kubernetes installation... Kubernetes version: v1.36.0 Installing Kubernetes binaries... Downloading Kubernetes binaries from https://github.com/loft-sh/kubernetes/releases/download... Resetting node... Ensuring kubelet is stopped... kubelet service not found Starting containerd... Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /etc/systemd/system/containerd.service. Starting kubelet... Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /etc/systemd/system/kubelet.service. Installation successful! Joining node into cluster... [preflight] Running pre-flight checks [preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"... [preflight] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it. [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s [kubelet-check] The kubelet is healthy after 1.003203387s [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ``` Your node should now be joined to the cluster. 1. Follow steps in [Worker Nodes Documentation](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/manage.md#reuse-a-node) to reset the node. 2. Proceed with installing vCluster standalone. Run the install script again ``` curl -sfL https://github.com/loft-sh/vcluster/releases/download/${VCLUSTER_VERSION}/install-standalone.sh | sh -s -- --vcluster-name my-new-vcluster --config ${PWD}/vcluster.yaml ``` --- # Node Requirements Supported Configurations Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. ## Control plane node requirements[​](#control-plane-node-requirements "Direct link to Control plane node requirements") * Network connectivity between any control plane and worker nodes being used for the cluster ## Host requirements[​](#host-requirements "Direct link to Host requirements") * Linux distribution with systemd support (required for service management) * `iptables` binaries installed (automatically available on Ubuntu, may need manual installation on other distributions) * `curl` installed * Root user access * Increased inotify watchers on each node. fs.inotify resource limits (defined by `fs.inotify.max_user_watches` and `fs.inotify.max_user_instances` system variables. For example, in Ubuntu these default to 8192 and 128 respectively, which is not enough.) info On RHEL 8 and 9, SELinux in enforcing mode is supported with the `vcluster-selinux` RPM. `install-standalone.sh` installs it automatically; no extra steps are required. See [SELinux support](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/selinux.md). ### Increase the inotify watchers limit[​](#increase-the-inotify-watchers-limit "Direct link to Increase the inotify watchers limit") Log into each node and manually increase the watchers limit. 1. Log into the Node. Use SSH or your preferred method for node access. 2. Confirm the current inotify watchers limit by running: ``` cat /proc/sys/fs/inotify/max_user_watches ``` 3. Increase the limit temporarily. ``` sudo sysctl fs.inotify.max_user_watches=524288 ``` Adjust the value as needed; 524288 is typically sufficient. 4. Increase the limit permanently. To make the change persistent across reboots, add it to the sysctl configuration: ``` echo "fs.inotify.max_user_watches=524288" | sudo tee -a /etc/sysctl.conf sudo sysctl -p ``` ## Supported operating systems[​](#supported-operating-systems "Direct link to Supported operating systems") | OS | Version | Additional Instructions | | ------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Ubuntu | 24.04 | | | Ubuntu | 22.04 | | | RHEL (Red Hat Enterprise Linux) | 9 | [Install `vcluster-selinux`](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/selinux.md) or disable SELinux | | RHEL (Red Hat Enterprise Linux) | 8 | [Install `vcluster-selinux`](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/selinux.md) and pin Kubernetes to 1.31, or disable SELinux | | RHEL (Red Hat Enterprise Linux) | 10 | Disable SELinux | | CentOS Stream | 9 | Disable SELinux and install `iptables` | | Rocky | 10 | Disable SELinux and install `iptables` | | Rocky | 9 | Disable SELinux and install `iptables` | ### CentOS Stream and Rocky support[​](#centos-stream-and-rocky-support "Direct link to CentOS Stream and Rocky support") Install iptables ``` yum install iptables-services systemctl enable iptables systemctl start iptables ``` --- Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. # Deploy vCluster in an air-gapped environment This document explains how to deploy vCluster in environments without internet access, known as air-gapped environments. important Enterprise features require the vCluster Platform. If you deploy a vCluster in an air-gapped environment and want to use enterprise features, you must also deploy the platform in air-gapped mode and connect the vCluster to it. A standalone air-gapped vCluster does not support any enterprise capabilities. ## Overview[​](#overview "Direct link to Overview") When deploying vCluster Standalone, there are artifacts that are typically accessed using an internet connection, but without access to the internet, these artifacts need to be available to the Kubernetes cluster through a private registry: * **vCluster Standalone Binary**—Typically retrieved from the [vCluster releases](https://github.com/loft-sh/vcluster/releases) as `vcluster-linux--standalone`. * **vCluster Standalone Install Script**—Typically retrieved from the [vCluster releases](https://github.com/loft-sh/vcluster/releases) as `install-standalone.sh`. * **Kubernetes Control-Plane image**—Pulled from `loft-sh/kubernetes` registry. After ensuring that vCluster Standalone binary and install script are present on the server where you want to install vCluster Standalone, and uploading the required images to your private registry, create the `vcluster.yaml` configuration file and follow the installation steps. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Access to a node that satisfies the [node requirements](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/node-requirements.md) ### Registry requirements[​](#registry-requirements "Direct link to Registry requirements") * OCI-compliant private registry - A private registry accessible to both the node where you install vCluster Standalone and a separate, internet-connected machine. * Ability to push images to your private registry. ### Deployment requirements[​](#deployment-requirements "Direct link to Deployment requirements") Ensure you have the following: * Access to pull the images from the private registry * On the internet-connected machine for populating the registry: * `wget` installed * `docker` installed ## Populate images to a private registry[​](#populate-images-private-registry "Direct link to Populate images to a private registry") Each vCluster release includes multiple assets to help you upload the images to your private registry. * `images-private-nodes.txt` - The required images to run vCluster Standalone, which assumes using the default Kubernetes version. * `images-private-nodes-optional.txt` - An optional set of images needed for worker node daemonsets and agents. * `download-images.sh` - A bash script that quickly iterates over all the images files to pull them and package them into a tarball to a machine that has internet access. * `push-images.sh` - A bash script that takes the tarball generated from the download script to push them to your private registry. ### Pull and push images[​](#pull-push-images "Direct link to Pull and push images") Modify the following with your specific values to replace on the whole page and generate copyable commands: VCLUSTER\_VERSION0.33.1 REGISTRYecr.io/myteam 1. Download the assets from the [vCluster GitHub release](https://github.com/loft-sh/vcluster/releases) and make the scripts executable. note The `images-private-nodes.txt` contains all required images for the default Kubernetes version. ``` wget https://github.com/loft-sh/vcluster/releases/download/v0.33.1/images-private-nodes.txt wget https://github.com/loft-sh/vcluster/releases/download/v0.33.1/download-images.sh wget https://github.com/loft-sh/vcluster/releases/download/v0.33.1/push-images.sh chmod +x ./download-images.sh chmod +x ./push-images.sh ``` 2. Run `download-images.sh` to pull all images and create a tarball of the images. Review the output to confirm all images were pulled successfully and packaged in the tarball. Download and package images ``` ./download-images.sh --image-list images-private-nodes.txt ``` Multi-platform support If you need to support multiple CPU architectures (e.g., AMD64 and ARM64), use the `--platforms` flag to specify which platforms to download. This is useful when your cluster has nodes with different architectures. Download images for multiple platforms ``` ./download-images.sh --platforms "linux/amd64,linux/arm64" --image-list images-private-nodes.txt ``` 3. Run `push-images.sh` to upload all required images to your private registry. When pushing images into your private registry, the public private registry is removed and only the repository and image name are pushed. This allows vCluster to set your private registry to use for all images used in deploying vCluster. Push images to private registry ``` ./push-images.sh --registry ecr.io/myteam --image-list images-private-nodes.txt ``` Additional push options **Multi-platform support**: If you downloaded images for multiple platforms, specify the same `--platforms` flag when pushing: ``` ./push-images.sh --registry ecr.io/myteam --image-list images-private-nodes.txt --platforms "linux/amd64,linux/arm64" ``` **Insecure registry**: If your private registry uses HTTP instead of HTTPS, add the `--insecure` flag: ``` ./push-images.sh --registry ecr.io/myteam --image-list images-private-nodes.txt --insecure ``` 4. **Optional**: By default, vCluster Standalone uses Flannel for CNI and Local Path Provisioner. To download and push these, download the `images-private-nodes-optional.txt` file. The `images-private-nodes-optional.txt` includes images for various optional configurations. You can edit the file and keep only the images you need, removing those for unused features or versions. note The `images-private-nodes-optional.txt` may include multiple Kubernetes distributions, versions, or other feature-dependent images. You can edit the file to retain only what's necessary for your deployment. Pull, package, and push all optional images ``` wget https://github.com/loft-sh/vcluster/releases/download/v0.33.1/images-private-nodes-optional.txt ./download-images.sh --image-list images-private-nodes-optional.txt --images vcluster-images-optional.tar.gz ./push-images.sh --registry ecr.io/myteam --image-list images-private-nodes-optional.txt ``` tip You can also use the `--platforms` and `--insecure` flags with these commands if needed, as described in the previous steps. ## Configure vCluster[​](#configure-vCluster "Direct link to Configure vCluster") The vcluster.yaml file contains all configuration settings for your vCluster deployment. Use a private registry without credentials Set the default private registry that doesn't have authentication. Authenticated private registries are not supported in this version. Setting the private registry ``` controlPlane: advanced: defaultImageRegistry: ecr.io/myteam ``` Before deploying, it's recommended to review the set of configuration options that cannot be updated post deployment. [These options](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md#control-plane-options) require deploying a new vCluster instead of upgrading your vCluster with new options. ## Deploy vCluster[​](#deploy-standalone "Direct link to Deploy vCluster") After your private registry is populated with the required images, you can proceed with installing the vCluster Standalone. ### Installation[​](#installation "Direct link to Installation") 1. Review your `/etc/vcluster/vcluster.yaml` file. Create vcluster.yaml configuration ``` controlPlane: advanced: defaultImageRegistry: ecr.io/myteam # default vCluster Standalone configuration standalone: enabled: true joinNode: enabled: true # Optional: Control Plane node will also be considered a worker node privateNodes: # Required for adding additional worker nodes enabled: true ``` 2. Save the path to the vCluster Standalone Binary and the install script: Modify the following with your specific values to generate a copyable command: INSTALL SCRIPT PATH/tmp/install-standalone.sh BINARY\_PATH/tmp/vcluster ``` /tmp/install-standalone.sh --vcluster-name standalone --binary /tmp/vcluster --skip-download ``` For further steps follow [the primary guide](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) --- # FIPS 140-2 enablement Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. The [National Institute of Standards and Technology (NIST)](https://www.nist.gov/federal-information-processing-standards-fips) develops Federal Information Processing Standards (FIPS) to ensure the security and interoperability of computer systems used by the U.S. government. FIPS 140-2 is a U.S. Federal Government security standard used to approve cryptographic modules. This document explains how vCluster Standalone and all its components are built with FIPS-validated cryptographic libraries. ## Use of FIPS-compatible Go toolchain[​](#use-of-fips-compatible-go-toolchain "Direct link to Use of FIPS-compatible Go toolchain") vCluster is written in [Go](https://go.dev/), and the FIPS-compliant builds are compiled using the `GOFIPS140=v1.0.0` environment variable. The FIPS 140-3 validated Go Cryptographic Module now underlies Go’s built-in crypto libraries, starting with the Go Cryptographic Module v1.0.0 that is included in Go 1.24. ### FIPS support in vCluster components[​](#fips-support-in-vcluster-components "Direct link to FIPS support in vCluster components") Most components used in vCluster are statically compiled with the GOFIPS140=v1.0.0 enabled Go compiler. From a component perspective, vCluster includes multiple sub-components it depends on. The list below contains components built in a FIPS-compliant manner: Kubernetes components sourced from [Kubernetes image with -full-fips tag suffix](https://github.com/loft-sh/kubernetes/pkgs/container/kubernetes): * containerd and containerd-shim * cni network plugins * kubeadm * kubectl * kubelet * kube-apiserver * kube-controller-manager * kube-scheduler * runc
Other components: * [Kine](https://github.com/loft-sh/fips-builds/pkgs/container/kine-fips) * [ETCD](https://github.com/loft-sh/fips-builds/pkgs/container/etcd-fips) * [Konnectivity Server](https://github.com/loft-sh/fips-builds/pkgs/container/konnectivity-server-fips)
### FIPS vCluster binaries[​](#fips-vcluster-binaries "Direct link to FIPS vCluster binaries") The vCluster Standalone FIPS-compliant binaries can be found in the vCluster [GitHub Releases](https://github.com/loft-sh/vcluster/releases). ## Run a FIPS-compliant vCluster[​](#run-a-fips-compliant-vcluster "Direct link to Run a FIPS-compliant vCluster") To run vCluster in a FIPS environment, you must pass `--fips` flag to install script. The following is an example of a `vcluster.yaml` file that one can use to create a FIPS-compliant vCluster instance: ``` controlPlane: distro: k8s: version: v1.34.2 # pins kubernetes components version backingStore: etcd: embedded: enabled: true # The use of embedded etcd is recommended, yet optional standalone: enabled: true privateNodes: enabled: true ``` To create the FIPS-compliant vCluster Standalone, save this config to `/etc/vcluster/vcluster.yaml` on the node and then run: Install FIPS-compliant vCluster Standalone on control plane node ``` curl -sfL https://github.com/loft-sh/vcluster/releases/download/v0.37.0/install-standalone.sh | sh -s -- --vcluster-name standalone --fips ``` As a result, vCluster Standalone will automatically use FIPS-compliant images and artifacts for vCluster Standalone and Kubernetes Components. info To use a different Kubernetes version in your tenant cluster, set the `controlPlane.distro.k8s.version` field in your configuration: ``` controlPlane: distro: k8s: image: tag: v1.34.2 ``` --- # Troubleshooting vCluster Standalone Control Plane Nodes Supported Configurations Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. Bringing up a Kubernetes cluster can face challenges, so here are some troubleshooting tips to help get you started. ## Check vCluster logs[​](#check-vcluster-logs "Direct link to Check vCluster logs") All vCluster logs are located in the control plane, view service logs using journalctl. View vCluster service logs ``` journalctl -u vcluster.service --since="2 minutes ago" -f ``` ## Common Issues[​](#common-issues "Direct link to Common Issues") ### Network connectivity[​](#network-connectivity "Direct link to Network connectivity") Ensure ports 6443 (API Server) and other required ports are accessible ### The systemd service[​](#the-systemd-service "Direct link to The systemd service") The vCluster service needs to be running at all times. ``` systemctl status vcluster.service ``` ### Node join failures[​](#node-join-failures "Direct link to Node join failures") Check that join tokens haven't expired and network connectivity exists between nodes. When running the join node script, if a 400 status code is received, check the URL directly to see if an error message exists. Alternatively you can pipe the entire response to the terminal and print it: ``` curl -sSLk "https:///node/join?token=" | { response=$(cat); echo "$response" | sh - 2>/dev/null || echo "Error: $response"; } ``` ### Resource constraints[​](#resource-constraints "Direct link to Resource constraints") Ensure adequate CPU, memory, and disk space on nodes. ## Unsupported configuration options[​](#unsupported-configuration-options "Direct link to Unsupported configuration options") ### Other vCluster feature limitations[​](#other-vcluster-feature-limitations "Direct link to Other vCluster feature limitations") Certain vCluster features are automatically disabled or unavailable. If you include these options in your `vcluster.yaml`, they are ignored or might cause configuration errors. The following features are not available: * `sync.*` - No resource syncing between tenant and control plane clusters * `integrations.*` - Integrations depend on syncing functionality * `networking.replicateServices` - Services are not replicated to host * `controlPlane.coredns.embedded: true` - Embedded CoreDNS conflicts with custom CNI * `controlPlane.distro.k8s.scheduler.enabled: false` - Virtual scheduler cannot be disabled * `sleep.*` - No ability to use auto sleep for workloads or control plane Unsupported vcluster.yaml options with private nodes ``` # These configurations are NOT supported with private nodes # Resource syncing between tenant and control plane clusters is disabled sync: toHost: services: enabled: false # Services cannot be synced to the control plane cluster secrets: enabled: false # Secrets cannot be synced to the control plane cluster # All other sync options (pods, configmaps, etc.) are also disabled # Platform integrations require syncing functionality integrations: metricsServer: enabled: false # Metrics server integration not supported # All other integrations are disabled due to sync dependency controlPlane: # DNS configuration limitations coredns: embedded: false # Embedded CoreDNS conflicts with custom CNI options distro: k8s: # Virtual scheduler is required for workload placement scheduler: enabled: true # Always enabled (cannot be disabled) # Host Path Mapper is not supported hostPathMapper: enabled: false # Service replication (networking.replicateServices) and sleep mode # (sleep.*) are also unavailable; see the list above. ``` --- # Scale CoreDNS for large node counts CoreDNS handles DNS resolution for every pod in the tenant cluster. By default, it runs a single replica (`controlPlane.coredns.deployment.replicas: 1`) as a separate Deployment, sized for small and medium tenant clusters. High DNS query volume, most often from AI and ML workloads with bursty pod scheduling, can push demand past what one replica handles. For why that happens, see [Why does CoreDNS become a DNS bottleneck?](https://vcluster.com/docs/vcluster/understand/coredns-bottlenecks.md). See [CoreDNS configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/coredns.md) for the full deployment and configuration reference. Already seeing DNS timeouts or `SERVFAIL` errors right now? See [Resolve CoreDNS DNS timeouts and SERVFAIL errors](https://vcluster.com/docs/vcluster/troubleshoot/coredns-dns-timeouts.md) instead. Doesn't apply to embedded CoreDNS This page covers the separate CoreDNS deployment type (`controlPlane.coredns.embedded: false`, the default). Neither approach below works with `controlPlane.coredns.embedded: true`: there's no `coredns` Deployment to scale, and `controlPlane.coredns.deployment.replicas` is silently ignored. See [Why embedded CoreDNS can't scale the same way](https://vcluster.com/docs/vcluster/understand/coredns-bottlenecks.md#why-embedded-coredns-cant-scale-the-same-way) for why, and for the options available instead. ## Scale CoreDNS[​](#scale-coredns "Direct link to Scale CoreDNS") Scale CoreDNS by either increasing the replica count or by running an autoscaler. Which one to reach for depends on how stable your node count is: * **Fixed or slow-changing node count** — increase the replica count directly. It's simpler to reason about and doesn't add an extra component to operate. * **Node count that changes over time**, such as autoscaling GPU node pools — run a node-proportional autoscaler so replica count tracks it automatically. ### Increase the replica count[​](#increase-the-replica-count "Direct link to Increase the replica count") For a fixed increase that doesn't need to track node count, raise `replicas` directly: vcluster.yaml — CoreDNS replica count ``` controlPlane: coredns: deployment: replicas: 3 ``` vCluster re-applies the CoreDNS Deployment every time the control plane pod starts, reconciling it against `vcluster.yaml` the same way `kubectl apply` reconciles a manifest against a live object. `replicas`, and any other CoreDNS setting, converges to whatever `vcluster.yaml` specifies rather than to a value set by a direct edit. Set `replicas` in `vcluster.yaml` to the floor you want instead of scaling the Deployment directly. See [Configuration Persistence](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/coredns.md#deploy-separate-coredns) for details. ### Run a node-proportional autoscaler[​](#run-a-node-proportional-autoscaler "Direct link to Run a node-proportional autoscaler") For tenant clusters where node count changes over time, such as autoscaling GPU node pools, keep the CoreDNS replica count proportional to node count. Don't rely on a fixed value. Start from upstream CoreDNS's own [cluster-proportional-autoscaler](https://github.com/kubernetes-sigs/cluster-proportional-autoscaler) convention of one replica per 16 nodes, then tighten it against your own measured query volume. One deployment on a small number of very large bare-metal GPU nodes tightened its ratio to 1:8. Its per-node core count was high enough that core-based scaling alone under-provisioned. The `coredns` Deployment in `kube-system` is a native object inside the tenant cluster, not a synced mirror of something on the control plane cluster. Run the autoscaler directly against the tenant cluster's own API server. It watches the tenant cluster's node count and scales `coredns` directly. The tenant cluster's own controller manager reconciles the change, and the resulting pods sync to the control plane cluster automatically. Apply the following to the tenant cluster, not the control plane cluster: coredns-autoscaler.yaml — apply inside the tenant cluster ``` apiVersion: v1 kind: ServiceAccount metadata: name: coredns-autoscaler namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: coredns-autoscaler-nodes rules: - apiGroups: [""] resources: ["nodes"] verbs: ["list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: coredns-autoscaler-nodes roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: coredns-autoscaler-nodes subjects: - kind: ServiceAccount name: coredns-autoscaler namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: coredns-autoscaler namespace: kube-system rules: - apiGroups: ["apps"] resources: ["deployments/scale"] resourceNames: ["coredns"] verbs: ["get", "update"] - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "create"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: coredns-autoscaler namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: coredns-autoscaler subjects: - kind: ServiceAccount name: coredns-autoscaler namespace: kube-system --- apiVersion: v1 kind: ConfigMap metadata: name: coredns-autoscaler namespace: kube-system data: linear: |- {"nodesPerReplica": 16, "min": 1, "preventSinglePointFailure": true} --- apiVersion: apps/v1 kind: Deployment metadata: name: coredns-autoscaler namespace: kube-system spec: replicas: 1 selector: matchLabels: k8s-app: coredns-autoscaler template: metadata: labels: k8s-app: coredns-autoscaler spec: serviceAccountName: coredns-autoscaler containers: - name: autoscaler # check the project's releases page for the current tag image: registry.k8s.io/cpa/cluster-proportional-autoscaler:v1.10.3 command: - /cluster-proportional-autoscaler - --namespace=kube-system - --configmap=coredns-autoscaler - --target=deployment/coredns - --logtostderr=true - --v=2 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true seccompProfile: type: RuntimeDefault ``` The `linear` mode config's `nodesPerReplica: 16` reproduces upstream's node-based convention. Tighten it, for example to 8, once your own measured query volume tells you a node needs more headroom than that. ### Settings to avoid[​](#settings-to-avoid "Direct link to Settings to avoid") Upstream offers a `coresPerReplica` companion ratio (256 cores per replica). On shared nodes, avoid it unless [`sync.fromHost.nodes.enabled`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md) is `true`. With the default `sync.fromHost.nodes.enabled: false`, the tenant cluster's API server only synthesizes Node objects for nodes actually running tenant pods. Those synthesized nodes report a fixed 16 CPU allocatable regardless of the real node's CPU count, which breaks core-based scaling. That same default also means the autoscaler's node count lags real fleet size. A node the physical cluster just added doesn't show up in the tenant cluster until a tenant pod actually lands on it. On [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md), this caveat doesn't apply. Private-node kubelets register real Node objects directly, with real CPU capacity, so `coresPerReplica` can be an appropriate ratio and node discovery doesn't lag pod placement. `preventSinglePointFailure: true` doesn't override `min`. It raises the node-derived replica count to 2 once the tenant cluster has more than one schedulable Ready node. It can also override `max` if `max` is set below 2. Below that threshold, or with a single schedulable node, `min` still governs. ## What to watch[​](#what-to-watch "Direct link to What to watch") Monitor CoreDNS alongside the [control-plane conditions](https://vcluster.com/docs/vcluster/deploy/control-plane/sizing-and-performance.md#operational-guidance), using CoreDNS's own Prometheus metrics ([CoreDNS metrics reference](https://coredns.io/plugins/metrics/)). vCluster's control-plane load-testing framework doesn't model DNS load today. Its existing KWOK and kube-burner workloads don't generate DNS traffic, so there's no vCluster-published SLO threshold for these signals yet. Establish your own baseline against your fleet's real query volume and pod density instead of borrowing a threshold from a workload that isn't yours. ### Request latency (p99)[​](#request-latency-p99 "Direct link to Request latency (p99)") ``` histogram_quantile(0.99, sum(rate(coredns_dns_request_duration_seconds_bucket[5m])) by (le) ) ``` **Why:** Tail DNS resolution latency over a 5-minute rate window. Validating a real boundary needs a sustained query rate matching your target pod density and churn. ### Error rate[​](#error-rate "Direct link to Error rate") ``` sum(rate(coredns_dns_responses_total{rcode="SERVFAIL"}[5m])) ``` **Why:** `SERVFAIL` over a 5-minute window is the reliable overload signal, unlike `NXDOMAIN`. See [What an overloaded CoreDNS looks like](https://vcluster.com/docs/vcluster/understand/coredns-bottlenecks.md#what-an-overloaded-coredns-looks-like) for why `NXDOMAIN` alone isn't reliable. Validating a real boundary needs induced upstream or API server latency to trigger genuine failures, not just query volume. ### Request throughput[​](#request-throughput "Direct link to Request throughput") ``` sum(rate(coredns_dns_requests_total[5m])) ``` **Why:** Total query rate over a 5-minute window. This is informational for capacity planning, not an alerting signal on its own. ### Replica CPU saturation[​](#replica-cpu-saturation "Direct link to Replica CPU saturation") **Why:** CoreDNS container CPU usage against the default 1 CPU limit signals when a replica is approaching the ceiling described in [Why does CoreDNS become a DNS bottleneck?](https://vcluster.com/docs/vcluster/understand/coredns-bottlenecks.md). Your existing container CPU dashboards likely already have a query for this; validating a real boundary needs the same dedicated DNS workload as the other signals above. ### Replica memory saturation[​](#replica-memory-saturation "Direct link to Replica memory saturation") **Why:** CoreDNS's default memory limit is 170Mi per replica, independent of the CPU limit and independent of replica count. An OOMKilled CoreDNS pod produces the same symptoms as a CPU-saturated one, DNS timeouts and `SERVFAIL`. Adding replicas doesn't help. Each new replica gets the same 170Mi ceiling, not a larger share of memory. Check container restart reason before attributing a DNS incident to query volume. For control-plane sizing and alert thresholds unrelated to DNS, see [Sizing and performance](https://vcluster.com/docs/vcluster/deploy/control-plane/sizing-and-performance.md). --- # Deploy vind with restricted network access Modify the following with your specific values to replace on the whole page and generate copyable commands: REGISTRYyour-registry.internal PROXY\_URLhttp\://proxy.internal:3128 NO\_PROXYlocalhost,127.0.0.1,your-registry.internal vind requires outbound HTTPS access during tenant cluster creation. Use a forward proxy and an internal registry to run vind in a restricted network. Fully disconnected environments aren't supported vind can't create a tenant cluster from an empty local cache without outbound access. The CLI downloads artifacts from GitHub and `ghcr.io` through hardcoded URLs. A custom vm-container image doesn't bypass these downloads. Use a forward proxy that can reach the public endpoints listed on this page. ## How vind uses the network[​](#how-vind-uses-the-network "Direct link to How vind uses the network") | When | What vind contacts | How to provide access | | ------------------------------------ | ------------------------------------------- | ------------------------------------------------------- | | Host pre-creation checks | `alpine` on Docker Hub | Pre-pull and tag the image on the Docker host | | CLI artifact staging | `ghcr.io/loft-sh/vcluster-pro:` | Configure the forward proxy on the host | | CLI artifact staging | `ghcr.io/loft-sh/kubernetes:-full` | Configure the forward proxy on the host | | vm-container creation | `ghcr.io/loft-sh/vm-container` | Mirror the image and set `experimental.docker.image` | | Inside the vm-container at bootstrap | `install-standalone.sh` on GitHub | Configure the forward proxy inside the vm-container | | Tenant cluster startup | Internal vCluster images | Set `controlPlane.advanced.defaultImageRegistry` | | Workload startup | Images referenced by workloads | Pre-load images into the host Docker store | | StatefulSet pod startup | The rewriteHosts init container image | Set `sync.toHost.pods.rewriteHosts.initContainer.image` | The proxy must support the HTTP `CONNECT` method for HTTPS traffic. A static file server, including `python3 -m http.server`, can't act as this proxy. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Docker installed and running with access to your internal registry * vCluster CLI installed — see [Deployment basics](https://vcluster.com/docs/vcluster/deploy/control-plane/docker-container/basics.md#step-1-install-vcluster-cli) * A forward proxy that can reach `github.com` and `ghcr.io` * The proxy is reachable from both the Docker host and the vind vm-container * Required images mirrored to your internal registry (`images.txt` is listed under [vCluster release assets](https://github.com/loft-sh/vcluster/releases)) ## Step 1: Configure the host proxy[​](#step-1-configure-the-host-proxy "Direct link to Step 1: Configure the host proxy") The `vcluster` CLI pulls two artifacts directly from `ghcr.io`. These downloads happen on the host before the CLI starts the vm-container: * `ghcr.io/loft-sh/vcluster-pro:` provides the vCluster binary. * `ghcr.io/loft-sh/kubernetes:-full` provides the Kubernetes binaries and bundle. Set the proxy environment variables in the shell where you run `vcluster create`: ``` export HTTP_PROXY=http://proxy.internal:3128 export HTTPS_PROXY=http://proxy.internal:3128 export NO_PROXY=localhost,127.0.0.1,your-registry.internal ``` `controlPlane.advanced.defaultImageRegistry` doesn't change these two artifact sources. The host proxy must allow access to `ghcr.io`. ## Step 2: Pre-pull Alpine on the Docker host[​](#step-2-pre-pull-alpine-on-the-docker-host "Direct link to Step 2: Pre-pull Alpine on the Docker host") Before `vcluster create` runs, vind pulls `alpine` directly on the host for pre-creation checks including port availability, containerd socket detection, and network reachability. There is no config flag to override this image name. Pull Alpine from your internal registry and retag it so vind finds it under the expected name: ``` docker pull your-registry.internal/library/alpine:latest docker tag your-registry.internal/library/alpine:latest alpine:latest ``` vind's registry proxy is enabled by default. It serves this image from the host Docker cache for subsequent pulls inside the tenant cluster. ## Step 3: Configure the vm-container and bootstrap proxy[​](#step-3-configure-the-vm-container-and-bootstrap-proxy "Direct link to Step 3: Configure the vm-container and bootstrap proxy") Mirror `ghcr.io/loft-sh/vm-container` to your internal registry. Then configure vind to use the mirrored image and pass the proxy settings into the container: ``` experimental: docker: # Pull the vm-container image from the internal registry. image: your-registry.internal/loft-sh/vm-container:latest # Route the bootstrap script download through the forward proxy. env: - HTTP_PROXY=http://proxy.internal:3128 - HTTPS_PROXY=http://proxy.internal:3128 - NO_PROXY=localhost,127.0.0.1,your-registry.internal ``` After the vm-container starts, the CLI downloads `install-standalone.sh` from GitHub. The URL is hardcoded, so the proxy must allow access to `github.com`. If the proxy runs on the Docker host, bind it to an interface reachable from the Docker network. Don't bind it only to loopback (`127.0.0.1`). ## Step 4: Redirect internal vCluster images[​](#step-4-redirect-internal-vcluster-images "Direct link to Step 4: Redirect internal vCluster images") `controlPlane.advanced.defaultImageRegistry` prepends a registry prefix to all images vCluster deploys internally, including the Kubernetes control plane, CoreDNS, syncer, etcd, and backing store images. It does not affect workload images. Mirror all images listed in `images.txt` from the [vCluster release assets](https://github.com/loft-sh/vcluster/releases) into your registry at the same repository path, then add the following to your `vcluster.yaml`: ``` # Pull images deployed by vCluster from the internal registry. controlPlane: advanced: defaultImageRegistry: your-registry.internal/vcluster/ ``` note Steps 5 and 6 aren't required for a basic tenant cluster. Complete them when your workloads need the described images. ## Step 5: Make workload images available through the registry proxy[​](#step-5-make-workload-images-available-through-the-registry-proxy "Direct link to Step 5: Make workload images available through the registry proxy") Complete this step if your workloads reference images from public registries such as `docker.io` or `ghcr.io`. vind enables a registry proxy by default. The tenant cluster pulls workload images from the host Docker daemon's image store through this proxy. Make each workload image available on the host under the name used in your manifests. The registry proxy then serves it without contacting the public registry. This is the same pattern used for Alpine in Step 2. Requirement The registry proxy serves local images only when the host Docker daemon uses the containerd image store. Verify with `docker info`, which should report `driver-type io.containerd.snapshotter.v1`. See the [Docker containerd image store guide](https://docs.docker.com/engine/storage/containerd/) to enable it. For each image your workloads reference, pull it from your internal registry and tag it to the public name used in your manifests: ``` docker pull your-registry.internal/library/nginx:1.27 docker tag your-registry.internal/library/nginx:1.27 nginx:1.27 ``` Repeat for every workload image, including any `ghcr.io` images. Tag each image with the exact name used in your manifests. The registry proxy serves the images from the host store for pulls inside the tenant cluster. ## Step 6: Override the rewriteHosts init container image[​](#step-6-override-the-rewritehosts-init-container-image "Direct link to Step 6: Override the rewriteHosts init container image") Complete this step if you run StatefulSets inside the tenant cluster. vCluster pulls `mirror.gcr.io/library/alpine:3.20` to rewrite fully qualified domain names for StatefulSet pods. Add the following to your `vcluster.yaml`: ``` # Pull the rewriteHosts init container from the internal registry. sync: toHost: pods: rewriteHosts: initContainer: image: registry: your-registry.internal repository: library/alpine tag: "3.20" ``` The `image` field takes `registry`, `repository`, and `tag` sub-fields, not a plain string. This format was introduced in v0.27.0. ## Step 7: Create the tenant cluster[​](#step-7-create-the-tenant-cluster "Direct link to Step 7: Create the tenant cluster") After configuring network access and images, switch to the Docker driver and create the tenant cluster: ``` vcluster use driver docker vcluster create my-cluster --values vcluster.yaml ``` For next steps, see the [Docker (vind) Quick Start](https://vcluster.com/docs/vcluster/quick-start/docker.md) to verify your cluster is working and explore pause/resume, LoadBalancer services, and worker nodes. ## Troubleshoot[​](#troubleshoot "Direct link to Troubleshoot") ### Bootstrap fails with exit status 6[​](#bootstrap-fails-with-exit-status-6 "Direct link to Bootstrap fails with exit status 6") ``` fatal failed to start vCluster standalone: exit status 6 ``` This error indicates that the vm-container couldn't reach GitHub. Confirm that `experimental.docker.env` contains the proxy variables and that the proxy is reachable from the Docker network. The proxy must support HTTPS connections through the HTTP `CONNECT` method. A static file server returns `501 Unsupported method ('CONNECT')` and can't serve this request. ### Cluster creation times out due to unreachable kubectl context[​](#cluster-creation-times-out-due-to-unreachable-kubectl-context "Direct link to Cluster creation times out due to unreachable kubectl context") ``` fatal Get "https://:6443/api/v1/namespaces/vcluster-my-cluster": dial tcp :6443: i/o timeout ``` The vCluster CLI checks the active kubectl context at startup. If that context points to an unreachable cluster (for example, a stopped Multipass VM), the command times out before Docker driver mode is entered. Check your current context and switch to one that is reachable: ``` kubectl config get-contexts kubectl config use-context ``` If the correct context is not listed, your shell may have a `KUBECONFIG` environment variable pointing to a separate kubeconfig file. Unset it to fall back to `~/.kube/config`: ``` unset KUBECONFIG kubectl config get-contexts kubectl config use-context ``` ### Enterprise features error when not logged into Platform[​](#enterprise-features-error-when-not-logged-into-platform "Direct link to Enterprise features error when not logged into Platform") ``` fatal you have vCluster pro features enabled, but seems like you are not logged in ... Please make sure to log into vCluster Platform to use vCluster pro features or run this command with --add=false ``` This occurs when the CLI has a vCluster Platform connection configured but you are not actively logged in. Pass `--add=false` to create the cluster without registering it with Platform: ``` vcluster create my-cluster --values vcluster.yaml --add=false ``` ### Permission denied creating staging directory[​](#permission-denied-creating-staging-directory "Direct link to Permission denied creating staging directory") ``` fatal create staging directory: mkdir /Users//.vcluster/docker/vcluster/.downloading: permission denied ``` A previous `sudo vcluster create` run left files in `~/.vcluster/docker/` owned by root. Fix the ownership and retry: ``` sudo chown -R $(whoami) ~/.vcluster/ vcluster create my-cluster --values vcluster.yaml ``` ## Example complete vcluster.yaml[​](#example-complete-vclusteryaml "Direct link to Example complete vcluster.yaml") The following example combines the configurable layers: ``` # Pull images deployed by vCluster from the internal registry. controlPlane: advanced: defaultImageRegistry: your-registry.internal/vcluster/ experimental: docker: # Pull the vm-container image from the internal registry. image: your-registry.internal/loft-sh/vm-container:latest # Route the bootstrap script download through the forward proxy. env: - HTTP_PROXY=http://proxy.internal:3128 - HTTPS_PROXY=http://proxy.internal:3128 - NO_PROXY=localhost,127.0.0.1,your-registry.internal # Pull the rewriteHosts init container from the internal registry. sync: toHost: pods: rewriteHosts: initContainer: image: registry: your-registry.internal repository: library/alpine tag: "3.20" ``` ## Known limitations[​](#known-limitations "Direct link to Known limitations") * vind doesn't support tenant cluster creation from an empty cache without outbound access. * The CLI doesn't provide registry overrides for the vCluster and Kubernetes artifacts that it stages on the host. * The CLI doesn't provide a local path or URL override for `install-standalone.sh`. * Baking `install-standalone.sh` into a custom vm-container image doesn't prevent the CLI from downloading the script from GitHub. --- # Deployment basics vCluster in Docker (vind) allows you to deploy a complete Kubernetes cluster using Docker containers without requiring a host Kubernetes cluster. This deployment method is ideal for local development, testing, and CI/CD scenarios. ## Overview[​](#overview "Direct link to Overview") When deploying vCluster as Docker containers: * **Control plane**: Runs as Docker containers (API server, etcd, controller manager, scheduler) * **Worker nodes**: Simulated as additional Docker containers * **Networking**: Uses Docker networking for inter-container communication * **Storage**: Uses Docker volumes for persistent data * **No control plane cluster required**: Complete independence from existing Kubernetes infrastructure ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") ### System requirements[​](#system-requirements "Direct link to System requirements") * **Docker**: Docker Desktop, Docker Engine or similar installed and running * **Operating System**: Linux, macOS, or Windows with Docker support * **Resources**: Minimum 2GB RAM and 2 CPU cores available to Docker * **Network**: Docker daemon accessible and able to create networks ### Docker configuration[​](#docker-configuration "Direct link to Docker configuration") Ensure Docker is properly configured: ``` docker --version docker info ``` For optimal performance, consider: * Enabling [containerd image storage](https://docs.docker.com/engine/storage/containerd/) for better registry proxy performance * Allocating sufficient resources in Docker Desktop preferences ## Deploy vCluster[​](#deploy-vcluster "Direct link to Deploy vCluster") ### Step 1: Install vCluster CLI[​](#step-1-install-vcluster-cli "Direct link to Step 1: Install vCluster CLI") * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` ### Step 2: Configure Docker Driver[​](#step-2-configure-docker-driver "Direct link to Step 2: Configure Docker Driver") Set vCluster to use the Docker driver: ``` vcluster use driver docker ``` Restricted networks If outbound access must use a forward proxy, follow the [restricted network setup](https://vcluster.com/docs/vcluster/deploy/control-plane/docker-container/air-gapped.md) guide before proceeding. ### Step 3: Create vCluster configuration (optional)[​](#step-3-create-vcluster-configuration-optional "Direct link to Step 3: Create vCluster configuration (optional)") Create a `vcluster.yaml` file to customize your Docker deployment: ``` experimental: docker: # Expose additional ports ports: - "8080:80" - "8443:443" # Mount host directories volumes: - "/host/data:/data" - "/var/run/docker.sock:/var/run/docker.sock" # Set environment variables env: - "DEBUG=true" - "LOG_LEVEL=info" # Add additional worker nodes nodes: - name: "worker-1" ports: - "9090:9090" - name: "worker-2" volumes: - "/host/logs:/logs" # Configure registry proxy (enabled by default) registryProxy: enabled: true # Configure load balancer (enabled by default) loadBalancer: enabled: true forwardPorts: true ``` ### Step 4: Deploy the vCluster[​](#step-4-deploy-the-vcluster "Direct link to Step 4: Deploy the vCluster") Deploy with default configuration: Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster ``` vcluster create my-vcluster ``` note vCluster deploys Kubernetes **v1.36.0** by default. To use a different version, set `controlPlane.distro.k8s.image.tag` in your `vcluster.yaml`. Or deploy with custom configuration: Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster ``` vcluster create my-vcluster --values vcluster.yaml ``` ### Step 5: Verify Deployment[​](#step-5-verify-deployment "Direct link to Step 5: Verify Deployment") Check that your vCluster is running: ``` kubectl get nodes kubectl get namespaces ``` Verify Docker containers: Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster ``` docker ps --filter name=my-vcluster ``` ## Configuration options[​](#configuration-options "Direct link to Configuration options") ### Container customization[​](#container-customization "Direct link to Container customization") * **Image**: Specify custom container images for control plane and worker nodes * **Ports**: Expose additional ports from containers to the host * **Volumes**: Mount host directories or Docker volumes * **Environment Variables**: Set custom environment variables * **Docker Arguments**: Pass additional arguments to Docker run commands ### Multi-node setup[​](#multi-node-setup "Direct link to Multi-node setup") Configure multiple worker nodes with individual settings: ``` experimental: docker: nodes: - name: "worker-1" image: "custom-registry/worker:latest" ports: - "3000:3000" volumes: - "/host/app:/app" - name: "worker-2" env: - "NODE_ENV=production" args: - "--privileged" ``` ### vCluster VPN[​](#vcluster-vpn "Direct link to vCluster VPN") You can join external nodes to your Docker-based vCluster using the vCluster VPN feature. This requires vCluster Platform to be installed. For detailed setup instructions, see the [vCluster VPN documentation](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/vpn.md). ``` experimental: docker: nodes: - name: my-node privateNodes: vpn: enabled: true nodeToNode: enabled: true ``` ## Troubleshoot[​](#troubleshoot "Direct link to Troubleshoot") ### Common issues[​](#common-issues "Direct link to Common issues") 1. **Docker daemon not accessible**: ``` docker info ``` 2. **Resource constraints**: Ensure Docker has sufficient CPU and memory allocated 3. **Permission issues on macOS**: Load balancer features may require running Docker with elevated privileges. Try running your terminal as an administrator or ensure Docker Desktop has the necessary permissions in System Preferences > Security & Privacy ### Debug[​](#debug "Direct link to Debug") Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster ``` # Check vCluster status vcluster list # Show all docker containers docker ps -a # View control plane logs docker exec -it vcluster.cp.my-vcluster journalctl -u vcluster --no-pager # View kubelet logs docker exec -it vcluster.node.my-vcluster.my-node journalctl -u kubelet --no-pager # View containerd logs docker exec -it vcluster.node.my-vcluster.my-node journalctl -u containerd --no-pager # View vcluster-vpn logs docker exec -it vcluster.node.my-vcluster.my-node journalctl -u vcluster-vpn --no-pager ``` ## Delete a Docker vCluster[​](#delete-a-docker-vcluster "Direct link to Delete a Docker vCluster") To remove a Docker-based vCluster and clean up all associated resources: Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster ``` vcluster delete my-vcluster ``` This command removes all Docker containers and networks created for the vCluster. Any data stored in Docker volumes is also deleted. ## Next steps[​](#next-steps "Direct link to Next steps") * [Configure Docker-specific settings](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/docker.md) * [Worker node options](https://vcluster.com/docs/vcluster/introduction/architecture.md#worker-nodes) * [Explore vCluster features](https://vcluster.com/docs/vcluster/quick-start/docker.md) ## Config reference[​](#config-reference "Direct link to Config reference") ## `docker` object [​](#docker "Direct link to docker")Docker allows you to configure Docker related settings when deploying a vCluster using Docker. ### `image` string [​](#docker-image "Direct link to docker-image")Image defines the image to use for the container. Defaults to ghcr.io/loft-sh/vm-container. ### `ports` string\[] [​](#docker-ports "Direct link to docker-ports")Ports defines extra port mappings to be added to the container. ### `volumes` string\[] [​](#docker-volumes "Direct link to docker-volumes")Volumes defines extra volumes to be added to the container. ### `env` string\[] [​](#docker-env "Direct link to docker-env")Env defines extra environment variables to be added to the container. Use key=value. ### `args` string\[] [​](#docker-args "Direct link to docker-args")Args defines extra arguments to be added to the docker run command of the container. ### `enabled` boolean [​](#docker-enabled "Direct link to docker-enabled")Enabled defines if the vCluster was deployed using Docker. This is automatically set by vCluster and should not be set by the user. ### `network` string [​](#docker-network "Direct link to docker-network")Network defines the network to use for the vCluster. If not specified, the a network will be created for the vCluster. ### `nodes` object\[] [​](#docker-nodes "Direct link to docker-nodes")Nodes defines the nodes of the vCluster. #### `image` string [​](#docker-nodes-image "Direct link to docker-nodes-image")Image defines the image to use for the container. Defaults to ghcr.io/loft-sh/vm-container. #### `ports` string\[] [​](#docker-nodes-ports "Direct link to docker-nodes-ports")Ports defines extra port mappings to be added to the container. #### `volumes` string\[] [​](#docker-nodes-volumes "Direct link to docker-nodes-volumes")Volumes defines extra volumes to be added to the container. #### `env` string\[] [​](#docker-nodes-env "Direct link to docker-nodes-env")Env defines extra environment variables to be added to the container. Use key=value. #### `args` string\[] [​](#docker-nodes-args "Direct link to docker-nodes-args")Args defines extra arguments to be added to the docker run command of the container. #### `name` string [​](#docker-nodes-name "Direct link to docker-nodes-name")Name defines the name of the node. If not specified, a random name will be generated. ### `registryProxy` object [​](#docker-registryProxy "Direct link to docker-registryProxy")Defines if docker images should be pulled from the host docker daemon. This prevents pulling images again and allows to use purely local images. Only works if containerd image storage is used. For more information, see #### `enabled` boolean true [​](#docker-registryProxy-enabled "Direct link to docker-registryProxy-enabled")Enabled defines if this option should be enabled. ### `loadBalancer` object [​](#docker-loadBalancer "Direct link to docker-loadBalancer")Defines if vCluster should configure load balancer services inside the vCluster. This might require sudo access on the host cluster for docker desktop or rancher desktop on macos. #### `enabled` boolean true [​](#docker-loadBalancer-enabled "Direct link to docker-loadBalancer-enabled")Enabled defines if this option should be enabled. #### `forwardPorts` boolean true [​](#docker-loadBalancer-forwardPorts "Direct link to docker-loadBalancer-forwardPorts")ForwardPorts defines if the load balancer ips should be made available locally via port forwarding. This will be only done if necessary for example on macos when using docker desktop. --- # Deployment basics There are multiple ways to deploy and manage your vCluster. Review the different choices and choose the one that suits your needs. ## Predeployment configuration options[​](#predeployment-configuration-options "Direct link to Predeployment configuration options") Before deploying, it's recommended to review the set of configuration options that cannot be updated post deployment. These options require deploying a new vCluster instead of upgrading your vCluster with new options. ### Control Plane Options[​](#control-plane-options "Direct link to Control Plane Options") Decide the various options of how you want your control plane deployed: * **[High availability](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/high-availability.md)** - Run multiple copies of vCluster components. * **[Rootless mode](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/rootless-mode.md)** - Deploy the vCluster pod without root access to the control plane cluster. * **Backing Store** - Decide how the data of your cluster is stored. Backing store options vCluster supports etcd or a relational database (using KINE) as the backend. This feature provides flexibility to vCluster operators. The available data store options allow you to select a data store that fits your use case. vCluster supports the following datastore options: * [Embedded SQLite](https://www.sqlite.org/index.html) (default with `PersistentVolume` (PV)) * [PostgreSQL](https://www.postgresql.org/) * [MySQL](https://www.mysql.com/) * [MariaDB](https://mariadb.org/) * [etcd](https://etcd.io/) warning After deploying your vCluster, there are limited migration paths to change your backing store. Review the backing store migration options before deploying. **Backing store options** * Embedded SQLite (Default) * Embedded SQLite (No PV) * Embedded etcd * Deployed etcd * MySQL / MariaDB * PostgreSQL This is the default, so you don't need to configure anything. If you want to explicitly set this option, you can use: ``` controlPlane: backingStore: database: embedded: enabled: true ``` By default, vCluster stores its data in a `PersistentVolumeClaim` (PVC). Alternatively, you can use an [`emptyDir`](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume to store tenant cluster data. To use an `emptyDir` to store the data instead of a `PersistentVolume`, create a `values.yaml` with the following contents: ``` controlPlane: statefulSet: persistence: volumeClaim: enabled: true ``` Then upgrade or recreate the vCluster with: ``` vcluster create my-vcluster -n my-vcluster --upgrade -f values.yaml ``` Potential data loss This method should only be used for testing purposes, as data is lost upon pod recreation. This is an enterprise feature that allows you to deploy etcd within each vCluster to enable high availability (HA), which isn’t supported with embedded SQLite: ``` controlPlane: backingStore: etcd: embedded: enabled: true ``` This deploys an etcd instance outside of the vCluster control plane pod that is used as a backing store: ``` controlPlane: backingStore: etcd: deploy: enabled: true ``` warning vCluster snapshots aren't supported with this backing store. See [Back up and restore](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md#back-up-and-restore). The option for MySQL and MariaDB typically has the following format: ``` controlPlane: backingStore: database: external: enabled: true dataSource: mysql://username:password@tcp(hostname:3306)/database-name ``` If you specify a database name and it does not exist, the server attempts to create it. The option for PostgreSQL typically has the following format: ``` controlPlane: backingStore: database: external: enabled: true dataSource: postgres://username:password@hostname:port/database-name ``` More advanced configuration parameters are available. For more information, see . If you specify a database name and it does not exist, the server attempts to create it. ### Worker Nodes[​](#worker-nodes "Direct link to Worker Nodes") Decide where you want your worker nodes to come from: * **[Nodes from the control plane cluster](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes.md)** - (Default) All worker nodes of the shared control plane cluster are used by the tenant cluster and all resources are synced to the single namespace that the vCluster is deployed on. * **[Syncing Namespaces](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md)** - Resources are synced to mapped namespaces on the control plane cluster. * **[Isolated workloads](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes/isolated-workloads.md)** - Different options to isolate a workload in a vCluster. * **[Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md)** - Enable adding individual nodes to the tenant cluster. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Prerequisites depend on your deployment model. Pick your path before continuing. * Kubernetes Cluster * Docker (vind) - [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) - [Helm v3.10.0+](https://helm.sh/docs/intro/install/) - Access to a Kubernetes v1.28+ cluster with permissions to deploy applications into a namespace. No cluster-admin or cluster-wide permissions are required. This cluster acts as the Control Plane Cluster for the tenant cluster deployment. * [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) * [Docker](https://docs.docker.com/get-docker/) installed and running No Kubernetes cluster required. Detailed instructions are provided for specific Control Plane Cluster environments: [EKS](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/eks.md), [AKS](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/aks.md), [GKE](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/gke.md), and [OpenShift](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/openshift.md). ## Deploy vCluster[​](#deploy-vcluster "Direct link to Deploy vCluster") YAML configuration If you're not sure which options to configure, you can update most settings later by upgrading your vCluster with an updated `vcluster.yaml`. However, some settings — such as what type of worker nodes or the backing store — can only be set during the initial deployment and cannot be changed during an upgrade. note vCluster deploys Kubernetes **v1.36.0** by default. To use a different version, set `controlPlane.distro.k8s.image.tag` in your `vcluster.yaml`. All of the deployment options below have the following assumptions: * A `vcluster.yaml` is provided. Refer to the `vcluster.yaml` reference docs to explore all configuration options. This file is optional and can be removed from the examples. * The vCluster is called `my-vcluster`. * The vCluster is deployed into the `team-x` namespace.
* vCluster CLI * Helm * Terraform * ArgoCD * Flux * Cluster API The vCluster CLI provides the most straightforward way to deploy and manage virtual clusters. 1. Install the vCluster CLI: * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` 2. Deploy vCluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` vcluster create my-vcluster --namespace team-x --values vcluster.yaml ``` note After installation, vCluster automatically switches your Kubernetes context to the new virtual cluster. You can now run `kubectl` commands against the virtual cluster. Helm provides fine-grained control over the deployment process and integrates well with existing Helm-based workflows. 1. Deploy vCluster using the `helm upgrade` command: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CONFIG FILEvcluster.yaml NAMESPACEteam-x ``` helm upgrade --install my-vcluster vcluster \ --values vcluster.yaml \ --repo https://charts.loft.sh \ --namespace team-x \ --repository-config='' \ --create-namespace ``` You can use Terraform to deploy vCluster as code with version control and state management. 1. Create a `main.tf` file to define your vCluster deployment using the Terraform Helm provider: ``` provider "helm" { kubernetes = { config_path = "~/.kube/config" } } resource "helm_release" "my_vcluster" { name = "my-vcluster" namespace = "team-x" create_namespace = true repository = "https://charts.loft.sh" chart = "vcluster" # If you didn't create a vcluster.yaml, remove the values section. values = [ file("${path.module}/vcluster.yaml") ] } ``` Helm Provider Version This configuration uses the Terraform Helm provider v3.x syntax where `kubernetes` is defined as an argument (`kubernetes = {`). If you're using Helm provider v2.x, use the block syntax instead (`kubernetes {`). To use v3.x, ensure your provider version is at least v3.0.0. 2. Install the required [Helm provider](https://registry.terraform.io/providers/hashicorp/helm/latest) and initialize Terraform: ``` terraform init ``` 3. Generate a plan to preview the changes: ``` terraform plan ```
Review the plan output to verify connectivity and proposed changes. 4. Deploy vCluster: ``` terraform apply ``` ArgoCD deployment enables GitOps workflows for vCluster management, and provides automated deployment, drift detection, and declarative configuration management through Git repositories. To deploy vCluster using ArgoCD, you need the following files: * `vcluster.yaml` for your vCluster configuration options. * `-app.yaml` for your ArgoCD `Application` definition. Replace `` with your actual cluster name.
1. Create the ArgoCD `Application` file `-app.yaml`, which references the vCluster Helm chart: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x ``` --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-vcluster namespace: argocd spec: project: default source: chart: vcluster repoURL: https://charts.loft.sh helm: releaseName: my-vcluster valueFiles: - vcluster.yaml destination: server: https://kubernetes.default.svc namespace: team-x ``` 2. Commit and push these files to your configured ArgoCD repository. 3. Sync your ArgoCD repository with your configured cluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster ``` argocd app sync my-vcluster ``` 1) Create a `HelmRepository` source in the Git repository that Flux monitors, so Flux can fetch the vCluster Helm charts automatically. In this example, the related files are stored under the `clusters` folder. Save the following file as `clusters/sources/vcluster-repository.yaml`: Modify the following with your specific values to generate a copyable command: REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system ``` --- apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository metadata: name: vcluster namespace: flux-system spec: interval: 1h url: https://charts.loft.sh ``` 2) The vCluster will be deployed into the namespace `team-x` of the host cluster. If this namespace doesn’t already exist, create it with: Modify the following with your specific values to generate a copyable command: NAMESPACEteam-x ``` kubectl create namespace team-x ``` 3) Create a vCluster `HelmRelease` file in your Git repository. This `HelmRelease` tells Flux how to deploy a vCluster in your Kubernetes cluster using the configured Helm charts in Step 1. Save the following file as `clusters/production/vcluster-demo.yaml`: Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system SECRET NAMEvcluster-flux-kubeconfig ``` --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: vcluster-demo namespace: team-x spec: interval: 10m chart: spec: chart: vcluster version: "0.28.x" sourceRef: kind: HelmRepository name: vcluster namespace: flux-system values: # Configure TLS SAN for the certificate controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" coredns: enabled: true exportKubeConfig: # Set a meaningful context name context: default # Use a server URL that matches the TLS SAN server: https://vcluster-demo.team-x.svc.cluster.local:443 # Skip TLS verification when Flux connects to the vCluster insecure: true # Specify the secret where the KubeConfig is stored secret: name: vcluster-flux-kubeconfig sync: toHost: ingresses: enabled: true ``` The content in `vcluster.yaml` should be placed under the `values`. Unlike ArgoCD, Flux requires the KubeConfig secret to be able to access the vCluster's API server and deploy to vCluster. The `exportKubeConfig` configuration under `values`: * Exports the virtual cluster KubeConfig as a Secret in the host namespace * Makes the Secret available for Flux to use with the `spec.kubeConfig` field * Uses a server URL that is accessible from the Flux controllers (replace `vcluster-name` and `team-x` with your actual values) * Sets `insecure: true` to automatically skip TLS certificate verification * Adds a TLS SAN (Subject Alternative Name) that matches the server URL, which helps prevent certificate verification errors KubeConfig Secret key The vCluster `exportKubeConfig` configuration creates a Secret with the KubeConfig data stored under the key `config`. When referring to this Secret in Flux resources, you must specify this key in the `secretRef.key` field, as shown in the examples below. Modify the following with your specific values to generate a copyable command: SECRET NAMEvcluster-flux-kubeconfig ``` # In Flux HelmRelease spec: kubeConfig: secretRef: name: vcluster-flux-kubeconfig key: config # Must match the key used in the vCluster-generated Secret ``` Certificate verification considerations When using vCluster with Flux, proper TLS certificate configuration is essential: 1. Set `exportKubeConfig.insecure: true` in your vCluster configuration 2. Configure proper TLS SANs with the `--tls-san` flag in vCluster configuration 3. Ensure the server URL matches the certificate's SAN Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x ``` # In your vCluster configuration controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" exportKubeConfig: server: https://vcluster-demo.team-x.svc.cluster.local:443 insecure: true ``` 4) After adding the `HelmRelease` and any supporting files to your Git repository, commit and push them: Modify the following with your specific values to generate a copyable command: FOLDER NAMEclusters ``` git add clusters/ git commit -m "Add vCluster demo configuration" git push ``` Once the changes are pushed, Flux will automatically detect them and deploy the vCluster according to the configuration in your repository. Cluster API (CAPI) provides lifecycle management for Kubernetes clusters. The vCluster CAPI provider enables you to manage virtual clusters using the same declarative APIs and tooling used for physical clusters. For more details, see the [Cluster API Provider for vCluster documentation](https://github.com/loft-sh/cluster-api-provider-vcluster). 1. Install the [`clusterctl` CLI](https://cluster-api.sigs.k8s.io/user/quick-start.html#install-clusterctl). 2. Install the vCluster provider: ``` clusterctl init --infrastructure vcluster:v0.2.0 ``` 3. Export environment variables for the Cluster API provider to create the manifest. The manifest is applied to your Kubernetes cluster, which deploys a vCluster. Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` export CLUSTER_NAME=my-vcluster export CLUSTER_NAMESPACE=team-x export VCLUSTER_YAML=$(awk '{printf "%s\n", $0}' vcluster.yaml) ``` 4. Create the namespace for the vCluster using the exported variable: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x ``` kubectl create namespace team-x ``` 5. Generate the required manifests and apply them using the exported variables: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CLUSTER NAMESPACEteam-x ``` clusterctl generate cluster my-vcluster \ --infrastructure vcluster \ --target-namespace team-x \ | kubectl apply -f - ``` Kubernetes version The Kubernetes version for the vCluster is not set at the CAPI provider command. Configure it in the `vcluster.yaml` file based on your Kubernetes distribution. 6. Wait for vCluster to become ready by monitoring the vCluster custom resource status: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x CLUSTER NAMEmy-vcluster ``` kubectl wait --for=condition=ready vcluster -n team-x my-vcluster --timeout=300s ``` ## Deploy with vCluster Platform[​](#deploy-with-vcluster-platform "Direct link to Deploy with vCluster Platform") Tenant clusters can be deployed directly as described above, or managed centrally through vCluster Platform — a management UI for tenant clusters, access control, and lifecycle operations. There are many [features](https://vcluster.com/docs/platform/.md) included as part of vCluster Platform beyond central management. If you already have vCluster Platform, you can create a tenant cluster directly from the CLI. Don't have Platform yet? Run `vcluster platform start` against your control plane cluster. See the [Platform install guide](https://vcluster.com/docs/platform/install/quick-start-guide.md) for the full process. When creating a cluster from the vCluster CLI, provide the project name to deploy into and the template name to use: ``` vcluster create [vcluster-name] --project [project-name] --template [template-name] --driver platform ``` If you omit `--template`, vCluster Platform resolves one automatically. * Uses the project's default template if one is set. * Auto-selects the template if the project only allows one. * Prompts you to choose if the project allows multiple templates without defining a default. * Fails with an error if no templates are allowed in the project. Kube-Context Running `vcluster create` adds a kube-context to your kube-config file and switches your active context to it. This allows you to immediately run `kubectl` commands for the new cluster right after creating it. Disable the automatic context switch by adding `--connect=false` to the `create` command. For the full workflow, including the platform UI steps, see [deploy tenant clusters in the platform](https://vcluster.com/docs/platform/use-platform/virtual-clusters/create/create-no-template.md). --- # Deploy vCluster on AKS This guide provides step-by-step instructions for deploying vCluster on [Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/products/kubernetes-service). ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before starting, ensure you have the following tools installed: * `kubectl` installed: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. * `vCluster CLI` * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` * [Azure CLI (az)](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) note Ensure you have the necessary permissions to create and manage AKS clusters in your Azure subscription. ## Create AKS cluster[​](#create-aks-cluster "Direct link to Create AKS cluster") Right subscription Ensure you are using the correct Azure subscription by running `az account show`. Start by creating an AKS cluster using the Azure CLI. First, set up your environment variables: Modify the following with your specific values to generate a copyable command: RESOURCE\_GROUPvcluster-demo-rg CLUSTER\_NAMEvcluster-demo LOCATIONeastus NODE\_SIZEStandard\_D4s\_v3 NODE\_COUNT2 ``` export RESOURCE_GROUP=vcluster-demo-rg export CLUSTER_NAME=vcluster-demo export LOCATION=eastus export NODE_SIZE=Standard_D4s_v3 export NODE_COUNT=2 ``` Create a resource group: Create resource group ``` az group create --name $RESOURCE_GROUP --location $LOCATION ``` Create the AKS cluster: Create AKS cluster ``` az aks create \ --resource-group $RESOURCE_GROUP \ --name $CLUSTER_NAME \ --location $LOCATION \ --node-count $NODE_COUNT \ --node-vm-size $NODE_SIZE \ --generate-ssh-keys ``` This command creates an AKS cluster with managed identity enabled and Azure Monitor for containers. ### Configure kubectl[​](#configure-kubectl "Direct link to Configure kubectl") After the cluster is created, get credentials to access the cluster: Get cluster credentials ``` az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME ``` ### Verify the control plane cluster creation[​](#verify-the-control-plane-cluster-creation "Direct link to Verify the control plane cluster creation") Verify the cluster by listing the nodes: List cluster nodes ``` kubectl get nodes ``` You should see output similar to: ``` NAME STATUS ROLES AGE VERSION aks-nodepool1-34960941-vmss000000 Ready 67m v1.29.9 aks-nodepool1-34960941-vmss000001 Ready 67m v1.29.9 ``` ## Predeployment configuration options[​](#predeployment-configuration-options "Direct link to Predeployment configuration options") Before deploying, it's recommended to review the set of configuration options that cannot be updated post deployment. These options require deploying a new vCluster instead of upgrading your vCluster with new options. ### Control Plane Options[​](#control-plane-options "Direct link to Control Plane Options") Decide the various options of how you want your control plane deployed: * **[High availability](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/high-availability.md)** - Run multiple copies of vCluster components. * **[Rootless mode](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/rootless-mode.md)** - Deploy the vCluster pod without root access to the control plane cluster. * **Backing Store** - Decide how the data of your cluster is stored. Backing store options vCluster supports etcd or a relational database (using KINE) as the backend. This feature provides flexibility to vCluster operators. The available data store options allow you to select a data store that fits your use case. vCluster supports the following datastore options: * [Embedded SQLite](https://www.sqlite.org/index.html) (default with `PersistentVolume` (PV)) * [PostgreSQL](https://www.postgresql.org/) * [MySQL](https://www.mysql.com/) * [MariaDB](https://mariadb.org/) * [etcd](https://etcd.io/) warning After deploying your vCluster, there are limited migration paths to change your backing store. Review the backing store migration options before deploying. **Backing store options** * Embedded SQLite (Default) * Embedded SQLite (No PV) * Embedded etcd * Deployed etcd * MySQL / MariaDB * PostgreSQL This is the default, so you don't need to configure anything. If you want to explicitly set this option, you can use: ``` controlPlane: backingStore: database: embedded: enabled: true ``` By default, vCluster stores its data in a `PersistentVolumeClaim` (PVC). Alternatively, you can use an [`emptyDir`](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume to store tenant cluster data. To use an `emptyDir` to store the data instead of a `PersistentVolume`, create a `values.yaml` with the following contents: ``` controlPlane: statefulSet: persistence: volumeClaim: enabled: true ``` Then upgrade or recreate the vCluster with: ``` vcluster create my-vcluster -n my-vcluster --upgrade -f values.yaml ``` Potential data loss This method should only be used for testing purposes, as data is lost upon pod recreation. This is an enterprise feature that allows you to deploy etcd within each vCluster to enable high availability (HA), which isn’t supported with embedded SQLite: ``` controlPlane: backingStore: etcd: embedded: enabled: true ``` This deploys an etcd instance outside of the vCluster control plane pod that is used as a backing store: ``` controlPlane: backingStore: etcd: deploy: enabled: true ``` warning vCluster snapshots aren't supported with this backing store. See [Back up and restore](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md#back-up-and-restore). The option for MySQL and MariaDB typically has the following format: ``` controlPlane: backingStore: database: external: enabled: true dataSource: mysql://username:password@tcp(hostname:3306)/database-name ``` If you specify a database name and it does not exist, the server attempts to create it. The option for PostgreSQL typically has the following format: ``` controlPlane: backingStore: database: external: enabled: true dataSource: postgres://username:password@hostname:port/database-name ``` More advanced configuration parameters are available. For more information, see . If you specify a database name and it does not exist, the server attempts to create it. ### Worker Nodes[​](#worker-nodes "Direct link to Worker Nodes") Decide where you want your worker nodes to come from: * **[Nodes from the control plane cluster](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes.md)** - (Default) All worker nodes of the shared control plane cluster are used by the tenant cluster and all resources are synced to the single namespace that the vCluster is deployed on. * **[Syncing Namespaces](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md)** - Resources are synced to mapped namespaces on the control plane cluster. * **[Isolated workloads](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes/isolated-workloads.md)** - Different options to isolate a workload in a vCluster. * **[Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md)** - Enable adding individual nodes to the tenant cluster. ## Deploy vCluster on AKS[​](#deploy-vcluster-on-aks-1 "Direct link to Deploy vCluster on AKS") YAML configuration If you're not sure which options to configure, you can update most settings later by upgrading your vCluster with an updated `vcluster.yaml`. However, some settings — such as what type of worker nodes or the backing store — can only be set during the initial deployment and cannot be changed during an upgrade. All of the deployment options below have the following assumptions: * A `vcluster.yaml` is provided. Refer to the `vcluster.yaml` reference docs to explore all configuration options. This file is optional and can be removed from the examples. * The vCluster is called `my-vcluster`. * The vCluster is deployed into the `team-x` namespace.
* vCluster CLI * Helm * Terraform * ArgoCD * Flux * Cluster API The vCluster CLI provides the most straightforward way to deploy and manage virtual clusters. 1. Install the vCluster CLI: * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` 2. Deploy vCluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` vcluster create my-vcluster --namespace team-x --values vcluster.yaml ``` note After installation, vCluster automatically switches your Kubernetes context to the new virtual cluster. You can now run `kubectl` commands against the virtual cluster. Helm provides fine-grained control over the deployment process and integrates well with existing Helm-based workflows. 1. Deploy vCluster using the `helm upgrade` command: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CONFIG FILEvcluster.yaml NAMESPACEteam-x ``` helm upgrade --install my-vcluster vcluster \ --values vcluster.yaml \ --repo https://charts.loft.sh \ --namespace team-x \ --repository-config='' \ --create-namespace ``` You can use Terraform to deploy vCluster as code with version control and state management. 1. Create a `main.tf` file to define your vCluster deployment using the Terraform Helm provider: ``` provider "helm" { kubernetes = { config_path = "~/.kube/config" } } resource "helm_release" "my_vcluster" { name = "my-vcluster" namespace = "team-x" create_namespace = true repository = "https://charts.loft.sh" chart = "vcluster" # If you didn't create a vcluster.yaml, remove the values section. values = [ file("${path.module}/vcluster.yaml") ] } ``` Helm Provider Version This configuration uses the Terraform Helm provider v3.x syntax where `kubernetes` is defined as an argument (`kubernetes = {`). If you're using Helm provider v2.x, use the block syntax instead (`kubernetes {`). To use v3.x, ensure your provider version is at least v3.0.0. 2. Install the required [Helm provider](https://registry.terraform.io/providers/hashicorp/helm/latest) and initialize Terraform: ``` terraform init ``` 3. Generate a plan to preview the changes: ``` terraform plan ```
Review the plan output to verify connectivity and proposed changes. 4. Deploy vCluster: ``` terraform apply ``` ArgoCD deployment enables GitOps workflows for vCluster management, and provides automated deployment, drift detection, and declarative configuration management through Git repositories. To deploy vCluster using ArgoCD, you need the following files: * `vcluster.yaml` for your vCluster configuration options. * `-app.yaml` for your ArgoCD `Application` definition. Replace `` with your actual cluster name.
1. Create the ArgoCD `Application` file `-app.yaml`, which references the vCluster Helm chart: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x ``` --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-vcluster namespace: argocd spec: project: default source: chart: vcluster repoURL: https://charts.loft.sh helm: releaseName: my-vcluster valueFiles: - vcluster.yaml destination: server: https://kubernetes.default.svc namespace: team-x ``` 2. Commit and push these files to your configured ArgoCD repository. 3. Sync your ArgoCD repository with your configured cluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster ``` argocd app sync my-vcluster ``` 1) Create a `HelmRepository` source in the Git repository that Flux monitors, so Flux can fetch the vCluster Helm charts automatically. In this example, the related files are stored under the `clusters` folder. Save the following file as `clusters/sources/vcluster-repository.yaml`: Modify the following with your specific values to generate a copyable command: REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system ``` --- apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository metadata: name: vcluster namespace: flux-system spec: interval: 1h url: https://charts.loft.sh ``` 2) The vCluster will be deployed into the namespace `team-x` of the host cluster. If this namespace doesn’t already exist, create it with: Modify the following with your specific values to generate a copyable command: NAMESPACEteam-x ``` kubectl create namespace team-x ``` 3) Create a vCluster `HelmRelease` file in your Git repository. This `HelmRelease` tells Flux how to deploy a vCluster in your Kubernetes cluster using the configured Helm charts in Step 1. Save the following file as `clusters/production/vcluster-demo.yaml`: Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system SECRET NAMEvcluster-flux-kubeconfig ``` --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: vcluster-demo namespace: team-x spec: interval: 10m chart: spec: chart: vcluster version: "0.28.x" sourceRef: kind: HelmRepository name: vcluster namespace: flux-system values: # Configure TLS SAN for the certificate controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" coredns: enabled: true exportKubeConfig: # Set a meaningful context name context: default # Use a server URL that matches the TLS SAN server: https://vcluster-demo.team-x.svc.cluster.local:443 # Skip TLS verification when Flux connects to the vCluster insecure: true # Specify the secret where the KubeConfig is stored secret: name: vcluster-flux-kubeconfig sync: toHost: ingresses: enabled: true ``` The content in `vcluster.yaml` should be placed under the `values`. Unlike ArgoCD, Flux requires the KubeConfig secret to be able to access the vCluster's API server and deploy to vCluster. The `exportKubeConfig` configuration under `values`: * Exports the virtual cluster KubeConfig as a Secret in the host namespace * Makes the Secret available for Flux to use with the `spec.kubeConfig` field * Uses a server URL that is accessible from the Flux controllers (replace `vcluster-name` and `team-x` with your actual values) * Sets `insecure: true` to automatically skip TLS certificate verification * Adds a TLS SAN (Subject Alternative Name) that matches the server URL, which helps prevent certificate verification errors KubeConfig Secret key The vCluster `exportKubeConfig` configuration creates a Secret with the KubeConfig data stored under the key `config`. When referring to this Secret in Flux resources, you must specify this key in the `secretRef.key` field, as shown in the examples below. Modify the following with your specific values to generate a copyable command: SECRET NAMEvcluster-flux-kubeconfig ``` # In Flux HelmRelease spec: kubeConfig: secretRef: name: vcluster-flux-kubeconfig key: config # Must match the key used in the vCluster-generated Secret ``` Certificate verification considerations When using vCluster with Flux, proper TLS certificate configuration is essential: 1. Set `exportKubeConfig.insecure: true` in your vCluster configuration 2. Configure proper TLS SANs with the `--tls-san` flag in vCluster configuration 3. Ensure the server URL matches the certificate's SAN Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x ``` # In your vCluster configuration controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" exportKubeConfig: server: https://vcluster-demo.team-x.svc.cluster.local:443 insecure: true ``` 4) After adding the `HelmRelease` and any supporting files to your Git repository, commit and push them: Modify the following with your specific values to generate a copyable command: FOLDER NAMEclusters ``` git add clusters/ git commit -m "Add vCluster demo configuration" git push ``` Once the changes are pushed, Flux will automatically detect them and deploy the vCluster according to the configuration in your repository. Cluster API (CAPI) provides lifecycle management for Kubernetes clusters. The vCluster CAPI provider enables you to manage virtual clusters using the same declarative APIs and tooling used for physical clusters. For more details, see the [Cluster API Provider for vCluster documentation](https://github.com/loft-sh/cluster-api-provider-vcluster). 1. Install the [`clusterctl` CLI](https://cluster-api.sigs.k8s.io/user/quick-start.html#install-clusterctl). 2. Install the vCluster provider: ``` clusterctl init --infrastructure vcluster:v0.2.0 ``` 3. Export environment variables for the Cluster API provider to create the manifest. The manifest is applied to your Kubernetes cluster, which deploys a vCluster. Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` export CLUSTER_NAME=my-vcluster export CLUSTER_NAMESPACE=team-x export VCLUSTER_YAML=$(awk '{printf "%s\n", $0}' vcluster.yaml) ``` 4. Create the namespace for the vCluster using the exported variable: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x ``` kubectl create namespace team-x ``` 5. Generate the required manifests and apply them using the exported variables: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CLUSTER NAMESPACEteam-x ``` clusterctl generate cluster my-vcluster \ --infrastructure vcluster \ --target-namespace team-x \ | kubectl apply -f - ``` Kubernetes version The Kubernetes version for the vCluster is not set at the CAPI provider command. Configure it in the `vcluster.yaml` file based on your Kubernetes distribution. 6. Wait for vCluster to become ready by monitoring the vCluster custom resource status: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x CLUSTER NAMEmy-vcluster ``` kubectl wait --for=condition=ready vcluster -n team-x my-vcluster --timeout=300s ``` ## Next steps[​](#next-steps "Direct link to Next steps") Now that you have vCluster running on AKS, consider exploring the [platform UI](https://vcluster.com/docs/platform/install/quick-start-guide.md) to manage your tenant clusters. ## Cleanup[​](#cleanup "Direct link to Cleanup") If you deployed the AKS cluster with this tutorial, and want to clean up the resources, run the following command: Clean up resources ``` az group delete --name $RESOURCE_GROUP --yes --no-wait ``` --- # Isolate clusters with Netris and NVIDIA BCM One deployment model for vCluster is through private nodes. It allows for stronger isolation between the tenant clusters, since the Pods in a tenant cluster don't share the same Nodes. Unlike cloud providers that offer stable network abstractions for automation, bare metal infrastructure lacks these built-in capabilities, making network configuration and management significantly more complex. This guide shows how vCluster with [Netris](https://netris.ai/) delivers an equally seamless integration for bare metal environments. It enables you to isolate tenant clusters spanning mixed bare metal and VM private nodes that share the same underlying hardware with an exclusive network. Netris server clusters provide network abstractions that can be configured in different ways depending on your hardware setup—including access ports, trunk ports, or other network configurations. ![Architecture](/docs/assets/images/netris_architecture-4193417116c3858b8203755608749943.svg) The goal of this guide is to have a tenant cluster on an isolated network: * Tenant cluster managed by vCluster platform * Control plane is reachable on the network behind a stable address * BCM server provisioned and joined as a private node * KubeVirt VM provisioned and joined as a private node * Nodes share the same network and can freely communicate * Hosts on other networks cannot reach the Nodes or workloads (unless explicitly peered) Like in traditional Kubernetes clusters, Nodes share the same network domain. Instead of control plane *nodes*, the vCluster control plane runs as (multihomed) Pods in the control plane cluster. This is transparent to the tenant cluster; however, the control plane cluster remains inaccessible. This can be repeated for any number of clusters. The VPCs of each tenant cluster may also be peered to access shared services across clusters. ## Netris[​](#netris "Direct link to Netris") Netris provides software for managing and automating the switch fabric in a datacenter. We use the [Netris controller](https://netris.ai/docs/controller/) to manage the architecture logically (by servers being members in [server clusters](https://netris.ai/docs/controller/server-clusters/)) while the [Netris agent](https://netris.ai/docs/agent/) will implement it on the physical switches. In this guide, we use VLAN trunk ports to attach the control plane and VMs to the exclusive network, while bare metal servers connect via access ports. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") We assume the following: * Hardware deployed * Netris controller & agent installed & configured * Networks configured in Netris (VPC and Server cluster) and BCM (See [below](#prepare-networks)) * Default network with BCM head node * Control plane cluster network * vCluster tenant network * Netris controller API is reachable from the control plane cluster network * BCM head node (API) is reachable from the control plane cluster network * Control plane cluster set up (See [below](#prepare-control-plane-cluster)) * BCM head node must be able to run commands on servers with plain `ssh` when they are on a different BCM network * At least one unused server in BCM is available ## Prepare networks[​](#prepare-networks "Direct link to Prepare networks") We must prepare the networks for the control plane cluster and a vCluster tenant in Netris & BCM. ### Netris server clusters[​](#netris-server-clusters "Direct link to Netris server clusters") 1. Create ServerClusterTemplates Create a template for the control plane cluster nodes: serverClusterTemplate.json ``` [ { "ipv4Gateway": "192.168.55.254/21", "ipv4DhcpEnabled": true, "postfix": "nodes", "serverNics": [ "eth9", "eth10" ], "type": "l2vpn", "vlan": "untagged", "vlanID": "auto" } ] ``` Create another one for the vCluster tenant network with a non-overlapping CIDR, we choose `192.168.67.254/21`. 2. Create ServerClusters Create a ServerCluster for each of the networks using the respective templates. For the control plane cluster, add the shared servers as endpoints. This configures the connected ports as access ports within the server cluster V-net, and it's not necessary to make any adjustments in the host OS. ![ServerCluster](/docs/assets/images/netris_create_host_cluster_servercluster-4274282d0c3a00627770eb910aeb964a.png) The one for the tenant network doesn't have any endpoints with untagged ports until later. However, we add the control plane cluster servers as *shared* endpoints. This configures the ports on the shared servers as trunk ports (additionally) carrying tagged VLAN traffic. These frames then ingress on a vlan interface that is a port on the bridge for the tenant we'll configure later. ![ServerCluster](/docs/assets/images/netris_create_vcluster_servercluster-e3158729d9b5d76bdb35122e592acadd.png) 3. Look up the tenant network V-net's VLAN id Netris creates a "V-net" and assigns it a VLAN id. We need the VLAN id (here it's `3`) later when we create the bridge. ![VnetId](/docs/assets/images/netris_edit_vcluster_vnet-e4882abad6d74b5dac2f43a121e81f78.png) ### BCM networks[​](#bcm-networks "Direct link to BCM networks") Create networks in BCM analogous to the server clusters in Netris: ![BCM Network](/docs/assets/images/bcm_create_network-4f4e5d1a2560f29867f7e2fa0ec11ed7.png) ## Prepare control plane cluster[​](#prepare-control-plane-cluster "Direct link to Prepare control plane cluster") info It is advisable to install Kubernetes after first configuring the network. ### Install Kubernetes[​](#install-kubernetes "Direct link to Install Kubernetes") Most Kubernetes distributions should work without any modifications. There must be a default StorageClass available. ### Install compatible CNI plugin[​](#install-compatible-cni-plugin "Direct link to Install compatible CNI plugin") Since the control plane cluster Nodes handle VLAN traffic, some CNI plugin configurations are not compatible. We have tested the following plugins: * Flannel * Calico in VXLAN mode (eBPF mode seems to drop VLAN tagged frames) * Cilium * Must set `cni.exclusive: false` * Must set `bpf.vlanBypass: '{0}'` ### Create bridge for tenant network on control plane cluster nodes[​](#create-bridge-for-tenant-network-on-control-plane-cluster-nodes "Direct link to Create bridge for tenant network on control plane cluster nodes") We create an unfiltered bridge on all control plane cluster nodes that is an extension of the Netris fabric for the tenant server cluster. The bridge has a vlan interface with the VLAN id that we looked up earlier as one of its ports. ``` VLAN_ID=3 INTERFACE=bond0 ip l add link $INTERFACE name $INTERFACE.$VLAN_ID type vlan id $VLAN_ID ip l set up $INTERFACE.$VLAN_ID ip l add name br-$VLAN_ID type bridge ip l set up br-$VLAN_ID ip l set name $INTERFACE.$VLAN_ID master br-$VLAN_ID ``` ### Create NetworkAttachmentDefinition[​](#create-networkattachmentdefinition "Direct link to Create NetworkAttachmentDefinition") Create a NetworkAttachmentDefinition that uses the bridge we just created: ``` apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: vcluster-netris-vms namespace: vcluster-platform spec: config: | { "cniVersion": "1.0.0", "type": "bridge", "bridge": "br-3", "promiscMode": true } ``` ### Install tooling[​](#install-tooling "Direct link to Install tooling") The following tools are required in the control plane cluster to manage interfaces that allow adding Pods and VMs to the tenant cluster network. #### Multus[​](#multus "Direct link to Multus") See install instructions in [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni). Multus adds additional interfaces to the vCluster control plane Pods and KubeVirt VMs to attach them to the tenant cluster VLAN. #### KubeVirt[​](#kubevirt "Direct link to KubeVirt") See install instructions in [KubeVirt](https://kubevirt.io/user-guide/docs/latest/administration/intro.html). KubeVirt works together with Multus and manages the VMs in the control plane cluster. The example below also uses the [containerized data importer](https://kubevirt.io/user-guide/storage/containerized_data_importer/) to create a large enough disk to install and run Kubernetes on top of Ubuntu 24.04. ## Start vCluster platform[​](#start-vcluster-platform "Direct link to Start vCluster platform") info Running vCluster platform and registering grants you 2 weeks of full access to all platform features, including private nodes which is required for this setup. Start vCluster Platform ``` vcluster platform start ``` Provide your email address and complete the onboarding process in the browser. ## Deploy vCluster[​](#deploy-vcluster "Direct link to Deploy vCluster") info The following steps can be repeated for additional vCluster deployments. ### Deploy vCluster control plane[​](#deploy-vcluster-control-plane "Direct link to Deploy vCluster control plane") 1. Create Netris credentials secret First, create a secret with your Netris credentials: ``` apiVersion: v1 kind: Secret metadata: name: netris-credentials namespace: vcluster-platform stringData: url: "https://netris.example.com/api" username: "admin" password: "password" ``` 2. Create vCluster with Netris integration Use the following configuration to enable the Netris integration with automatic kube-vip setup: vcluster.yaml ``` integrations: netris: enabled: true connector: netris-credentials kubeVip: serverCluster: netris-nodes bridge: br-3 ipRange: "192.168.67.253-192.168.67.253" # fixed ip for this guide controlPlane: statefulSet: highAvailability: replicas: 3 backingStore: etcd: embedded: enabled: true coredns: enabled: true privateNodes: enabled: true ``` Connect to the host Kubernetes cluster and run: ``` vcluster platform create vcluster -n vcluster-netris netris --values vcluster.yaml ``` The control plane is accessible on the exclusive network at a fixed IP by means of kube-vip and the automatic configuration with information from Netris. For more details on kube-vip configuration, see [Kube-vip](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/kube-vip.md). ## Join bare metal server[​](#join-bare-metal-server "Direct link to Join bare metal server") ### Configure BCM node provider[​](#configure-bcm-node-provider "Direct link to Configure BCM node provider") 1. Create BCM credentials secret ``` apiVersion: v1 kind: Secret metadata: name: bcm-credentials namespace: vcluster-platform stringData: username: "your-bcm-username" password: "your-bcm-password" ``` 2. Create BCM Node Provider ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: bcm-netris spec: bcm: secretRef: name: bcm-credentials namespace: vcluster-platform endpoint: "https://your-bcm-endpoint.com" nodeTypes: - name: dgx-node nodes: - dgx-01 properties: bcm.vcluster.com/network-interface: bond0 ``` 3. Update vCluster configuration Add the BCM node provider to your vCluster configuration using auto-nodes: vcluster.yaml ``` privateNodes: enabled: true autoNodes: - provider: bcm-netris static: - name: dgx-pool quantity: 1 nodeTypeSelector: - property: vcluster.com/node-type value: dgx-node properties: netris.vcluster.com/server-cluster: netris-nodes ``` 4. Verify node provisioning The BCM node provider will automatically provision the bare metal server as a node in the tenant cluster. You can monitor the node claim status. Run in the control plane cluster: ``` kubectl get nodeclaims -A NAMESPACE NAME STATUS VCLUSTER TYPE CREATED AT p-default netris-8gnqh Available netris bcm-netris.dgx-node 2025-10-27T19:39:45Z ``` Once the node joins the cluster, the claim transitions from `Pending` to `Available` and the node will appear in the tenant cluster: ``` kubectl get nodes NAME STATUS ROLES AGE VERSION dgx-01 Ready 68s v1.34.1 ``` ## Join VM running in control plane cluster infrastructure[​](#join-vm-running-in-control-plane-cluster-infrastructure "Direct link to Join VM running in control plane cluster infrastructure") ### Configure KubeVirt node provider[​](#configure-kubevirt-node-provider "Direct link to Configure KubeVirt node provider") 1. Create KubeVirt Node Provider ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: kubevirt-netris spec: properties: kubevirt.vcluster.com/network-interface: enp1s0 kubeVirt: clusterRef: cluster: loft-cluster namespace: vcluster-platform virtualMachineTemplate: spec: dataVolumeTemplates: - metadata: name: containerdisk spec: source: registry: url: docker://quay.io/containerdisks/ubuntu:24.04 pvc: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi template: spec: domain: devices: disks: - name: containerdisk disk: bus: virtio volumes: - name: containerdisk dataVolume: name: containerdisk nodeTypes: - name: netris-vm maxCapacity: 5 resources: cpu: "2" memory: "4Gi" ``` 2. Update vCluster configuration Add both node providers to your vCluster configuration using auto-nodes: vcluster.yaml ``` privateNodes: enabled: true autoNodes: - provider: bcm-netris static: - name: dgx-pool quantity: 1 - provider: kubevirt-netris static: - name: vm-pool quantity: 1 properties: netris.vcluster.com/server-cluster: netris kubevirt.vcluster.com/network-attachment-definition: vcluster-platform/vcluster-netris-vms ``` 3. Verify VM provisioning The KubeVirt node provider will automatically create VMs that join the tenant cluster. You can monitor the node claims. Run in the control plane cluster: ``` kubectl get nodeclaims -A NAMESPACE NAME STATUS VCLUSTER TYPE CREATED AT p-default netris-swwnh Available netris bcm-netris.dgx-node 2025-10-27T19:48:06Z p-default netris-8gnqh Available netris kubevirt-netris.netris-vm 2025-10-27T19:39:45Z ``` Once the VM joins the cluster, the node claim transitions from `Pending` to `Available` and the VM will appear in the tenant cluster: ``` kubectl get nodes NAME STATUS ROLES AGE VERSION dgx-01 Ready 5m11s v1.34.1 netris-8gnqh Ready 20s v1.34.1 ``` --- # Deploy vCluster on EKS This guide provides step-by-step instructions for deploying vCluster on [Amazon EKS](https://aws.amazon.com/.). ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before starting, ensure you have the following tools installed: * `kubectl` installed: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. * `vCluster CLI` * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` * [AWS CLI](https://aws.amazon.com/cli/) version 1.16.156 or greater note AWS IAM permissions to create roles and policies * [eksctl](https://eksctl.io/) installed for cluster management note [Upgrade](https://github.com/eksctl-io/eksctl?tab=readme-ov-file#installation) `eksctl` to the latest version to ensure latest Kubernetes version is deployed. ## Create EKS cluster[​](#create-eks-cluster "Direct link to Create EKS cluster") Start by creating EKS cluster using `eksctl`. This command creates a file named `cluster.yaml` with the required settings. Adjust the cluster name, region, and instance type as needed. Modify the following with your specific values to generate a copyable command: CLUSTER\_NAMEvcluster-demo REGIONeu-central-1 INSTANCE\_TYPEt3.medium ``` # This will create a file with your custom values cat << EOF > cluster.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: vcluster-demo region: eu-central-1 iam: withOIDC: true nodeGroups: - name: ng-1 instanceType: t3.medium desiredCapacity: 2 iam: withAddonPolicies: ebs: true volumeSize: 80 addons: - name: aws-ebs-csi-driver version: latest attachPolicyARNs: - arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy EOF ``` The file defines a cluster with two `t3.medium` instances located in the `eu-central-1` region. The configuration includes: * OIDC provider enabled for IAM roles for service accounts * Node group with EBS addon policy for volume management * EBS CSI driver addon with the official AWS managed IAM policy Create the cluster by running: Create EKS cluster ``` eksctl create cluster -f cluster.yaml ``` kubeconfig update This command automatically updates your `kubeconfig` file with the new cluster configuration. This process typically takes about 15-20 minutes. ### Verify the control plane cluster creation[​](#verify-the-control-plane-cluster-creation "Direct link to Verify the control plane cluster creation") Verify the installation by checking if the CSI driver pods are running: Verify CSI driver installation ``` kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver ``` Expected output should look similar to: ``` NAME READY STATUS RESTARTS AGE ebs-csi-controller-794b4448b-fhjxr 6/6 Running 0 2m14s ebs-csi-controller-794b4448b-j94g5 6/6 Running 0 2m14s ebs-csi-node-crz7p 3/3 Running 0 2m14s ebs-csi-node-jg8n8 3/3 Running 0 2m14s ``` ### Configure storage class[​](#configure-storage-class "Direct link to Configure storage class") `vCluster` requires a default StorageClass for its persistent volumes. EKS provides the `gp2` StorageClass by default, but `gp3` is required. Create a new StorageClass: gp3 StorageClass configuration ``` cat <. If you specify a database name and it does not exist, the server attempts to create it. ### Worker Nodes[​](#worker-nodes "Direct link to Worker Nodes") Decide where you want your worker nodes to come from: * **[Nodes from the control plane cluster](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes.md)** - (Default) All worker nodes of the shared control plane cluster are used by the tenant cluster and all resources are synced to the single namespace that the vCluster is deployed on. * **[Syncing Namespaces](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md)** - Resources are synced to mapped namespaces on the control plane cluster. * **[Isolated workloads](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes/isolated-workloads.md)** - Different options to isolate a workload in a vCluster. * **[Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md)** - Enable adding individual nodes to the tenant cluster. ## Deploy vCluster on EKS[​](#deploy-vcluster-on-eks-1 "Direct link to Deploy vCluster on EKS") YAML configuration If you're not sure which options to configure, you can update most settings later by upgrading your vCluster with an updated `vcluster.yaml`. However, some settings — such as what type of worker nodes or the backing store — can only be set during the initial deployment and cannot be changed during an upgrade. All of the deployment options below have the following assumptions: * A `vcluster.yaml` is provided. Refer to the `vcluster.yaml` reference docs to explore all configuration options. This file is optional and can be removed from the examples. * The vCluster is called `my-vcluster`. * The vCluster is deployed into the `team-x` namespace.
* vCluster CLI * Helm * Terraform * ArgoCD * Flux * Cluster API The vCluster CLI provides the most straightforward way to deploy and manage virtual clusters. 1. Install the vCluster CLI: * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` 2. Deploy vCluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` vcluster create my-vcluster --namespace team-x --values vcluster.yaml ``` note After installation, vCluster automatically switches your Kubernetes context to the new virtual cluster. You can now run `kubectl` commands against the virtual cluster. Helm provides fine-grained control over the deployment process and integrates well with existing Helm-based workflows. 1. Deploy vCluster using the `helm upgrade` command: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CONFIG FILEvcluster.yaml NAMESPACEteam-x ``` helm upgrade --install my-vcluster vcluster \ --values vcluster.yaml \ --repo https://charts.loft.sh \ --namespace team-x \ --repository-config='' \ --create-namespace ``` You can use Terraform to deploy vCluster as code with version control and state management. 1. Create a `main.tf` file to define your vCluster deployment using the Terraform Helm provider: ``` provider "helm" { kubernetes = { config_path = "~/.kube/config" } } resource "helm_release" "my_vcluster" { name = "my-vcluster" namespace = "team-x" create_namespace = true repository = "https://charts.loft.sh" chart = "vcluster" # If you didn't create a vcluster.yaml, remove the values section. values = [ file("${path.module}/vcluster.yaml") ] } ``` Helm Provider Version This configuration uses the Terraform Helm provider v3.x syntax where `kubernetes` is defined as an argument (`kubernetes = {`). If you're using Helm provider v2.x, use the block syntax instead (`kubernetes {`). To use v3.x, ensure your provider version is at least v3.0.0. 2. Install the required [Helm provider](https://registry.terraform.io/providers/hashicorp/helm/latest) and initialize Terraform: ``` terraform init ``` 3. Generate a plan to preview the changes: ``` terraform plan ```
Review the plan output to verify connectivity and proposed changes. 4. Deploy vCluster: ``` terraform apply ``` ArgoCD deployment enables GitOps workflows for vCluster management, and provides automated deployment, drift detection, and declarative configuration management through Git repositories. To deploy vCluster using ArgoCD, you need the following files: * `vcluster.yaml` for your vCluster configuration options. * `-app.yaml` for your ArgoCD `Application` definition. Replace `` with your actual cluster name.
1. Create the ArgoCD `Application` file `-app.yaml`, which references the vCluster Helm chart: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x ``` --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-vcluster namespace: argocd spec: project: default source: chart: vcluster repoURL: https://charts.loft.sh helm: releaseName: my-vcluster valueFiles: - vcluster.yaml destination: server: https://kubernetes.default.svc namespace: team-x ``` 2. Commit and push these files to your configured ArgoCD repository. 3. Sync your ArgoCD repository with your configured cluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster ``` argocd app sync my-vcluster ``` 1) Create a `HelmRepository` source in the Git repository that Flux monitors, so Flux can fetch the vCluster Helm charts automatically. In this example, the related files are stored under the `clusters` folder. Save the following file as `clusters/sources/vcluster-repository.yaml`: Modify the following with your specific values to generate a copyable command: REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system ``` --- apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository metadata: name: vcluster namespace: flux-system spec: interval: 1h url: https://charts.loft.sh ``` 2) The vCluster will be deployed into the namespace `team-x` of the host cluster. If this namespace doesn’t already exist, create it with: Modify the following with your specific values to generate a copyable command: NAMESPACEteam-x ``` kubectl create namespace team-x ``` 3) Create a vCluster `HelmRelease` file in your Git repository. This `HelmRelease` tells Flux how to deploy a vCluster in your Kubernetes cluster using the configured Helm charts in Step 1. Save the following file as `clusters/production/vcluster-demo.yaml`: Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system SECRET NAMEvcluster-flux-kubeconfig ``` --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: vcluster-demo namespace: team-x spec: interval: 10m chart: spec: chart: vcluster version: "0.28.x" sourceRef: kind: HelmRepository name: vcluster namespace: flux-system values: # Configure TLS SAN for the certificate controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" coredns: enabled: true exportKubeConfig: # Set a meaningful context name context: default # Use a server URL that matches the TLS SAN server: https://vcluster-demo.team-x.svc.cluster.local:443 # Skip TLS verification when Flux connects to the vCluster insecure: true # Specify the secret where the KubeConfig is stored secret: name: vcluster-flux-kubeconfig sync: toHost: ingresses: enabled: true ``` The content in `vcluster.yaml` should be placed under the `values`. Unlike ArgoCD, Flux requires the KubeConfig secret to be able to access the vCluster's API server and deploy to vCluster. The `exportKubeConfig` configuration under `values`: * Exports the virtual cluster KubeConfig as a Secret in the host namespace * Makes the Secret available for Flux to use with the `spec.kubeConfig` field * Uses a server URL that is accessible from the Flux controllers (replace `vcluster-name` and `team-x` with your actual values) * Sets `insecure: true` to automatically skip TLS certificate verification * Adds a TLS SAN (Subject Alternative Name) that matches the server URL, which helps prevent certificate verification errors KubeConfig Secret key The vCluster `exportKubeConfig` configuration creates a Secret with the KubeConfig data stored under the key `config`. When referring to this Secret in Flux resources, you must specify this key in the `secretRef.key` field, as shown in the examples below. Modify the following with your specific values to generate a copyable command: SECRET NAMEvcluster-flux-kubeconfig ``` # In Flux HelmRelease spec: kubeConfig: secretRef: name: vcluster-flux-kubeconfig key: config # Must match the key used in the vCluster-generated Secret ``` Certificate verification considerations When using vCluster with Flux, proper TLS certificate configuration is essential: 1. Set `exportKubeConfig.insecure: true` in your vCluster configuration 2. Configure proper TLS SANs with the `--tls-san` flag in vCluster configuration 3. Ensure the server URL matches the certificate's SAN Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x ``` # In your vCluster configuration controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" exportKubeConfig: server: https://vcluster-demo.team-x.svc.cluster.local:443 insecure: true ``` 4) After adding the `HelmRelease` and any supporting files to your Git repository, commit and push them: Modify the following with your specific values to generate a copyable command: FOLDER NAMEclusters ``` git add clusters/ git commit -m "Add vCluster demo configuration" git push ``` Once the changes are pushed, Flux will automatically detect them and deploy the vCluster according to the configuration in your repository. Cluster API (CAPI) provides lifecycle management for Kubernetes clusters. The vCluster CAPI provider enables you to manage virtual clusters using the same declarative APIs and tooling used for physical clusters. For more details, see the [Cluster API Provider for vCluster documentation](https://github.com/loft-sh/cluster-api-provider-vcluster). 1. Install the [`clusterctl` CLI](https://cluster-api.sigs.k8s.io/user/quick-start.html#install-clusterctl). 2. Install the vCluster provider: ``` clusterctl init --infrastructure vcluster:v0.2.0 ``` 3. Export environment variables for the Cluster API provider to create the manifest. The manifest is applied to your Kubernetes cluster, which deploys a vCluster. Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` export CLUSTER_NAME=my-vcluster export CLUSTER_NAMESPACE=team-x export VCLUSTER_YAML=$(awk '{printf "%s\n", $0}' vcluster.yaml) ``` 4. Create the namespace for the vCluster using the exported variable: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x ``` kubectl create namespace team-x ``` 5. Generate the required manifests and apply them using the exported variables: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CLUSTER NAMESPACEteam-x ``` clusterctl generate cluster my-vcluster \ --infrastructure vcluster \ --target-namespace team-x \ | kubectl apply -f - ``` Kubernetes version The Kubernetes version for the vCluster is not set at the CAPI provider command. Configure it in the `vcluster.yaml` file based on your Kubernetes distribution. 6. Wait for vCluster to become ready by monitoring the vCluster custom resource status: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x CLUSTER NAMEmy-vcluster ``` kubectl wait --for=condition=ready vcluster -n team-x my-vcluster --timeout=300s ``` This configuration ensures that: * Service accounts are properly synced between tenant and control plane clusters * Persistent volume claims are handled correctly * The `gp3` storage class created earlier is used ### Allow internal DNS resolution[​](#allow-internal-dns-resolution "Direct link to Allow internal DNS resolution") By default, vCluster runs a CoreDNS component inside the tenant cluster. This component listens on port `1053` instead of the standard DNS port `53` to avoid conflicts with the control plane cluster DNS. On EKS, if the CoreDNS pod and other tenant cluster pods are scheduled on different nodes, DNS resolution may fail. This happens because AWS creates separate security groups for the EKS control plane and worker nodes, and the default node security group does not allow inbound traffic on port `1053`. To resolve this, manually update the EKS node security group to allow inbound TCP and UDP traffic on port `1053` between nodes. tip This step is especially important for EKS clusters created using Terraform or other automation tools that apply restrictive network settings by default. ## Next steps[​](#next-steps "Direct link to Next steps") Now that you have vCluster running on EKS, consider: * Setting up the [platform UI](https://vcluster.com/docs/platform/install/quick-start-guide.md) to manage your tenant clusters. * Integrating with [Karpenter](https://vcluster.com/docs/platform/integrations/karpenter.md) for autoscaling. ### Pod identity[​](#pod-identity "Direct link to Pod identity") When using the platform you can easily enable [Pod Identity](https://vcluster.com/docs/vcluster/third-party-integrations/pod-identity/eks-pod-identity.md). ## Cleanup[​](#cleanup "Direct link to Cleanup") If you deployed the EKS cluster with this tutorial, and want to clean up the resources, run the following command: Clean up resources ``` eksctl delete cluster -f cluster.yaml --disable-nodegroup-eviction ``` --- # Deploy vCluster on GKE This guide provides step-by-step instructions for deploying vCluster on [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine). ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before starting, ensure you have the following tools installed: * `kubectl`: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. * vCluster CLI * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` * [Google Cloud SDK (`gcloud` CLI)](https://cloud.google.com/sdk/docs/install) note Ensure you have the necessary IAM permissions to create clusters and manage cloud services. ## Create GKE cluster[​](#create-gke-cluster "Direct link to Create GKE cluster") Start by creating a zonal GKE cluster using the `gcloud` CLI. First, set up your environment variables: tip Project ID can be found in the Google Cloud Console under the project name. Alternatively use `gcloud project list` to list all projects and their IDs. To check which project is active, use `gcloud config get-value project`. Modify the following with your specific values to generate a copyable command: PROJECT\_IDdevelopment CLUSTER\_NAMEvcluster-demo ZONEeurope-west1-b MACHINE\_TYPEe2-standard-4 ``` export PROJECT_ID=development export CLUSTER_NAME=vcluster-demo export ZONE=europe-west1-b export MACHINE_TYPE=e2-standard-4 ``` Configure `gcloud` and enable the required APIs and set default project: Configure gcloud ``` gcloud config set project $PROJECT_ID gcloud services enable container.googleapis.com ``` Create the cluster: Create GKE cluster ``` gcloud container clusters create $CLUSTER_NAME \ --zone $ZONE \ --machine-type $MACHINE_TYPE \ --num-nodes 2 ``` info This process typically takes about 10-15 minutes. This command creates a GKE cluster named `vcluster-demo` in the europe-west1-b zone with two nodes of type e2-standard-4. kubeconfig update This command automatically updates your `kubeconfig` file with the new cluster configuration. ### Verify the control plane cluster creation[​](#verify-the-control-plane-cluster-creation "Direct link to Verify the control plane cluster creation") Verify the cluster by listing the nodes: List cluster nodes ``` kubectl get nodes ``` You should see output similar to: ``` NAME STATUS ROLES AGE VERSION gke-vcluster-demo-default-pool-a1b2c3d4-e5f6 Ready 5m v1.30.5-gke.1443001 gke-vcluster-demo-default-pool-a1b2c3d4-g7h8 Ready 5m v1.30.5-gke.1443001 ``` ## Predeployment configuration options[​](#predeployment-configuration-options "Direct link to Predeployment configuration options") Before deploying, it's recommended to review the set of configuration options that cannot be updated post deployment. These options require deploying a new vCluster instead of upgrading your vCluster with new options. ### Control Plane Options[​](#control-plane-options "Direct link to Control Plane Options") Decide the various options of how you want your control plane deployed: * **[High availability](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/high-availability.md)** - Run multiple copies of vCluster components. * **[Rootless mode](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/rootless-mode.md)** - Deploy the vCluster pod without root access to the control plane cluster. * **Backing Store** - Decide how the data of your cluster is stored. Backing store options vCluster supports etcd or a relational database (using KINE) as the backend. This feature provides flexibility to vCluster operators. The available data store options allow you to select a data store that fits your use case. vCluster supports the following datastore options: * [Embedded SQLite](https://www.sqlite.org/index.html) (default with `PersistentVolume` (PV)) * [PostgreSQL](https://www.postgresql.org/) * [MySQL](https://www.mysql.com/) * [MariaDB](https://mariadb.org/) * [etcd](https://etcd.io/) warning After deploying your vCluster, there are limited migration paths to change your backing store. Review the backing store migration options before deploying. **Backing store options** * Embedded SQLite (Default) * Embedded SQLite (No PV) * Embedded etcd * Deployed etcd * MySQL / MariaDB * PostgreSQL This is the default, so you don't need to configure anything. If you want to explicitly set this option, you can use: ``` controlPlane: backingStore: database: embedded: enabled: true ``` By default, vCluster stores its data in a `PersistentVolumeClaim` (PVC). Alternatively, you can use an [`emptyDir`](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume to store tenant cluster data. To use an `emptyDir` to store the data instead of a `PersistentVolume`, create a `values.yaml` with the following contents: ``` controlPlane: statefulSet: persistence: volumeClaim: enabled: true ``` Then upgrade or recreate the vCluster with: ``` vcluster create my-vcluster -n my-vcluster --upgrade -f values.yaml ``` Potential data loss This method should only be used for testing purposes, as data is lost upon pod recreation. This is an enterprise feature that allows you to deploy etcd within each vCluster to enable high availability (HA), which isn’t supported with embedded SQLite: ``` controlPlane: backingStore: etcd: embedded: enabled: true ``` This deploys an etcd instance outside of the vCluster control plane pod that is used as a backing store: ``` controlPlane: backingStore: etcd: deploy: enabled: true ``` warning vCluster snapshots aren't supported with this backing store. See [Back up and restore](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md#back-up-and-restore). The option for MySQL and MariaDB typically has the following format: ``` controlPlane: backingStore: database: external: enabled: true dataSource: mysql://username:password@tcp(hostname:3306)/database-name ``` If you specify a database name and it does not exist, the server attempts to create it. The option for PostgreSQL typically has the following format: ``` controlPlane: backingStore: database: external: enabled: true dataSource: postgres://username:password@hostname:port/database-name ``` More advanced configuration parameters are available. For more information, see . If you specify a database name and it does not exist, the server attempts to create it. ### Worker Nodes[​](#worker-nodes "Direct link to Worker Nodes") Decide where you want your worker nodes to come from: * **[Nodes from the control plane cluster](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes.md)** - (Default) All worker nodes of the shared control plane cluster are used by the tenant cluster and all resources are synced to the single namespace that the vCluster is deployed on. * **[Syncing Namespaces](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md)** - Resources are synced to mapped namespaces on the control plane cluster. * **[Isolated workloads](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes/isolated-workloads.md)** - Different options to isolate a workload in a vCluster. * **[Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md)** - Enable adding individual nodes to the tenant cluster. ## Deploy vCluster on GKE[​](#deploy-vcluster-on-gke-1 "Direct link to Deploy vCluster on GKE") YAML configuration If you're not sure which options to configure, you can update most settings later by upgrading your vCluster with an updated `vcluster.yaml`. However, some settings — such as what type of worker nodes or the backing store — can only be set during the initial deployment and cannot be changed during an upgrade. All of the deployment options below have the following assumptions: * A `vcluster.yaml` is provided. Refer to the `vcluster.yaml` reference docs to explore all configuration options. This file is optional and can be removed from the examples. * The vCluster is called `my-vcluster`. * The vCluster is deployed into the `team-x` namespace.
* vCluster CLI * Helm * Terraform * ArgoCD * Flux * Cluster API The vCluster CLI provides the most straightforward way to deploy and manage virtual clusters. 1. Install the vCluster CLI: * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` 2. Deploy vCluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` vcluster create my-vcluster --namespace team-x --values vcluster.yaml ``` note After installation, vCluster automatically switches your Kubernetes context to the new virtual cluster. You can now run `kubectl` commands against the virtual cluster. Helm provides fine-grained control over the deployment process and integrates well with existing Helm-based workflows. 1. Deploy vCluster using the `helm upgrade` command: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CONFIG FILEvcluster.yaml NAMESPACEteam-x ``` helm upgrade --install my-vcluster vcluster \ --values vcluster.yaml \ --repo https://charts.loft.sh \ --namespace team-x \ --repository-config='' \ --create-namespace ``` You can use Terraform to deploy vCluster as code with version control and state management. 1. Create a `main.tf` file to define your vCluster deployment using the Terraform Helm provider: ``` provider "helm" { kubernetes = { config_path = "~/.kube/config" } } resource "helm_release" "my_vcluster" { name = "my-vcluster" namespace = "team-x" create_namespace = true repository = "https://charts.loft.sh" chart = "vcluster" # If you didn't create a vcluster.yaml, remove the values section. values = [ file("${path.module}/vcluster.yaml") ] } ``` Helm Provider Version This configuration uses the Terraform Helm provider v3.x syntax where `kubernetes` is defined as an argument (`kubernetes = {`). If you're using Helm provider v2.x, use the block syntax instead (`kubernetes {`). To use v3.x, ensure your provider version is at least v3.0.0. 2. Install the required [Helm provider](https://registry.terraform.io/providers/hashicorp/helm/latest) and initialize Terraform: ``` terraform init ``` 3. Generate a plan to preview the changes: ``` terraform plan ```
Review the plan output to verify connectivity and proposed changes. 4. Deploy vCluster: ``` terraform apply ``` ArgoCD deployment enables GitOps workflows for vCluster management, and provides automated deployment, drift detection, and declarative configuration management through Git repositories. To deploy vCluster using ArgoCD, you need the following files: * `vcluster.yaml` for your vCluster configuration options. * `-app.yaml` for your ArgoCD `Application` definition. Replace `` with your actual cluster name.
1. Create the ArgoCD `Application` file `-app.yaml`, which references the vCluster Helm chart: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x ``` --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-vcluster namespace: argocd spec: project: default source: chart: vcluster repoURL: https://charts.loft.sh helm: releaseName: my-vcluster valueFiles: - vcluster.yaml destination: server: https://kubernetes.default.svc namespace: team-x ``` 2. Commit and push these files to your configured ArgoCD repository. 3. Sync your ArgoCD repository with your configured cluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster ``` argocd app sync my-vcluster ``` 1) Create a `HelmRepository` source in the Git repository that Flux monitors, so Flux can fetch the vCluster Helm charts automatically. In this example, the related files are stored under the `clusters` folder. Save the following file as `clusters/sources/vcluster-repository.yaml`: Modify the following with your specific values to generate a copyable command: REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system ``` --- apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository metadata: name: vcluster namespace: flux-system spec: interval: 1h url: https://charts.loft.sh ``` 2) The vCluster will be deployed into the namespace `team-x` of the host cluster. If this namespace doesn’t already exist, create it with: Modify the following with your specific values to generate a copyable command: NAMESPACEteam-x ``` kubectl create namespace team-x ``` 3) Create a vCluster `HelmRelease` file in your Git repository. This `HelmRelease` tells Flux how to deploy a vCluster in your Kubernetes cluster using the configured Helm charts in Step 1. Save the following file as `clusters/production/vcluster-demo.yaml`: Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system SECRET NAMEvcluster-flux-kubeconfig ``` --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: vcluster-demo namespace: team-x spec: interval: 10m chart: spec: chart: vcluster version: "0.28.x" sourceRef: kind: HelmRepository name: vcluster namespace: flux-system values: # Configure TLS SAN for the certificate controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" coredns: enabled: true exportKubeConfig: # Set a meaningful context name context: default # Use a server URL that matches the TLS SAN server: https://vcluster-demo.team-x.svc.cluster.local:443 # Skip TLS verification when Flux connects to the vCluster insecure: true # Specify the secret where the KubeConfig is stored secret: name: vcluster-flux-kubeconfig sync: toHost: ingresses: enabled: true ``` The content in `vcluster.yaml` should be placed under the `values`. Unlike ArgoCD, Flux requires the KubeConfig secret to be able to access the vCluster's API server and deploy to vCluster. The `exportKubeConfig` configuration under `values`: * Exports the virtual cluster KubeConfig as a Secret in the host namespace * Makes the Secret available for Flux to use with the `spec.kubeConfig` field * Uses a server URL that is accessible from the Flux controllers (replace `vcluster-name` and `team-x` with your actual values) * Sets `insecure: true` to automatically skip TLS certificate verification * Adds a TLS SAN (Subject Alternative Name) that matches the server URL, which helps prevent certificate verification errors KubeConfig Secret key The vCluster `exportKubeConfig` configuration creates a Secret with the KubeConfig data stored under the key `config`. When referring to this Secret in Flux resources, you must specify this key in the `secretRef.key` field, as shown in the examples below. Modify the following with your specific values to generate a copyable command: SECRET NAMEvcluster-flux-kubeconfig ``` # In Flux HelmRelease spec: kubeConfig: secretRef: name: vcluster-flux-kubeconfig key: config # Must match the key used in the vCluster-generated Secret ``` Certificate verification considerations When using vCluster with Flux, proper TLS certificate configuration is essential: 1. Set `exportKubeConfig.insecure: true` in your vCluster configuration 2. Configure proper TLS SANs with the `--tls-san` flag in vCluster configuration 3. Ensure the server URL matches the certificate's SAN Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x ``` # In your vCluster configuration controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" exportKubeConfig: server: https://vcluster-demo.team-x.svc.cluster.local:443 insecure: true ``` 4) After adding the `HelmRelease` and any supporting files to your Git repository, commit and push them: Modify the following with your specific values to generate a copyable command: FOLDER NAMEclusters ``` git add clusters/ git commit -m "Add vCluster demo configuration" git push ``` Once the changes are pushed, Flux will automatically detect them and deploy the vCluster according to the configuration in your repository. Cluster API (CAPI) provides lifecycle management for Kubernetes clusters. The vCluster CAPI provider enables you to manage virtual clusters using the same declarative APIs and tooling used for physical clusters. For more details, see the [Cluster API Provider for vCluster documentation](https://github.com/loft-sh/cluster-api-provider-vcluster). 1. Install the [`clusterctl` CLI](https://cluster-api.sigs.k8s.io/user/quick-start.html#install-clusterctl). 2. Install the vCluster provider: ``` clusterctl init --infrastructure vcluster:v0.2.0 ``` 3. Export environment variables for the Cluster API provider to create the manifest. The manifest is applied to your Kubernetes cluster, which deploys a vCluster. Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` export CLUSTER_NAME=my-vcluster export CLUSTER_NAMESPACE=team-x export VCLUSTER_YAML=$(awk '{printf "%s\n", $0}' vcluster.yaml) ``` 4. Create the namespace for the vCluster using the exported variable: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x ``` kubectl create namespace team-x ``` 5. Generate the required manifests and apply them using the exported variables: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CLUSTER NAMESPACEteam-x ``` clusterctl generate cluster my-vcluster \ --infrastructure vcluster \ --target-namespace team-x \ | kubectl apply -f - ``` Kubernetes version The Kubernetes version for the vCluster is not set at the CAPI provider command. Configure it in the `vcluster.yaml` file based on your Kubernetes distribution. 6. Wait for vCluster to become ready by monitoring the vCluster custom resource status: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x CLUSTER NAMEmy-vcluster ``` kubectl wait --for=condition=ready vcluster -n team-x my-vcluster --timeout=300s ``` ## Next steps[​](#next-steps "Direct link to Next steps") Now that you have vCluster running on GKE, consider setting up the [platform UI](https://vcluster.com/docs/platform/install/quick-start-guide.md) to manage your tenant clusters. ## Workload Identity[​](#workload-identity "Direct link to Workload Identity") When using the platform you can easily enable [Workload Identity](https://vcluster.com/docs/vcluster/third-party-integrations/pod-identity/gke-workload-identity.md). ## Cleanup[​](#cleanup "Direct link to Cleanup") If you deployed the GKE cluster with this tutorial, and want to clean up the resources, run the following command: Clean up resources ``` gcloud container clusters delete $CLUSTER_NAME --zone $ZONE --quiet ``` --- # Deploy on OpenShift OpenShift imposes restrictions that are not common to other Kubernetes distributions, which requires additional configuration options. By default, OpenShift doesn't allow running containers with the root user, but it assigns a random UID from the allowed range automatically. vCluster needs to be configured to enable running as a non-root user. vCluster requires `create` permission for the `endpoints/restricted` resource in the default group when running on OpenShift. This permission is required because OpenShift has an additional built-in admission controller for the Endpoint resources, which denies the creation of the endpoints pointing into the cluster network or service network CIDR ranges. This additional role adds the required permission necessary for certain networking features. Required OpenShift Configuration Options ``` controlPlane: statefulSet: security: podSecurityContext: fsGroup: 12345 containerSecurityContext: runAsUser: 12345 runAsNonRoot: true rbac: role: extraRules: - apiGroups: [""] resources: ["endpoints/restricted"] verbs: ["create"] ``` ## Main predeployment configuration options[​](#main-predeployment-configuration-options "Direct link to Main predeployment configuration options") Before deploying, it's recommended to review the set of configuration options that cannot be updated post deployment. These options require deploying a new vCluster instead of upgrading your vCluster with new options. ### Control Plane Options[​](#control-plane-options "Direct link to Control Plane Options") Decide the various options of how you want your control plane deployed: * **[High availability](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/high-availability.md)** - Run multiple copies of vCluster components. * **[Rootless mode](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/rootless-mode.md)** - Deploy the vCluster pod without root access to the control plane cluster. * **Backing Store** - Decide how the data of your cluster is stored. Backing store options vCluster supports etcd or a relational database (using KINE) as the backend. This feature provides flexibility to vCluster operators. The available data store options allow you to select a data store that fits your use case. vCluster supports the following datastore options: * [Embedded SQLite](https://www.sqlite.org/index.html) (default with `PersistentVolume` (PV)) * [PostgreSQL](https://www.postgresql.org/) * [MySQL](https://www.mysql.com/) * [MariaDB](https://mariadb.org/) * [etcd](https://etcd.io/) warning After deploying your vCluster, there are limited migration paths to change your backing store. Review the backing store migration options before deploying. **Backing store options** * Embedded SQLite (Default) * Embedded SQLite (No PV) * Embedded etcd * Deployed etcd * MySQL / MariaDB * PostgreSQL This is the default, so you don't need to configure anything. If you want to explicitly set this option, you can use: ``` controlPlane: backingStore: database: embedded: enabled: true ``` By default, vCluster stores its data in a `PersistentVolumeClaim` (PVC). Alternatively, you can use an [`emptyDir`](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume to store tenant cluster data. To use an `emptyDir` to store the data instead of a `PersistentVolume`, create a `values.yaml` with the following contents: ``` controlPlane: statefulSet: persistence: volumeClaim: enabled: true ``` Then upgrade or recreate the vCluster with: ``` vcluster create my-vcluster -n my-vcluster --upgrade -f values.yaml ``` Potential data loss This method should only be used for testing purposes, as data is lost upon pod recreation. This is an enterprise feature that allows you to deploy etcd within each vCluster to enable high availability (HA), which isn’t supported with embedded SQLite: ``` controlPlane: backingStore: etcd: embedded: enabled: true ``` This deploys an etcd instance outside of the vCluster control plane pod that is used as a backing store: ``` controlPlane: backingStore: etcd: deploy: enabled: true ``` warning vCluster snapshots aren't supported with this backing store. See [Back up and restore](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md#back-up-and-restore). The option for MySQL and MariaDB typically has the following format: ``` controlPlane: backingStore: database: external: enabled: true dataSource: mysql://username:password@tcp(hostname:3306)/database-name ``` If you specify a database name and it does not exist, the server attempts to create it. The option for PostgreSQL typically has the following format: ``` controlPlane: backingStore: database: external: enabled: true dataSource: postgres://username:password@hostname:port/database-name ``` More advanced configuration parameters are available. For more information, see . If you specify a database name and it does not exist, the server attempts to create it. ### Worker Nodes[​](#worker-nodes "Direct link to Worker Nodes") Decide where you want your worker nodes to come from: * **[Nodes from the control plane cluster](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes.md)** - (Default) All worker nodes of the shared control plane cluster are used by the tenant cluster and all resources are synced to the single namespace that the vCluster is deployed on. * **[Syncing Namespaces](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md)** - Resources are synced to mapped namespaces on the control plane cluster. * **[Isolated workloads](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes/isolated-workloads.md)** - Different options to isolate a workload in a vCluster. * **[Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md)** - Enable adding individual nodes to the tenant cluster. ## Deploy vCluster on OpenShift[​](#deploy-vcluster-on-openshift "Direct link to Deploy vCluster on OpenShift") YAML configuration If you're not sure which options to configure, you can update most settings later by upgrading your vCluster with an updated `vcluster.yaml`. However, some settings — such as what type of worker nodes or the backing store — can only be set during the initial deployment and cannot be changed during an upgrade. All of the deployment options below have the following assumptions: * A `vcluster.yaml` is provided. Refer to the `vcluster.yaml` reference docs to explore all configuration options. This file is optional and can be removed from the examples. * The vCluster is called `my-vcluster`. * The vCluster is deployed into the `team-x` namespace.
* vCluster CLI * Helm * Terraform * ArgoCD * Flux * Cluster API The vCluster CLI provides the most straightforward way to deploy and manage virtual clusters. 1. Install the vCluster CLI: * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` 2. Deploy vCluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` vcluster create my-vcluster --namespace team-x --values vcluster.yaml ``` note After installation, vCluster automatically switches your Kubernetes context to the new virtual cluster. You can now run `kubectl` commands against the virtual cluster. Helm provides fine-grained control over the deployment process and integrates well with existing Helm-based workflows. 1. Deploy vCluster using the `helm upgrade` command: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CONFIG FILEvcluster.yaml NAMESPACEteam-x ``` helm upgrade --install my-vcluster vcluster \ --values vcluster.yaml \ --repo https://charts.loft.sh \ --namespace team-x \ --repository-config='' \ --create-namespace ``` You can use Terraform to deploy vCluster as code with version control and state management. 1. Create a `main.tf` file to define your vCluster deployment using the Terraform Helm provider: ``` provider "helm" { kubernetes = { config_path = "~/.kube/config" } } resource "helm_release" "my_vcluster" { name = "my-vcluster" namespace = "team-x" create_namespace = true repository = "https://charts.loft.sh" chart = "vcluster" # If you didn't create a vcluster.yaml, remove the values section. values = [ file("${path.module}/vcluster.yaml") ] } ``` Helm Provider Version This configuration uses the Terraform Helm provider v3.x syntax where `kubernetes` is defined as an argument (`kubernetes = {`). If you're using Helm provider v2.x, use the block syntax instead (`kubernetes {`). To use v3.x, ensure your provider version is at least v3.0.0. 2. Install the required [Helm provider](https://registry.terraform.io/providers/hashicorp/helm/latest) and initialize Terraform: ``` terraform init ``` 3. Generate a plan to preview the changes: ``` terraform plan ```
Review the plan output to verify connectivity and proposed changes. 4. Deploy vCluster: ``` terraform apply ``` ArgoCD deployment enables GitOps workflows for vCluster management, and provides automated deployment, drift detection, and declarative configuration management through Git repositories. To deploy vCluster using ArgoCD, you need the following files: * `vcluster.yaml` for your vCluster configuration options. * `-app.yaml` for your ArgoCD `Application` definition. Replace `` with your actual cluster name.
1. Create the ArgoCD `Application` file `-app.yaml`, which references the vCluster Helm chart: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x ``` --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-vcluster namespace: argocd spec: project: default source: chart: vcluster repoURL: https://charts.loft.sh helm: releaseName: my-vcluster valueFiles: - vcluster.yaml destination: server: https://kubernetes.default.svc namespace: team-x ``` 2. Commit and push these files to your configured ArgoCD repository. 3. Sync your ArgoCD repository with your configured cluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster ``` argocd app sync my-vcluster ``` 1) Create a `HelmRepository` source in the Git repository that Flux monitors, so Flux can fetch the vCluster Helm charts automatically. In this example, the related files are stored under the `clusters` folder. Save the following file as `clusters/sources/vcluster-repository.yaml`: Modify the following with your specific values to generate a copyable command: REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system ``` --- apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository metadata: name: vcluster namespace: flux-system spec: interval: 1h url: https://charts.loft.sh ``` 2) The vCluster will be deployed into the namespace `team-x` of the host cluster. If this namespace doesn’t already exist, create it with: Modify the following with your specific values to generate a copyable command: NAMESPACEteam-x ``` kubectl create namespace team-x ``` 3) Create a vCluster `HelmRelease` file in your Git repository. This `HelmRelease` tells Flux how to deploy a vCluster in your Kubernetes cluster using the configured Helm charts in Step 1. Save the following file as `clusters/production/vcluster-demo.yaml`: Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system SECRET NAMEvcluster-flux-kubeconfig ``` --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: vcluster-demo namespace: team-x spec: interval: 10m chart: spec: chart: vcluster version: "0.28.x" sourceRef: kind: HelmRepository name: vcluster namespace: flux-system values: # Configure TLS SAN for the certificate controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" coredns: enabled: true exportKubeConfig: # Set a meaningful context name context: default # Use a server URL that matches the TLS SAN server: https://vcluster-demo.team-x.svc.cluster.local:443 # Skip TLS verification when Flux connects to the vCluster insecure: true # Specify the secret where the KubeConfig is stored secret: name: vcluster-flux-kubeconfig sync: toHost: ingresses: enabled: true ``` The content in `vcluster.yaml` should be placed under the `values`. Unlike ArgoCD, Flux requires the KubeConfig secret to be able to access the vCluster's API server and deploy to vCluster. The `exportKubeConfig` configuration under `values`: * Exports the virtual cluster KubeConfig as a Secret in the host namespace * Makes the Secret available for Flux to use with the `spec.kubeConfig` field * Uses a server URL that is accessible from the Flux controllers (replace `vcluster-name` and `team-x` with your actual values) * Sets `insecure: true` to automatically skip TLS certificate verification * Adds a TLS SAN (Subject Alternative Name) that matches the server URL, which helps prevent certificate verification errors KubeConfig Secret key The vCluster `exportKubeConfig` configuration creates a Secret with the KubeConfig data stored under the key `config`. When referring to this Secret in Flux resources, you must specify this key in the `secretRef.key` field, as shown in the examples below. Modify the following with your specific values to generate a copyable command: SECRET NAMEvcluster-flux-kubeconfig ``` # In Flux HelmRelease spec: kubeConfig: secretRef: name: vcluster-flux-kubeconfig key: config # Must match the key used in the vCluster-generated Secret ``` Certificate verification considerations When using vCluster with Flux, proper TLS certificate configuration is essential: 1. Set `exportKubeConfig.insecure: true` in your vCluster configuration 2. Configure proper TLS SANs with the `--tls-san` flag in vCluster configuration 3. Ensure the server URL matches the certificate's SAN Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x ``` # In your vCluster configuration controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" exportKubeConfig: server: https://vcluster-demo.team-x.svc.cluster.local:443 insecure: true ``` 4) After adding the `HelmRelease` and any supporting files to your Git repository, commit and push them: Modify the following with your specific values to generate a copyable command: FOLDER NAMEclusters ``` git add clusters/ git commit -m "Add vCluster demo configuration" git push ``` Once the changes are pushed, Flux will automatically detect them and deploy the vCluster according to the configuration in your repository. Cluster API (CAPI) provides lifecycle management for Kubernetes clusters. The vCluster CAPI provider enables you to manage virtual clusters using the same declarative APIs and tooling used for physical clusters. For more details, see the [Cluster API Provider for vCluster documentation](https://github.com/loft-sh/cluster-api-provider-vcluster). 1. Install the [`clusterctl` CLI](https://cluster-api.sigs.k8s.io/user/quick-start.html#install-clusterctl). 2. Install the vCluster provider: ``` clusterctl init --infrastructure vcluster:v0.2.0 ``` 3. Export environment variables for the Cluster API provider to create the manifest. The manifest is applied to your Kubernetes cluster, which deploys a vCluster. Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` export CLUSTER_NAME=my-vcluster export CLUSTER_NAMESPACE=team-x export VCLUSTER_YAML=$(awk '{printf "%s\n", $0}' vcluster.yaml) ``` 4. Create the namespace for the vCluster using the exported variable: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x ``` kubectl create namespace team-x ``` 5. Generate the required manifests and apply them using the exported variables: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CLUSTER NAMESPACEteam-x ``` clusterctl generate cluster my-vcluster \ --infrastructure vcluster \ --target-namespace team-x \ | kubectl apply -f - ``` Kubernetes version The Kubernetes version for the vCluster is not set at the CAPI provider command. Configure it in the `vcluster.yaml` file based on your Kubernetes distribution. 6. Wait for vCluster to become ready by monitoring the vCluster custom resource status: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x CLUSTER NAMEmy-vcluster ``` kubectl wait --for=condition=ready vcluster -n team-x my-vcluster --timeout=300s ``` ## Next steps[​](#next-steps "Direct link to Next steps") Now that you have vCluster running on OpenShift, consider setting up the [platform UI](https://vcluster.com/docs/platform/install/quick-start-guide.md) to manage your tenant clusters. --- # Deploy in high availability By default, vCluster runs one instance of each of its components. This deployment method is recommended for any use cases that are very ephemeral, such as dev environments and CI/CD, but for production use cases, it's recommended to run vCluster with more redundancy. We recommend deploying vCluster in high availability (HA) to run multiple copies of the vCluster components so that the tenant cluster is more resistant to failures. HA protects a tenant cluster from failures such as a single vCluster control plane pod or eligible node becoming unavailable. It does not protect against a full outage of the Kubernetes cluster that hosts the vCluster control plane. For more information, see [What happens during control plane outages?](https://vcluster.com/docs/vcluster/understand/control-plane-outages.md). ## Enable high availability[​](#enable-high-availability "Direct link to Enable high availability") When running vCluster with high availability, the default backing store option (that is, embedded database) **must be changed** to any of the other [backing store options](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store.md). ### Deployed etcd YAML example[​](#deployed-etcd-yaml-example "Direct link to Deployed etcd YAML example") Running in HA with deployed etcd ``` controlPlane: # Deploy etcd with 3 replicas backingStore: etcd: deploy: enabled: true statefulSet: highAvailability: replicas: 3 # Deploy vCluster with 3 replicas statefulSet: highAvailability: replicas: 3 ``` ### Embedded etcd YAML example[​](#embedded-etcd-yaml-example "Direct link to Embedded etcd YAML example") Running in HA with embedded etcd ``` controlPlane: # Deploy etcd with 3 replicas backingStore: etcd: embedded: enabled: true # Deploy vCluster with 3 replicas statefulSet: highAvailability: replicas: 3 ``` ## Other predeployment configuration options[​](#other-predeployment-configuration-options "Direct link to Other predeployment configuration options") Before deploying, it's recommended to review the set of configuration options that cannot be updated post deployment. These options require deploying a new vCluster instead of upgrading your vCluster with new options. ### Control Plane Options[​](#control-plane-options "Direct link to Control Plane Options") Decide the various options of how you want your control plane deployed: * **[High availability](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/high-availability.md)** - Run multiple copies of vCluster components. * **[Rootless mode](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/rootless-mode.md)** - Deploy the vCluster pod without root access to the control plane cluster. * **Backing Store** - Decide how the data of your cluster is stored. Backing store options vCluster supports etcd or a relational database (using KINE) as the backend. This feature provides flexibility to vCluster operators. The available data store options allow you to select a data store that fits your use case. vCluster supports the following datastore options: * [Embedded SQLite](https://www.sqlite.org/index.html) (default with `PersistentVolume` (PV)) * [PostgreSQL](https://www.postgresql.org/) * [MySQL](https://www.mysql.com/) * [MariaDB](https://mariadb.org/) * [etcd](https://etcd.io/) warning After deploying your vCluster, there are limited migration paths to change your backing store. Review the backing store migration options before deploying. **Backing store options** * Embedded SQLite (Default) * Embedded SQLite (No PV) * Embedded etcd * Deployed etcd * MySQL / MariaDB * PostgreSQL This is the default, so you don't need to configure anything. If you want to explicitly set this option, you can use: ``` controlPlane: backingStore: database: embedded: enabled: true ``` By default, vCluster stores its data in a `PersistentVolumeClaim` (PVC). Alternatively, you can use an [`emptyDir`](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume to store tenant cluster data. To use an `emptyDir` to store the data instead of a `PersistentVolume`, create a `values.yaml` with the following contents: ``` controlPlane: statefulSet: persistence: volumeClaim: enabled: true ``` Then upgrade or recreate the vCluster with: ``` vcluster create my-vcluster -n my-vcluster --upgrade -f values.yaml ``` Potential data loss This method should only be used for testing purposes, as data is lost upon pod recreation. This is an enterprise feature that allows you to deploy etcd within each vCluster to enable high availability (HA), which isn’t supported with embedded SQLite: ``` controlPlane: backingStore: etcd: embedded: enabled: true ``` This deploys an etcd instance outside of the vCluster control plane pod that is used as a backing store: ``` controlPlane: backingStore: etcd: deploy: enabled: true ``` warning vCluster snapshots aren't supported with this backing store. See [Back up and restore](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md#back-up-and-restore). The option for MySQL and MariaDB typically has the following format: ``` controlPlane: backingStore: database: external: enabled: true dataSource: mysql://username:password@tcp(hostname:3306)/database-name ``` If you specify a database name and it does not exist, the server attempts to create it. The option for PostgreSQL typically has the following format: ``` controlPlane: backingStore: database: external: enabled: true dataSource: postgres://username:password@hostname:port/database-name ``` More advanced configuration parameters are available. For more information, see . If you specify a database name and it does not exist, the server attempts to create it. ### Worker Nodes[​](#worker-nodes "Direct link to Worker Nodes") Decide where you want your worker nodes to come from: * **[Nodes from the control plane cluster](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes.md)** - (Default) All worker nodes of the shared control plane cluster are used by the tenant cluster and all resources are synced to the single namespace that the vCluster is deployed on. * **[Syncing Namespaces](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md)** - Resources are synced to mapped namespaces on the control plane cluster. * **[Isolated workloads](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes/isolated-workloads.md)** - Different options to isolate a workload in a vCluster. * **[Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md)** - Enable adding individual nodes to the tenant cluster. ## Deploy vCluster with high availability[​](#deploy-vcluster-with-high-availability "Direct link to Deploy vCluster with high availability") YAML configuration If you're not sure which options to configure, you can update most settings later by upgrading your vCluster with an updated `vcluster.yaml`. However, some settings — such as what type of worker nodes or the backing store — can only be set during the initial deployment and cannot be changed during an upgrade. All of the deployment options below have the following assumptions: * A `vcluster.yaml` is provided. Refer to the `vcluster.yaml` reference docs to explore all configuration options. This file is optional and can be removed from the examples. * The vCluster is called `my-vcluster`. * The vCluster is deployed into the `team-x` namespace.
* vCluster CLI * Helm * Terraform * ArgoCD * Flux * Cluster API The vCluster CLI provides the most straightforward way to deploy and manage virtual clusters. 1. Install the vCluster CLI: * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` 2. Deploy vCluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` vcluster create my-vcluster --namespace team-x --values vcluster.yaml ``` note After installation, vCluster automatically switches your Kubernetes context to the new virtual cluster. You can now run `kubectl` commands against the virtual cluster. Helm provides fine-grained control over the deployment process and integrates well with existing Helm-based workflows. 1. Deploy vCluster using the `helm upgrade` command: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CONFIG FILEvcluster.yaml NAMESPACEteam-x ``` helm upgrade --install my-vcluster vcluster \ --values vcluster.yaml \ --repo https://charts.loft.sh \ --namespace team-x \ --repository-config='' \ --create-namespace ``` You can use Terraform to deploy vCluster as code with version control and state management. 1. Create a `main.tf` file to define your vCluster deployment using the Terraform Helm provider: ``` provider "helm" { kubernetes = { config_path = "~/.kube/config" } } resource "helm_release" "my_vcluster" { name = "my-vcluster" namespace = "team-x" create_namespace = true repository = "https://charts.loft.sh" chart = "vcluster" # If you didn't create a vcluster.yaml, remove the values section. values = [ file("${path.module}/vcluster.yaml") ] } ``` Helm Provider Version This configuration uses the Terraform Helm provider v3.x syntax where `kubernetes` is defined as an argument (`kubernetes = {`). If you're using Helm provider v2.x, use the block syntax instead (`kubernetes {`). To use v3.x, ensure your provider version is at least v3.0.0. 2. Install the required [Helm provider](https://registry.terraform.io/providers/hashicorp/helm/latest) and initialize Terraform: ``` terraform init ``` 3. Generate a plan to preview the changes: ``` terraform plan ```
Review the plan output to verify connectivity and proposed changes. 4. Deploy vCluster: ``` terraform apply ``` ArgoCD deployment enables GitOps workflows for vCluster management, and provides automated deployment, drift detection, and declarative configuration management through Git repositories. To deploy vCluster using ArgoCD, you need the following files: * `vcluster.yaml` for your vCluster configuration options. * `-app.yaml` for your ArgoCD `Application` definition. Replace `` with your actual cluster name.
1. Create the ArgoCD `Application` file `-app.yaml`, which references the vCluster Helm chart: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x ``` --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-vcluster namespace: argocd spec: project: default source: chart: vcluster repoURL: https://charts.loft.sh helm: releaseName: my-vcluster valueFiles: - vcluster.yaml destination: server: https://kubernetes.default.svc namespace: team-x ``` 2. Commit and push these files to your configured ArgoCD repository. 3. Sync your ArgoCD repository with your configured cluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster ``` argocd app sync my-vcluster ``` 1) Create a `HelmRepository` source in the Git repository that Flux monitors, so Flux can fetch the vCluster Helm charts automatically. In this example, the related files are stored under the `clusters` folder. Save the following file as `clusters/sources/vcluster-repository.yaml`: Modify the following with your specific values to generate a copyable command: REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system ``` --- apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository metadata: name: vcluster namespace: flux-system spec: interval: 1h url: https://charts.loft.sh ``` 2) The vCluster will be deployed into the namespace `team-x` of the host cluster. If this namespace doesn’t already exist, create it with: Modify the following with your specific values to generate a copyable command: NAMESPACEteam-x ``` kubectl create namespace team-x ``` 3) Create a vCluster `HelmRelease` file in your Git repository. This `HelmRelease` tells Flux how to deploy a vCluster in your Kubernetes cluster using the configured Helm charts in Step 1. Save the following file as `clusters/production/vcluster-demo.yaml`: Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system SECRET NAMEvcluster-flux-kubeconfig ``` --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: vcluster-demo namespace: team-x spec: interval: 10m chart: spec: chart: vcluster version: "0.28.x" sourceRef: kind: HelmRepository name: vcluster namespace: flux-system values: # Configure TLS SAN for the certificate controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" coredns: enabled: true exportKubeConfig: # Set a meaningful context name context: default # Use a server URL that matches the TLS SAN server: https://vcluster-demo.team-x.svc.cluster.local:443 # Skip TLS verification when Flux connects to the vCluster insecure: true # Specify the secret where the KubeConfig is stored secret: name: vcluster-flux-kubeconfig sync: toHost: ingresses: enabled: true ``` The content in `vcluster.yaml` should be placed under the `values`. Unlike ArgoCD, Flux requires the KubeConfig secret to be able to access the vCluster's API server and deploy to vCluster. The `exportKubeConfig` configuration under `values`: * Exports the virtual cluster KubeConfig as a Secret in the host namespace * Makes the Secret available for Flux to use with the `spec.kubeConfig` field * Uses a server URL that is accessible from the Flux controllers (replace `vcluster-name` and `team-x` with your actual values) * Sets `insecure: true` to automatically skip TLS certificate verification * Adds a TLS SAN (Subject Alternative Name) that matches the server URL, which helps prevent certificate verification errors KubeConfig Secret key The vCluster `exportKubeConfig` configuration creates a Secret with the KubeConfig data stored under the key `config`. When referring to this Secret in Flux resources, you must specify this key in the `secretRef.key` field, as shown in the examples below. Modify the following with your specific values to generate a copyable command: SECRET NAMEvcluster-flux-kubeconfig ``` # In Flux HelmRelease spec: kubeConfig: secretRef: name: vcluster-flux-kubeconfig key: config # Must match the key used in the vCluster-generated Secret ``` Certificate verification considerations When using vCluster with Flux, proper TLS certificate configuration is essential: 1. Set `exportKubeConfig.insecure: true` in your vCluster configuration 2. Configure proper TLS SANs with the `--tls-san` flag in vCluster configuration 3. Ensure the server URL matches the certificate's SAN Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x ``` # In your vCluster configuration controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" exportKubeConfig: server: https://vcluster-demo.team-x.svc.cluster.local:443 insecure: true ``` 4) After adding the `HelmRelease` and any supporting files to your Git repository, commit and push them: Modify the following with your specific values to generate a copyable command: FOLDER NAMEclusters ``` git add clusters/ git commit -m "Add vCluster demo configuration" git push ``` Once the changes are pushed, Flux will automatically detect them and deploy the vCluster according to the configuration in your repository. Cluster API (CAPI) provides lifecycle management for Kubernetes clusters. The vCluster CAPI provider enables you to manage virtual clusters using the same declarative APIs and tooling used for physical clusters. For more details, see the [Cluster API Provider for vCluster documentation](https://github.com/loft-sh/cluster-api-provider-vcluster). 1. Install the [`clusterctl` CLI](https://cluster-api.sigs.k8s.io/user/quick-start.html#install-clusterctl). 2. Install the vCluster provider: ``` clusterctl init --infrastructure vcluster:v0.2.0 ``` 3. Export environment variables for the Cluster API provider to create the manifest. The manifest is applied to your Kubernetes cluster, which deploys a vCluster. Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` export CLUSTER_NAME=my-vcluster export CLUSTER_NAMESPACE=team-x export VCLUSTER_YAML=$(awk '{printf "%s\n", $0}' vcluster.yaml) ``` 4. Create the namespace for the vCluster using the exported variable: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x ``` kubectl create namespace team-x ``` 5. Generate the required manifests and apply them using the exported variables: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CLUSTER NAMESPACEteam-x ``` clusterctl generate cluster my-vcluster \ --infrastructure vcluster \ --target-namespace team-x \ | kubectl apply -f - ``` Kubernetes version The Kubernetes version for the vCluster is not set at the CAPI provider command. Configure it in the `vcluster.yaml` file based on your Kubernetes distribution. 6. Wait for vCluster to become ready by monitoring the vCluster custom resource status: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x CLUSTER NAMEmy-vcluster ``` kubectl wait --for=condition=ready vcluster -n team-x my-vcluster --timeout=300s ``` ## Check the pods of the vCluster on the control plane cluster[​](#check-the-pods-of-the-vcluster-on-the-control-plane-cluster "Direct link to Check the pods of the vCluster on the control plane cluster") Ensure that your kubecontext is set to the control plane cluster. warning If you've used the vCluster CLI, your kubecontext automatically switches and connects to the tenant cluster. You can get the pods in the namespace of the tenant cluster. The namespace is called `vcluster-my-vcluster` in the control plane cluster, and contains the components of the vCluster. ``` kubectl get pods -n vcluster-my-vcluster ``` Example using the deployed etcd yaml: ``` NAME READY STATUS RESTARTS AGE my-vcluster-7c5c5844c5-27j2v 0/1 Running 0 20s my-vcluster-7c5c5844c5-gb2sm 0/1 Running 0 20s my-vcluster-7c5c5844c5-pwn7k 0/1 Running 0 20s my-vcluster-etcd-0 0/1 Running 0 20s my-vcluster-etcd-1 0/1 Running 0 20s my-vcluster-etcd-2 0/1 Running 0 20s ``` There are three replicas running of each component (control plane and etcd) of the vCluster. If one API server pod goes down, the vCluster continues working. To see which nodes in the control plane cluster that these pods were scheduled on, add the `-o wide` flag to the `kubectl get pods` command: ``` kubectl get pods -n vcluster-my-vcluster -o wide ``` The hostnames of the nodes will be listed in the `NODES` column. --- # Deploy vCluster in an air-gapped environment Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. This document explains how to deploy vCluster in environments without internet access, known as air-gapped environments. important Enterprise features require the vCluster Platform. If you deploy a vCluster in an air-gapped environment and want to use enterprise features, you must also deploy the platform in air-gapped mode and connect the vCluster to it. A standalone air-gapped vCluster does not support any enterprise capabilities. ## Overview[​](#overview "Direct link to Overview") When deploying vCluster, there are artifacts that are typically accessed using an internet connection, but without access to the internet, these artifacts need to be available to the Kubernetes cluster through a private registry: * **vCluster Helm chart**—Typically retrieved from the [vCluster Labs Helm chart repository](https://charts.loft.sh/). * **Container images referenced in the Helm chart**—Usually pulled from various public container registries. After uploading the required artifacts to your private registry, create the `vcluster.yaml` configuration file and prepare the control plane cluster for deployment. warning When using tenant clusters in air-gapped environments, the `config.experimental.deploy.vcluster.helm` [configuration setting](https://vcluster.com/docs/vcluster/configure/vcluster-yaml#experimental-deploy-vcluster) does not work with external Helm repositories since they cannot be accessed. This means custom Helm charts from repositories like `charts.bitnami.com` cannot be used for tenant cluster deployments. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") note You must have the vCluster Platform deployed in your air-gapped environment to use this guide. To prepare your environment for air-gapped deployment, see the [platform air-gapped installation guide](https://vcluster.com/docs/platform/install/air-gapped.md). ### Registry requirements[​](#registry-requirements "Direct link to Registry requirements") * OCI-compliant private registry with a `/charts` folder - A private registry accessible to both the control plane cluster and a separate, internet-connected machine. * Ability to push images and Helm charts to your private registry. ### Deployment requirements[​](#deployment-requirements "Direct link to Deployment requirements") Ensure you have the following: * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Run the command `kubectl auth can-i create clusterrole -A` to verify that your current kube-context has administrative privileges. info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm`: Helm v3.10 is required for deploying the platform. Refer to the [Helm Installation Guide](https://Helm.sh/docs/intro/install/) if you need to install it. * `kubectl`: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. - Access to pull the Helm chart and images from the private registry - On the internet-connected machine for populating the registry: * `wget` installed * `docker` installed * You are logged into GitHub Container Registry ## Populate images to a private registry[​](#populate-images-private-registry "Direct link to Populate images to a private registry") Each vCluster release includes multiple assets to help you upload the images to your private registry. * `images.txt` - The required images to run vCluster, which assumes using the default Kubernetes version. * `images-optional.txt` - An optional set of images needed for additional features, including different Kubernetes distributions and versions. * `download-images.sh` - A bash script that quickly iterates over all the images files to pull them and package them into a tarball to a machine that has internet access. * `push-images.sh` - A bash script that takes the tarball generated from the download script to push them to your private registry. ### Pull and push images[​](#pull-push-images "Direct link to Pull and push images") 1. Set environment variables for the version of vCluster that you want to deploy and the private registry. Export environment variables ``` export VCLUSTER_VERSION=0.25.0 # Replace with the desired version export REGISTRY=ecr.io/myteam # This should be a prefix; do not include any image paths ``` 2. Download the assets from the [vCluster GitHub release](https://github.com/loft-sh/vcluster/releases) and make the scripts executable. note The `images.txt` contains all distributions for the default Kubernetes version. You can edit the file and remove the images for unused distributions. Download assets and prepare scripts ``` wget https://github.com/loft-sh/vcluster/releases/download/v"${VCLUSTER_VERSION}"/images.txt wget https://github.com/loft-sh/vcluster/releases/download/v"${VCLUSTER_VERSION}"/download-images.sh wget https://github.com/loft-sh/vcluster/releases/download/v"${VCLUSTER_VERSION}"/push-images.sh chmod +x ./download-images.sh chmod +x ./push-images.sh ``` 3. Run `download-images.sh` to pull all images and create a tarball of the images. Review the output to confirm all images were pulled successfully and packaged in the tarball. Download and package images ``` ./download-images.sh --image-list images.txt ``` Multi-platform support If you need to support multiple CPU architectures (e.g., AMD64 and ARM64), use the `--platforms` flag to specify which platforms to download. This is useful when your cluster has nodes with different architectures. Download images for multiple platforms ``` ./download-images.sh --platforms "linux/amd64,linux/arm64" --image-list images.txt ``` 4. Run `push-images.sh` to upload all required images to your private registry. When pushing images into your private registry, the public private registry is removed and only the repository and image name are pushed. This allows vCluster to set your private registry to use for all images used in deploying vCluster. Push images to private registry ``` ./push-images.sh --registry ${REGISTRY} ``` Additional push options **Multi-platform support**: If you downloaded images for multiple platforms, specify the same `--platforms` flag when pushing: ``` ./push-images.sh --registry ${REGISTRY} --platforms "linux/amd64,linux/arm64" ``` **Insecure registry**: If your private registry uses HTTP instead of HTTPS, add the `--insecure` flag: ``` ./push-images.sh --registry ${REGISTRY} --insecure ``` 5. **Optional**: If you need additional features (such as different Kubernetes versions or distributions), download the `images-optional.txt` file. It contains extra container images required for these features, which you'll need to pull and push to your private registry. The `images-optional.txt` includes images for various optional configurations. You can edit the file and keep only the images you need, removing those for unused features or versions. note The `images-optional.txt` may include multiple Kubernetes distributions, versions, or other feature-dependent images. You can edit the file to retain only what's necessary for your deployment. Pull, package, and push all optional images ``` wget https://github.com/loft-sh/vcluster/releases/download/v${VCLUSTER_VERSION}/images-optional.txt ./download-images.sh --image-list images-optional.txt --images vcluster-images-optional.tar.gz ./push-images.sh --registry ${REGISTRY} --images vcluster-images-optional.tar.gz ``` tip You can also use the `--platforms` and `--insecure` flags with these commands if needed, as described in the previous steps. ## Populate the vCluster Helm chart to a private registry[​](#populate-vCluster-Helm-chart-private-registry "Direct link to Populate the vCluster Helm chart to a private registry") You must push the vCluster Helm chart to your OCI-compliant private registry. ### Pull and push the vCluster Helm chart[​](#download-push-vCluster-Helm-chart "Direct link to Pull and push the vCluster Helm chart") 1. **Optional**: If you haven’t already set the environment variables, set them now before continuing. The private registry assumes having a `/charts` folder, which is where to push all the Helm charts. Export environment variables ``` export VCLUSTER_VERSION=0.25.0 # Replace with the desired version export REGISTRY=ecr.io/myteam # A charts folder is expected ``` note The private registry assumes having a `/charts` folder, which is where to push all the Helm charts. 2. Pull the vCluster Helm chart and push it into your private registry with the OCI protocol. Pull and push the Helm chart to your private registry ``` helm pull vcluster --repo https://charts.loft.sh --version ${VCLUSTER_VERSION} helm push vcluster-${VCLUSTER_VERSION}.tgz oci://${REGISTRY}/charts ``` ## Configure vCluster[​](#configure-vCluster "Direct link to Configure vCluster") The `vcluster.yaml` file contains all configuration settings for your vCluster deployment. Use a private registry without credentials Set the default private registry that doesn't have authentication. Setting the private registry ``` controlPlane: advanced: defaultImageRegistry: ecr.io/myteam # Replace with your private registry ``` Use an authenticated private registry For registries that require authentication, create a Kubernetes secret in the namespace where you deploy the vCluster. Assign the secret as an image pull secret for the vCluster control plane to access required images. Optionally, you can use the same image pull secret for workloads inside the vCluster that pull from the same registry. Image pull secrets configuration ``` controlPlane: advanced: defaultImageRegistry: ecr.io/myteam # Replace with your private image registry serviceAccount: imagePullSecrets: # Uses credentials for the vCluster control plane - name: registry-credentials-secret-name # Replace with the name of the secret deployed on the host cluster of where the vCluster is deployed workloadServiceAccount: # Uses credentials for any workloads created in the vCluster imagePullSecrets: - name: registry-credentials-secret-name # Replace with the name of the secret deployed on the host cluster of where your vCluster is deployed ``` Use a non-default Kubernetes version If you downloaded `images-optional.txt` to use additional features (such as a different Kubernetes version), configure vCluster to use those features in your deployment. Specific Kubernetes version configuration ``` controlPlane: distro: k8s: image: tag: v1.31.1 # Replace with the Kubernetes version that you have chosen ``` Replace the alpine image If you use the `sync.toHost.pods.rewriteHosts` feature, manually replace the full path of the [alpine image](https://hub.docker.com/_/alpine) with your private registry path. The `defaultImageRegistry` setting does not apply to this image. Replace the alpine image ``` sync: toHost: pods: rewriteHosts: initContainer: image: your-registry/library/alpine:3.20 ``` ### Other predeployment configuration options[​](#other-predeployment-configuration-options "Direct link to Other predeployment configuration options") Before deploying, it's recommended to review the set of configuration options that cannot be updated post deployment. These options require deploying a new vCluster instead of upgrading your vCluster with new options. ### Control Plane Options[​](#control-plane-options "Direct link to Control Plane Options") Decide the various options of how you want your control plane deployed: * **[High availability](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/high-availability.md)** - Run multiple copies of vCluster components. * **[Rootless mode](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/rootless-mode.md)** - Deploy the vCluster pod without root access to the control plane cluster. * **Backing Store** - Decide how the data of your cluster is stored. Backing store options vCluster supports etcd or a relational database (using KINE) as the backend. This feature provides flexibility to vCluster operators. The available data store options allow you to select a data store that fits your use case. vCluster supports the following datastore options: * [Embedded SQLite](https://www.sqlite.org/index.html) (default with `PersistentVolume` (PV)) * [PostgreSQL](https://www.postgresql.org/) * [MySQL](https://www.mysql.com/) * [MariaDB](https://mariadb.org/) * [etcd](https://etcd.io/) warning After deploying your vCluster, there are limited migration paths to change your backing store. Review the backing store migration options before deploying. **Backing store options** * Embedded SQLite (Default) * Embedded SQLite (No PV) * Embedded etcd * Deployed etcd * MySQL / MariaDB * PostgreSQL This is the default, so you don't need to configure anything. If you want to explicitly set this option, you can use: ``` controlPlane: backingStore: database: embedded: enabled: true ``` By default, vCluster stores its data in a `PersistentVolumeClaim` (PVC). Alternatively, you can use an [`emptyDir`](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume to store tenant cluster data. To use an `emptyDir` to store the data instead of a `PersistentVolume`, create a `values.yaml` with the following contents: ``` controlPlane: statefulSet: persistence: volumeClaim: enabled: true ``` Then upgrade or recreate the vCluster with: ``` vcluster create my-vcluster -n my-vcluster --upgrade -f values.yaml ``` Potential data loss This method should only be used for testing purposes, as data is lost upon pod recreation. This is an enterprise feature that allows you to deploy etcd within each vCluster to enable high availability (HA), which isn’t supported with embedded SQLite: ``` controlPlane: backingStore: etcd: embedded: enabled: true ``` This deploys an etcd instance outside of the vCluster control plane pod that is used as a backing store: ``` controlPlane: backingStore: etcd: deploy: enabled: true ``` warning vCluster snapshots aren't supported with this backing store. See [Back up and restore](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md#back-up-and-restore). The option for MySQL and MariaDB typically has the following format: ``` controlPlane: backingStore: database: external: enabled: true dataSource: mysql://username:password@tcp(hostname:3306)/database-name ``` If you specify a database name and it does not exist, the server attempts to create it. The option for PostgreSQL typically has the following format: ``` controlPlane: backingStore: database: external: enabled: true dataSource: postgres://username:password@hostname:port/database-name ``` More advanced configuration parameters are available. For more information, see . If you specify a database name and it does not exist, the server attempts to create it. ### Worker Nodes[​](#worker-nodes "Direct link to Worker Nodes") Decide where you want your worker nodes to come from: * **[Nodes from the control plane cluster](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes.md)** - (Default) All worker nodes of the shared control plane cluster are used by the tenant cluster and all resources are synced to the single namespace that the vCluster is deployed on. * **[Syncing Namespaces](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md)** - Resources are synced to mapped namespaces on the control plane cluster. * **[Isolated workloads](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes/isolated-workloads.md)** - Different options to isolate a workload in a vCluster. * **[Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md)** - Enable adding individual nodes to the tenant cluster. ## Create a platform access key[​](#create-platform-access-key "Direct link to Create a platform access key") Before deploying vCluster, you must [create an access key](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/platform.md) to authenticate your tenant cluster. This step is essential for connecting your air-gapped vCluster to the platform. ## Deploy vCluster[​](#deploy-vcluster "Direct link to Deploy vCluster") After the platform is deployed, the access key is created, and your private registry is populated with the required images and charts, you can proceed with deploying the vCluster. ### Set up the control plane cluster and deploy[​](#set-up-the-control-plane-cluster-and-deploy "Direct link to Set up the control plane cluster and deploy") 1. On the control plane cluster, create the namespace for the vCluster, where the secrets and vCluster control plane pod are to be deployed in: Create vCluster namespace ``` export VCLUSTER_NAMESPACE=vcluster-my-vcluster # Replace with the name of the namespace you want to deploy your vCluster into kubectl create namespace ${VCLUSTER_NAMESPACE} ``` 2. **Optional**: Create a secret to authenticate to your private registry. If your private registry requires authentication, create a Kubernetes secret that stores your login credentials. Place this secret in the same namespace where the vCluster is deployed so the control plane can pull the required container images. Create Kubernetes secret for login credentials ``` kubectl create -f - <-etcd` in the list (see [External etcd deployments](#external-etcd-deployments) section below). ## Configure proxy settings[​](#configure-proxy-settings "Direct link to Configure proxy settings") Configure the proxy environment variables through the vCluster StatefulSet configuration: vcluster.yaml ``` controlPlane: statefulSet: env: # HTTP proxy for non-encrypted traffic - name: HTTP_PROXY value: http://corp-proxy.example.com:3128 # HTTPS proxy for encrypted traffic - name: HTTPS_PROXY value: http://corp-proxy.example.com:3128 # NO_PROXY - services that should bypass the proxy # Include '-etcd' if using external etcd backing store - name: NO_PROXY value: localhost,127.0.0.1,.svc,.svc.cluster.local,.local,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 ``` ## Configure required no-proxy entries[​](#configure-required-no-proxy-entries "Direct link to Configure required no-proxy entries") ### Required entries[​](#required-entries "Direct link to Required entries") The `NO_PROXY` environment variable must include the following entries for vCluster to function correctly: | Entry | Purpose | Required | | ---------------------- | -------------------------------- | ---------------------------------- | | `localhost` | Local loopback | Yes | | `127.0.0.1` | Local loopback IP | Yes | | `.svc` | Service DNS suffix | Yes | | `.svc.cluster.local` | Full cluster DNS suffix | Yes | | Cluster CIDR ranges | Internal pod/service networks | Yes | | `-etcd` | vCluster's external etcd service | Only for external etcd deployments | ### Example configurations[​](#example-configurations "Direct link to Example configurations") * Basic configuration * Extended configuration Minimal proxy configuration with required entries: vcluster.yaml ``` controlPlane: statefulSet: env: - name: HTTP_PROXY value: http://corp-proxy.example.com:3128 - name: HTTPS_PROXY value: http://corp-proxy.example.com:3128 - name: NO_PROXY value: localhost,127.0.0.1,.svc,.svc.cluster.local,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 ``` Comprehensive configuration including additional internal services and networks with authentication: vcluster.yaml ``` controlPlane: statefulSet: env: # Proxy URLs with authentication variables - name: HTTP_PROXY value: http://$(PROXY_USERNAME):$(PROXY_PASSWORD)@corp-proxy.example.com:3128 - name: HTTPS_PROXY value: http://$(PROXY_USERNAME):$(PROXY_PASSWORD)@corp-proxy.example.com:3128 - name: NO_PROXY value: localhost,127.0.0.1,.svc,.svc.cluster.local,.local,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16,.example.internal # Proxy credentials from Kubernetes secret - name: PROXY_USERNAME valueFrom: secretKeyRef: name: proxy-credentials key: username - name: PROXY_PASSWORD valueFrom: secretKeyRef: name: proxy-credentials key: password ``` ## Deploy vCluster with proxy settings[​](#deploy-vcluster-with-proxy-settings "Direct link to Deploy vCluster with proxy settings") 1. Create a namespace for your vCluster: ``` kubectl create namespace vcluster-proxy ``` 2. Create your `vcluster.yaml` configuration file with the proxy settings: Modify the following with your specific values to generate a copyable command: HTTP\_PROXY\_URLhttp\://corp-proxy.example.com:3128 HTTPS\_PROXY\_URLhttp\://corp-proxy.example.com:3128 VCLUSTER\_NAMEdemo-vcluster CLUSTER\_CIDR10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 ``` controlPlane: statefulSet: env: - name: HTTP_PROXY value: http://corp-proxy.example.com:3128 - name: HTTPS_PROXY value: http://corp-proxy.example.com:3128 - name: NO_PROXY # Include vCluster etcd service explicitly for external etcd deployments value: demo-vcluster-etcd,localhost,127.0.0.1,.svc,.svc.cluster.local,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 ``` 3. (Optional) If your proxy requires authentication, create a secret with the credentials: ``` kubectl create secret generic proxy-credentials \ --namespace vcluster-proxy \ --from-literal=username='your-proxy-username' \ --from-literal=password='your-proxy-password' ``` Then update your `vcluster.yaml` to reference the secret: Modify the following with your specific values to generate a copyable command: PROXY\_HOSTcorp-proxy.example.com:3128 VCLUSTER\_NAMEdemo-vcluster CLUSTER\_CIDR10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 ``` controlPlane: statefulSet: env: # Proxy URLs with authentication - name: HTTP_PROXY value: http://$(PROXY_USERNAME):$(PROXY_PASSWORD)@corp-proxy.example.com:3128 - name: HTTPS_PROXY value: http://$(PROXY_USERNAME):$(PROXY_PASSWORD)@corp-proxy.example.com:3128 - name: NO_PROXY value: demo-vcluster-etcd,localhost,127.0.0.1,.svc,.svc.cluster.local,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 # Proxy credentials from secret - name: PROXY_USERNAME valueFrom: secretKeyRef: name: proxy-credentials key: username - name: PROXY_PASSWORD valueFrom: secretKeyRef: name: proxy-credentials key: password ``` 4. Deploy vCluster with the configuration: ``` vcluster create my-vcluster \ --namespace vcluster-proxy \ --values vcluster.yaml ``` 5. Verify that the proxy settings are applied: ``` kubectl get statefulset -n vcluster-proxy my-vcluster -o yaml | grep -A 5 "env:" ``` You should see your proxy environment variables in the output. ## Troubleshoot proxy issues[​](#troubleshoot-proxy-issues "Direct link to Troubleshoot proxy issues") ### vCluster fails to start with etcd connection errors[​](#vcluster-fails-to-start-with-etcd-connection-errors "Direct link to vCluster fails to start with etcd connection errors") For vCluster deployments using external etcd, if you see errors like "failed to connect to etcd" or proxy logs showing `TCP_DENIED/403` for etcd connections on port 2379, verify that: 1. The etcd service name (`-etcd`) is explicitly listed in your `NO_PROXY` configuration 2. The `NO_PROXY` value doesn't have spaces between entries (use commas only) 3. The environment variables are correctly applied to the StatefulSet 4. You're using the correct service name format (check with `kubectl get svc -n `) ### Verify proxy configuration[​](#verify-proxy-configuration "Direct link to Verify proxy configuration") Check that the proxy settings are correctly applied to the vCluster pods: ``` # Check the environment variables in the running pod kubectl exec -n vcluster-proxy my-vcluster-0 -- env | grep -E "PROXY|proxy" ``` ### Test connectivity[​](#test-connectivity "Direct link to Test connectivity") Verify that internal services bypass the proxy: ``` # Test etcd connectivity from within the vCluster pod kubectl exec -n vcluster-proxy my-vcluster-0 -- curl -I http://my-vcluster-etcd:2379/health ``` ## External etcd deployments[​](#external-etcd-deployments "Direct link to External etcd deployments") When using external etcd as the backing store for vCluster instead of the default embedded SQLite, you **must** include the etcd service name explicitly in the `NO_PROXY` environment variable. The service name follows the pattern `-etcd`. This requirement is critical because: * The Go HTTP client used by vCluster requires exact hostname matches for services without a leading dot * Domain patterns like `.local` or `.svc.cluster.local` do not cover the etcd service name * Without this explicit entry, vCluster will route etcd connections through the proxy, causing connection failures ### Configure proxy for external etcd[​](#configure-proxy-for-external-etcd "Direct link to Configure proxy for external etcd") vcluster-external-etcd.yaml ``` # External etcd configuration with proxy settings controlPlane: # Use external etcd as backing store backingStore: etcd: deploy: enabled: true statefulSet: highAvailability: replicas: 3 # Proxy configuration for control plane statefulSet: env: - name: HTTP_PROXY value: http://corp-proxy.example.com:3128 - name: HTTPS_PROXY value: http://corp-proxy.example.com:3128 # CRITICAL: Must include the etcd service name # Pattern: -etcd (e.g., my-vcluster-etcd) - name: NO_PROXY value: my-vcluster-etcd,localhost,127.0.0.1,.svc,.svc.cluster.local,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 ``` Critical for external etcd When using external etcd, omitting the etcd service name (format: `-etcd`) from `NO_PROXY` will cause vCluster to fail with connection errors to the etcd service. The proxy will deny these connections with 403 errors. This is the most common cause of proxy-related failures in external etcd deployments. ## Additional considerations[​](#additional-considerations "Direct link to Additional considerations") ### Use authenticated proxies[​](#use-authenticated-proxies "Direct link to Use authenticated proxies") If your corporate proxy requires authentication, use Kubernetes secrets to securely manage credentials: * Using secrets (recommended) * Using URL in secret * Inline (not recommended) First, create a Kubernetes secret with your proxy credentials: ``` kubectl create secret generic proxy-credentials \ --namespace \ --from-literal=username='your-proxy-username' \ --from-literal=password='your-proxy-password' ``` Then reference the secret in your vCluster configuration: vcluster.yaml ``` controlPlane: statefulSet: env: # Proxy URLs with authentication variables - name: HTTP_PROXY value: http://$(PROXY_USERNAME):$(PROXY_PASSWORD)@corp-proxy.example.com:3128 - name: HTTPS_PROXY value: http://$(PROXY_USERNAME):$(PROXY_PASSWORD)@corp-proxy.example.com:3128 - name: NO_PROXY value: localhost,127.0.0.1,.svc,.svc.cluster.local,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 # Proxy credentials from secret - name: PROXY_USERNAME valueFrom: secretKeyRef: name: proxy-credentials key: username - name: PROXY_PASSWORD valueFrom: secretKeyRef: name: proxy-credentials key: password ``` Store the complete proxy URL with credentials in a secret: ``` kubectl create secret generic proxy-config \ --namespace \ --from-literal=http-proxy-url='http://username:password@corp-proxy.example.com:3128' \ --from-literal=https-proxy-url='http://username:password@corp-proxy.example.com:3128' ``` Reference the secret in your configuration: vcluster.yaml ``` controlPlane: statefulSet: env: - name: HTTP_PROXY valueFrom: secretKeyRef: name: proxy-config key: http-proxy-url - name: HTTPS_PROXY valueFrom: secretKeyRef: name: proxy-config key: https-proxy-url - name: NO_PROXY value: localhost,127.0.0.1,.svc,.svc.cluster.local,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 ``` Security risk Hardcoding credentials in configuration files is not recommended as they can be exposed in logs, version control, or to unauthorized users. vcluster.yaml ``` controlPlane: statefulSet: env: - name: HTTP_PROXY value: http://username:password@corp-proxy.example.com:3128 - name: HTTPS_PROXY value: http://username:password@corp-proxy.example.com:3128 - name: NO_PROXY value: localhost,127.0.0.1,.svc,.svc.cluster.local,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 ``` ### Proxy settings for vCluster workloads[​](#proxy-settings-for-vcluster-workloads "Direct link to Proxy settings for vCluster workloads") The proxy configuration shown previously applies only to the vCluster control plane. If you need proxy settings for workloads running inside the tenant cluster, configure them separately through: * Pod environment variables in your workload manifests * ConfigMaps or Secrets mounted to your workloads * Admission webhooks that inject proxy settings automatically ## Related topics[​](#related-topics "Direct link to Related topics") * [Deploy vCluster in an air-gapped environment](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/air-gapped.md) * [StatefulSet configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/deployment/statefulset.md) * [vCluster networking](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking.md) --- # FIPS 140-2 enablement | | | Enterprise | | | | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [FIPS](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/fips.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. The [National Institute of Standards and Technology (NIST)](https://www.nist.gov/federal-information-processing-standards-fips) develops Federal Information Processing Standards (FIPS) to ensure the security and interoperability of computer systems used by the U.S. government. FIPS 140-2 is a U.S. Federal Government security standard used to approve cryptographic modules. This document explains how vCluster Standalone and all its components are built with FIPS-validated cryptographic libraries. ## Use of FIPS compatible Go toolchain[​](#use-of-fips-compatible-go-toolchain "Direct link to Use of FIPS compatible Go toolchain") vCluster is written in [Go](https://go.dev/), and the FIPS-compliant builds are compiled using the `GOFIPS140=v1.0.0` environment variable. The FIPS 140-3 validated Go Cryptographic Module now underlies Go’s built-in crypto libraries, starting with the Go Cryptographic Module v1.0.0 that is included in Go 1.24. ### FIPS support in vCluster components[​](#fips-support-in-vcluster-components "Direct link to FIPS support in vCluster components") Most components used in vCluster are statically compiled with the GOFIPS140=v1.0.0 enabled Go compiler. From a component perspective, vCluster includes multiple sub-components it depends on. The list below contains components built in a FIPS-compliant manner: * [Kubernetes API Server](https://github.com/loft-sh/fips-builds/pkgs/container/kubernetes-fips) * [Kubernetes Controller Manager](https://github.com/loft-sh/fips-builds/pkgs/container/kubernetes-fips) * [Kubernetes Scheduler](https://github.com/loft-sh/fips-builds/pkgs/container/kubernetes-fips) * [Kine](https://github.com/loft-sh/fips-builds/pkgs/container/kine-fips) * [ETCD](https://github.com/loft-sh/fips-builds/pkgs/container/etcd-fips)
caution vCluster currently does not provide FIPS-compliant builds of CoreDNS or Helm. One must use the integrated CoreDNS feature of vCluster. ### FIPS vCluster Images[​](#fips-vcluster-images "Direct link to FIPS vCluster Images") The vCluster FIPS-compliant images can be found in the vCluster [GitHub Container Registry](https://github.com/loft-sh/vcluster-pro/pkgs/container/vcluster-pro-fips). ## Run a FIPS-compliant vCluster[​](#run-a-fips-compliant-vcluster "Direct link to Run a FIPS-compliant vCluster") To run vCluster in a FIPS environment, you must reconfigure the repositories used to reference the FIPS images and enable CoreDNS. The following is an example of a `vcluster.yaml` file that one can use to create a FIPS-compliant vCluster instance: ``` controlPlane: statefulSet: image: repository: loft-sh/vcluster-pro-fips distro: k8s: image: registry: ghcr.io repository: loft-sh/kubernetes tag: v1.33.5-fips # specify Kubernetes version here with -fips suffix coredns: embedded: true backingStore: etcd: embedded: # The use of embedded etcd is recommended, yet optional enabled: true ``` To create the FIPS-compliant vCluster instance, run: ``` vcluster create my-fips-vcluster -f vcluster.yaml ``` info To use a different Kubernetes version in your tenant cluster than the control plane cluster, set the `controlPlane.distro.k8s.image.tag` field in your configuration: ``` controlPlane: distro: k8s: image: tag: v1.34.2-fips ``` --- # Control plane configuration Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** The following section covers cluster-wide security areas. The assessment focuses on authentication and authorization mechanisms and audit logging configuration. Assessment focus for vCluster involves verifying audit logging is enabled and alternative mechanisms such as OIDC are used. Key areas include ensuring that traditional authentication methods like client certificates, service account tokens, and bootstrap tokens are replaced with more secure alternatives for user authentication. Control numbering The control numbers used throughout this guide (3.1.1, 3.2.1, etc.) correlate directly to the official CIS Kubernetes Benchmark control numbers. This allows you to cross-reference with the official CIS documentation and maintain consistency with standard security frameworks. ## 3.1 Authentication and authorization[​](#31-authentication-and-authorization "Direct link to 3.1 Authentication and authorization") ### 3.1.1 Client certificate authentication should not be used for users[​](#311-client-certificate-authentication-should-not-be-used-for-users "Direct link to 3.1.1 Client certificate authentication should not be used for users") **Result:** WARN Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented in place of client certificates. Client certificate authentication, while functional, creates operational challenges around certificate lifecycle management and lacks the advanced features provided by modern authentication systems. OIDC integration allows for centralized identity management, multi-factor authentication, and easier user provisioning and deprovisioning. ### 3.1.2 Service account token authentication should not be used for users[​](#312-service-account-token-authentication-should-not-be-used-for-users "Direct link to 3.1.2 Service account token authentication should not be used for users") **Result:** WARN Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented in place of service account tokens. Service account tokens are designed for workload authentication rather than human users. Using service account tokens for user authentication bypasses important security controls and audit trails that are essential for compliance and security monitoring. ### 3.1.3 Bootstrap token authentication should not be used for users[​](#313-bootstrap-token-authentication-should-not-be-used-for-users "Direct link to 3.1.3 Bootstrap token authentication should not be used for users") **Result:** WARN Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented in place of bootstrap tokens. Bootstrap tokens are intended for node bootstrapping and initial cluster setup, not for ongoing user authentication. Using bootstrap tokens for user access creates security risks and prevents proper user identity tracking and access control. ## 3.2 Logging[​](#32-logging "Direct link to 3.2 Logging") Platform audit logging If you use vCluster Platform, you can configure audit logging centrally through the platform instead of per-cluster API server flags. See [Audit Logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md). ### 3.2.1 Ensure that a minimal audit policy is created[​](#321-ensure-that-a-minimal-audit-policy-is-created "Direct link to 3.2.1 Ensure that a minimal audit policy is created") **Result:** PASS 1. Follow the [Kubernetes documentation](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/) and create a config map with a minimal audit policy: audit-config-configmap.yaml ``` apiVersion: v1 kind: ConfigMap metadata: name: audit-config namespace: vcluster-my-vcluster data: audit-policy.yaml: | apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata ``` 2. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --audit-policy-file=/etc/kubernetes/audit-policy.yaml - --audit-log-path=/dev/stdout statefulSet: persistence: addVolumes: - name: audit-policy configMap: name: audit-config addVolumeMounts: - name: audit-policy mountPath: /etc/kubernetes ``` 3. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 4. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--audit-policy-file` is set. **Expected Result:** ``` '--audit-policy-file' is present ``` **Returned Value:** ``` 26 root 0:08 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --audit-policy-file=/etc/kubernetes/audit-policy.yaml ``` ### 3.2.2 Ensure that the audit policy covers key security concerns[​](#322-ensure-that-the-audit-policy-covers-key-security-concerns "Direct link to 3.2.2 Ensure that the audit policy covers key security concerns") **Result:** WARN Review the audit policy provided for the cluster and ensure that it covers key security areas. The audit policy should include access to Secrets managed by the cluster, taking care to only log Metadata for requests to Secrets, ConfigMaps, and TokenReviews to avoid the risk of logging sensitive data. Additionally, ensure coverage of modification of pod and deployment objects and use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. For most requests, minimally logging at the Metadata level is recommended as the most basic level of logging. Consider modification of the audit policy in use on the cluster to include these items at a minimum. The following sample policy satisfies these criteria: audit-config-configmap.yaml ``` apiVersion: v1 kind: ConfigMap metadata: name: audit-config namespace: vcluster-my-vcluster data: audit-policy.yaml: | apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata resources: - group: "" resources: ["secrets", "configmaps"] - group: "authentication.k8s.io" resources: ["tokenreviews"] - level: RequestResponse resources: - group: "" resources: ["pods"] - group: "apps" resources: ["deployments", "replicasets"] verbs: ["create", "update", "patch", "delete"] - level: RequestResponse resources: - group: "" resources: ["pods/exec", "pods/portforward", "pods/proxy", "services/proxy"] - level: Metadata omitStages: - RequestReceived ``` --- # Control plane security configuration Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** The following section covers security recommendations for the direct configuration of Kubernetes control plane processes. The assessment focuses on API Server configuration and security settings, Controller Manager security parameters, Scheduler security configurations, and general control plane security practices. Assessment focus for vCluster involves checking the extraArgs configurations in your `vcluster.yaml` file and ensuring proper security parameters are set for the virtualized API server, controller manager, and scheduler. Since vCluster virtualizes the control plane components, verification requires examining container processes rather than traditional system-level checks. Control numbering The control numbers used throughout this guide (1.1.1, 1.2.1, etc.) correlate directly to the official CIS Kubernetes Benchmark control numbers. This allows you to cross-reference with the official CIS documentation and maintain consistency with standard security frameworks. note For auditing each control, create the vCluster using default values as shown below, unless specified otherwise. ``` vcluster create my-vcluster --connect=false ``` ## 1.1 Master node configuration files[​](#11-master-node-configuration-files "Direct link to 1.1 Master node configuration files") ### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive[​](#111-ensure-that-the-api-server-pod-specification-file-permissions-are-set-to-644-or-more-restrictive "Direct link to 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/kube-apiserver.yaml` as found in kubeadm-based clusters. Instead, the vCluster API server runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file permissions check defined in this control not applicable in the context of vCluster. ### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root[​](#112-ensure-that-the-api-server-pod-specification-file-ownership-is-set-to-root "Direct link to 112-ensure-that-the-api-server-pod-specification-file-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/kube-apiserver.yaml` as found in kubeadm-based clusters. Instead, the vCluster API server runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file ownership check defined in this control not applicable in the context of vCluster. ### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive[​](#113-ensure-that-the-controller-manager-pod-specification-file-permissions-are-set-to-644-or-more-restrictive "Direct link to 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/kube-controller-manager.yaml` as found in kubeadm-based clusters. Instead, the vCluster controller manager runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file permissions check defined in this control not applicable in the context of vCluster. ### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root[​](#114-ensure-that-the-controller-manager-pod-specification-file-ownership-is-set-to-root "Direct link to 114-ensure-that-the-controller-manager-pod-specification-file-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/kube-controller-manager.yaml` as found in kubeadm-based clusters. Instead, the vCluster controller manager runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file ownership check defined in this control not applicable in the context of vCluster. ### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive[​](#115-ensure-that-the-scheduler-pod-specification-file-permissions-are-set-to-644-or-more-restrictive "Direct link to 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/kube-scheduler.yaml` as found in kubeadm-based clusters. Instead, the vCluster scheduler runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file permissions check defined in this control not applicable in the context of vCluster. ### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root[​](#116-ensure-that-the-scheduler-pod-specification-file-ownership-is-set-to-root "Direct link to 116-ensure-that-the-scheduler-pod-specification-file-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/kube-scheduler.yaml` as found in kubeadm-based clusters. Instead, the vCluster scheduler runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file ownership check defined in this control not applicable in the context of vCluster. ### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive[​](#117-ensure-that-the-etcd-pod-specification-file-permissions-are-set-to-644-or-more-restrictive "Direct link to 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/etcd.yaml` as found in kubeadm-based clusters. In case of embedded etcd, the etcd runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file permissions check defined in this control not applicable in the context of vCluster. ### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root[​](#118-ensure-that-the-etcd-pod-specification-file-ownership-is-set-to-root "Direct link to 118-ensure-that-the-etcd-pod-specification-file-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/etcd.yaml` as found in kubeadm-based clusters. In case of embedded etcd, the etcd runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file ownership check defined in this control not applicable in the context of vCluster. ### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive[​](#119-ensure-that-the-container-network-interface-file-permissions-are-set-to-644-or-more-restrictive "Direct link to 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive") **Result:** NOT APPLICABLE vCluster does not configure or manage Container Network Interface (CNI) settings. Networking is handled entirely by the control plane cluster's CNI plugin. As a result, there are no CNI configuration files such as `/etc/cni/net.d/*.conf` present within the vCluster container. You should evaluate this control on the underlying control plane cluster as it is not applicable in vCluster environments. ### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root[​](#1110-ensure-that-the-container-network-interface-file-ownership-is-set-to-root "Direct link to 1110-ensure-that-the-container-network-interface-file-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster does not configure or manage Container Network Interface (CNI) settings. Networking is handled entirely by the control plane cluster's CNI plugin. As a result, there are no CNI configuration files such as `/etc/cni/net.d/*.conf` present within the vCluster container. You should evaluate this control on the underlying control plane cluster as it is not applicable in vCluster environments. ### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive[​](#1111-ensure-that-the-etcd-data-directory-permissions-are-set-to-700-or-more-restrictive "Direct link to 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive") **Result:** PASS 1. Get the etcd data directory, passed as an argument to `--data-dir`, from the following command: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` 2. Run the audit command to verify the permissions on the data directory. If they do not match the expected result, run the following command to set the appropriate permissions: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- chmod 700 /data/etcd ``` **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- stat -c permissions=%a /data/etcd ``` Verify that the etcd data directory permissions are set to 700 or more restrictive. **Expected results:** ``` permissions has value 700, expected 700 or more restrictive ``` **Returned value:** ``` permissions=700 ``` ### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd[​](#1112-ensure-that-the-etcd-data-directory-ownership-is-set-to-etcd "Direct link to 1112-ensure-that-the-etcd-data-directory-ownership-is-set-to-etcd") **Result:** NOT APPLICABLE The control recommends that the etcd data directory be owned by the etcd user and group (etcd:etcd) to follow least privilege principles. However, in vCluster, etcd is embedded and runs as `root` within the syncer container. There is no separate etcd user present in the container. The directory ownership check defined in this control is not applicable in the context of vCluster. ### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive[​](#1113-ensure-that-the-adminconf-file-permissions-are-set-to-600-or-more-restrictive "Direct link to 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- stat -c permissions=%a /data/pki/admin.conf ``` Verify that the admin.conf file permissions are 600 or more restrictive. **Expected results:** ``` permissions has value 600, expected 600 or more restrictive ``` **Returned value:** ``` permissions=600 ``` ### 1.1.14 Ensure that the admin.conf file ownership is set to root:root[​](#1114-ensure-that-the-adminconf-file-ownership-is-set-to-root "Direct link to 1114-ensure-that-the-adminconf-file-ownership-is-set-to-root") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- stat -c %U:%G /data/pki/admin.conf ``` Verify that the admin.conf file ownership is set to root:root. **Expected results:** ``` root:root ``` **Returned value:** ``` root:root ``` ### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive[​](#1115-ensure-that-the-schedulerconf-file-permissions-are-set-to-600-or-more-restrictive "Direct link to 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive") **Result:** PASS Get the scheduler kubeconfig file, passed as an argument to `--kubeconfig`, from the following command: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-scheduler ``` **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- stat -c permissions=%a /data/pki/scheduler.conf ``` Verify that the scheduler kubeconfig file permissions are set to 600 or more restrictive. **Expected results:** ``` permissions has value 600, expected 600 or more restrictive ``` **Returned value:** ``` permissions=600 ``` ### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root[​](#1116-ensure-that-the-schedulerconf-file-ownership-is-set-to-root "Direct link to 1116-ensure-that-the-schedulerconf-file-ownership-is-set-to-root") **Result:** PASS Get the scheduler kubeconfig file, passed as an argument to `--kubeconfig`, from the following command: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-scheduler ``` **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- stat -c %U:%G /data/pki/scheduler.conf ``` Verify that the scheduler kubeconfig file ownership is set to root:root. **Expected results:** ``` root:root ``` **Returned value:** ``` root:root ``` ### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive[​](#1117-ensure-that-the-controller-managerconf-file-permissions-are-set-to-600-or-more-restrictive "Direct link to 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive") **Result:** PASS Get the controller-manager kubeconfig file, passed as an argument to `--kubeconfig`, from the following command: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- stat -c permissions=%a /data/pki/controller-manager.conf ``` Verify that the controller-manager kubeconfig file permissions are set to 600 or more restrictive. **Expected results:** ``` permissions has value 600, expected 600 or more restrictive ``` **Returned value:** ``` permissions=600 ``` ### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root[​](#1118-ensure-that-the-controller-managerconf-file-ownership-is-set-to-root "Direct link to 1118-ensure-that-the-controller-managerconf-file-ownership-is-set-to-root") **Result:** PASS Get the controller-manager kubeconfig file, passed as an argument to `--kubeconfig`, from the following command: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- stat -c %U:%G /data/pki/controller-manager.conf ``` Verify that the controller-manager kubeconfig file ownership is set to root:root. **Expected results:** ``` root:root ``` **Returned value:** ``` root:root ``` ### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root[​](#1119-ensure-that-the-kubernetes-pki-directory-and-file-ownership-is-set-to-root "Direct link to 1119-ensure-that-the-kubernetes-pki-directory-and-file-ownership-is-set-to-root") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- sh -c "find /data/pki -not -user root -o -not -group root | wc -l | grep -q '^0$' && echo 'All files owned by root' || echo 'Some files not owned by root'" ``` Verify that the ownership of all files and directories in this hierarchy is set to root:root. **Expected results:** ``` All files owned by root ``` **Returned value:** ``` All files owned by root ``` ### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive[​](#1120-ensure-that-the-kubernetes-pki-certificate-file-permissions-are-set-to-600-or-more-restrictive "Direct link to 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive") **Result:** PASS Run the audit command to verify the permissions on the certificate files. If they do not match the expected result, run the following command to set the appropriate permissions: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- sh -c "find /data/pki -iname '*.crt' -exec chmod 600 {} \;" ``` **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- sh -c "find /data/pki -iname '*.crt' -exec stat -c permissions=%a {} \;" ``` Verify that the permissions on all the certificate files are 600 or more restrictive. **Expected results:** ``` permissions on all the certificate files are 600 or more restrictive ``` **Returned value:** ``` permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 ``` ### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600[​](#1121-ensure-that-the-kubernetes-pki-key-file-permissions-are-set-to-600 "Direct link to 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600") **Result:** PASS Run the audit command to verify the permissions on the key files. If they do not match the expected result, run the following command to set the appropriate permissions: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- sh -c "find /data/pki -iname '*.key' -exec chmod 600 {} \;" ``` **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- sh -c "find /data/pki -iname '*.key' -exec stat -c permissions=%a {} \;" ``` Verify that the permissions on all the key files are 600 or more restrictive. **Expected results:** ``` permissions on all the key files are 600 or more restrictive ``` **Returned value:** ``` permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 ``` ## 1.2 API Server[​](#12-api-server "Direct link to 1.2 API Server") ### 1.2.1 Ensure that the --anonymous-auth argument is set to false[​](#121-ensure-that-the---anonymous-auth-argument-is-set-to-false "Direct link to 1.2.1 Ensure that the --anonymous-auth argument is set to false") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --anonymous-auth=false ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--anonymous-auth` argument is set to false. **Expected results:** ``` '--anonymous-auth' is equal to 'false' ``` **Returned value:** ``` 41 root 0:07 /binaries/kube-apiserver --service-cluster-ip-range=10.96.0.0/16 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --profiling=false --advertise-address=127.0.0.1 --endpoint-reconciler-type=none --anonymous-auth=false ``` ### 1.2.2 Ensure that the --token-auth-file parameter is not set[​](#122-ensure-that-the---token-auth-file-parameter-is-not-set "Direct link to 1.2.2 Ensure that the --token-auth-file parameter is not set") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the --token-auth-file parameter is not set. **Expected results:** ``` '--token-auth-file' is not present ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --enable-admission-plugins=AlwaysPullImages,DenyServiceExternalIPs,NodeRestriction --request-timeout=300s --encryption-provider-config=/etc/encryption/encryption-config.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 ``` ### 1.2.3 Ensure that the --DenyServiceExternalIPs is set[​](#123-ensure-that-the---denyserviceexternalips-is-set "Direct link to 1.2.3 Ensure that the --DenyServiceExternalIPs is set") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --enable-admission-plugins=DenyServiceExternalIPs ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `DenyServiceExternalIPs` argument exists as a string value in --enable-admission-plugins. **Expected results:** ``` 'DenyServiceExternalIPs' argument exist as a string value in the --enable-admission-plugins list. ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --enable-admission-plugins=AlwaysPullImages,DenyServiceExternalIPs,NodeRestriction --request-timeout=300s --encryption-provider-config=/etc/encryption/encryption-config.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 ``` ### 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate[​](#124-ensure-that-the---kubelet-client-certificate-and---kubelet-client-key-arguments-are-set-as-appropriate "Direct link to 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate") **Result:** NOT APPLICABLE The control recommends setting up certificate-based kubelet authentication to ensure that the apiserver authenticates itself to kubelets when submitting requests. However, since vCluster does not interact directly with kubelets running on the control plane cluster, these client certificates are not required. The check defined in this control is not applicable in the context of vCluster. ### 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate[​](#125-ensure-that-the---kubelet-certificate-authority-argument-is-set-as-appropriate "Direct link to 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate") **Result:** NOT APPLICABLE The control recommends ensuring that the kubelet certificate authority is set appropriately. However, since vCluster does not interact directly with kubelets running on the control plane cluster, verifying certificates using certificate authority is not required. The check defined in this control is not applicable in the context of vCluster. ### 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow[​](#126-ensure-that-the---authorization-mode-argument-is-not-set-to-alwaysallow "Direct link to 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--authorization-mode` argument exists and is not set to AlwaysAllow. **Expected results:** ``` 'AlwaysAllow' argument does not exist as a string value in the --authorization-mode list. ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --enable-admission-plugins=AlwaysPullImages,DenyServiceExternalIPs,NodeRestriction --request-timeout=300s --encryption-provider-config=/etc/encryption/encryption-config.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 ``` ### 1.2.7 Ensure that the --authorization-mode argument includes Node[​](#127-ensure-that-the---authorization-mode-argument-includes-node "Direct link to 1.2.7 Ensure that the --authorization-mode argument includes Node") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --authorization-mode=Node ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--authorization-mode` argument exists and is set to a value that includes Node. **Expected results:** ``` 'Node' argument exists as a string value in the --authorization-mode list. ``` **Returned value:** ``` 47 root 0:10 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/data/pki/etcd/ca.crt --etcd-certfile=/data/pki/apiserver-etcd-client.crt --etcd-keyfile=/data/pki/apiserver-etcd-client.key --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --authorization-mode=Node ``` ### 1.2.8 Ensure that the --authorization-mode argument includes RBAC[​](#128-ensure-that-the---authorization-mode-argument-includes-rbac "Direct link to 1.2.8 Ensure that the --authorization-mode argument includes RBAC") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--authorization-mode` argument exists and is set to a value that includes RBAC. **Expected results:** ``` 'RBAC' argument exists as a string value in the --authorization-mode list. ``` **Returned value:** ``` 47 root 0:10 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/data/pki/etcd/ca.crt --etcd-certfile=/data/pki/apiserver-etcd-client.crt --etcd-keyfile=/data/pki/apiserver-etcd-client.key --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --authorization-mode=Node ``` ### 1.2.9 Ensure that the admission control plugin EventRateLimit is set[​](#129-ensure-that-the-admission-control-plugin-eventratelimit-is-set "Direct link to 1.2.9 Ensure that the admission control plugin EventRateLimit is set") **Result:** PASS 1. Follow the Kubernetes documentation and set the desired limits in a configuration file. Create a config map in the vCluster namespace that contains the configuration file: admission-control.yaml ``` apiVersion: v1 kind: ConfigMap metadata: name: admission-control namespace: vcluster-my-vcluster data: admission-control.yaml: | apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: EventRateLimit configuration: apiVersion: eventratelimit.admission.k8s.io/v1alpha1 kind: Configuration limits: - type: Server qps: 50 burst: 100 ``` 2. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --enable-admission-plugins=EventRateLimit - --admission-control-config-file=/etc/kubernetes/admission-control.yaml statefulSet: persistence: addVolumes: - name: admission-control configMap: name: admission-control addVolumeMounts: - name: admission-control mountPath: /etc/kubernetes ``` 3. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 4. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--enable-admission-plugins` argument is set to a value that includes EventRateLimit. **Expected results:** ``` 'EventRateLimit' argument exist as a string value in the --enable-admission-plugins list. ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --enable-admission-plugins=EventRateLimit --admission-control-config-file=/etc/kubernetes/admission-control.yaml ``` ### 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set[​](#1210-ensure-that-the-admission-control-plugin-alwaysadmit-is-not-set "Direct link to 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that if the `--enable-admission-plugins` argument is set, its value does not include AlwaysAdmit. **Expected results:** ``` 'AlwaysAdmit' argument does not exist as a string value in the --enable-admission-plugins list. ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --enable-admission-plugins=EventRateLimit --admission-control-config-file=/etc/kubernetes/admission-control.yaml ``` ### 1.2.11 Ensure that the admission control plugin AlwaysPullImages is set[​](#1211-ensure-that-the-admission-control-plugin-alwayspullimages-is-set "Direct link to 1.2.11 Ensure that the admission control plugin AlwaysPullImages is set") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --enable-admission-plugins=AlwaysPullImages ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--enable-admission-plugins` argument is set to a value that includes AlwaysPullImages. **Expected results:** ``` 'AlwaysPullImages' argument exist as a string value in the --enable-admission-plugins list. ``` **Returned value:** ``` 45 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --enable-admission-plugins=AlwaysPullImages ``` ### 1.2.12 Ensure that the admission control plugin ServiceAccount is set[​](#1212-ensure-that-the-admission-control-plugin-serviceaccount-is-set "Direct link to 1.2.12 Ensure that the admission control plugin ServiceAccount is set") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--disable-admission-plugins` argument is set to a value that does not include ServiceAccount. **Expected results:** ``` --disable-admission-plugins is not set. ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.13 Ensure that the admission control plugin NamespaceLifecycle is set[​](#1213-ensure-that-the-admission-control-plugin-namespacelifecycle-is-set "Direct link to 1.2.13 Ensure that the admission control plugin NamespaceLifecycle is set") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--disable-admission-plugins` argument is set to a value that does not include NamespaceLifecycle. **Expected results:** ``` --disable-admission-plugins is not set. ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.14 Ensure that the admission control plugin NodeRestriction is set[​](#1214-ensure-that-the-admission-control-plugin-noderestriction-is-set "Direct link to 1.2.14 Ensure that the admission control plugin NodeRestriction is set") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --enable-admission-plugins=NodeRestriction ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--enable-admission-plugins` argument is set to a value that includes NodeRestriction. **Expected results:** ``` 'NodeRestriction' argument exist as a string value in the --enable-admission-plugins list. ``` **Returned value:** ``` 44 root 0:03 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --enable-admission-plugins=NodeRestriction ``` ### 1.2.15 Ensure that the --profiling argument is set to false[​](#1215-ensure-that-the---profiling-argument-is-set-to-false "Direct link to 1.2.15 Ensure that the --profiling argument is set to false") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--profiling` argument is set to false. **Expected results:** ``` '--profiling' is equal to 'false' ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.16 Ensure that the --audit-log-path argument is set[​](#1216-ensure-that-the---audit-log-path-argument-is-set "Direct link to 1.2.16 Ensure that the --audit-log-path argument is set") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --audit-log-path=/var/log/audit.log statefulSet: persistence: addVolumes: - name: audit-log emptyDir: {} addVolumeMounts: - name: audit-log mountPath: /var/log ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--audit-log-path` argument is set as appropriate. **Expected results:** ``` '--audit-log-path' is present ``` **Returned value:** ``` 45 root 0:03 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --audit-log-path=/var/log/audit.log ``` ### 1.2.17 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate[​](#1217-ensure-that-the---audit-log-maxage-argument-is-set-to-30-or-as-appropriate "Direct link to 1.2.17 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --audit-log-path=/var/log/audit.log - --audit-log-maxage=30 statefulSet: persistence: addVolumes: - name: audit-log emptyDir: {} addVolumeMounts: - name: audit-log mountPath: /var/log ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--audit-log-maxage` argument is set to 30 or as appropriate. **Expected results:** ``` '--audit-log-maxage' is greater or equal to 30 ``` **Returned value:** ``` 45 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --audit-log-path=/var/log/audit.log --audit-log-maxage=30 ``` ### 1.2.18 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate[​](#1218-ensure-that-the---audit-log-maxbackup-argument-is-set-to-10-or-as-appropriate "Direct link to 1.2.18 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --audit-log-path=/var/log/audit.log - --audit-log-maxbackup=10 statefulSet: persistence: addVolumes: - name: audit-log emptyDir: {} addVolumeMounts: - name: audit-log mountPath: /var/log ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--audit-log-maxbackup` argument is set to 10 or as appropriate. **Expected results:** ``` '--audit-log-maxbackup' is greater or equal to 10 ``` **Returned value:** ``` 44 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --audit-log-path=/var/log/audit.log --audit-log-maxbackup=10 ``` ### 1.2.19 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate[​](#1219-ensure-that-the---audit-log-maxsize-argument-is-set-to-100-or-as-appropriate "Direct link to 1.2.19 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --audit-log-path=/var/log/audit.log - --audit-log-maxsize=100 statefulSet: persistence: addVolumes: - name: audit-log emptyDir: {} addVolumeMounts: - name: audit-log mountPath: /var/log ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--audit-log-maxsize` argument is set to 100 or as appropriate. **Expected results:** ``` '--audit-log-maxsize' is greater or equal to 100 ``` **Returned value:** ``` 43 root 0:01 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --audit-log-path=/var/log/audit.log --audit-log-maxsize=100 ``` ### 1.2.20 Ensure that the --request-timeout argument is set as appropriate[​](#1220-ensure-that-the---request-timeout-argument-is-set-as-appropriate "Direct link to 1.2.20 Ensure that the --request-timeout argument is set as appropriate") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --request-timeout=300s ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--request-timeout` argument is either not set or set to an appropriate value. **Expected results:** ``` '--request-timeout' is set to 300s ``` **Returned value:** ``` 43 root 0:03 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --request-timeout=300s ``` ### 1.2.21 Ensure that the --service-account-lookup argument is set to true[​](#1221-ensure-that-the---service-account-lookup-argument-is-set-to-true "Direct link to 1.2.21 Ensure that the --service-account-lookup argument is set to true") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --service-account-lookup=true ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that if the `--service-account-lookup` argument exists it is set to true. **Expected results:** ``` '--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' ``` **Returned value:** ``` 43 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --service-account-lookup=true ``` ### 1.2.22 Ensure that the --service-account-key-file argument is set as appropriate[​](#1222-ensure-that-the---service-account-key-file-argument-is-set-as-appropriate "Direct link to 1.2.22 Ensure that the --service-account-key-file argument is set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--service-account-key-file` argument exists and is set as appropriate. **Expected results:** ``` '--service-account-key-file' argument exists and is set appropriately ``` **Returned value:** ``` 45 root 0:01 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.23 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate[​](#1223-ensure-that-the---etcd-certfile-and---etcd-keyfile-arguments-are-set-as-appropriate "Direct link to 1.2.23 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true backingStore: etcd: embedded: enabled: true ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--etcd-certfile` and `--etcd-keyfile` arguments exist and they are set as appropriate. **Expected results:** ``` '--etcd-certfile' is present AND '--etcd-keyfile' is present ``` **Returned value:** ``` 47 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/data/pki/etcd/ca.crt --etcd-certfile=/data/pki/apiserver-etcd-client.crt --etcd-keyfile=/data/pki/apiserver-etcd-client.key --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.24 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate[​](#1224-ensure-that-the---tls-cert-file-and---tls-private-key-file-arguments-are-set-as-appropriate "Direct link to 1.2.24 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the --tls-cert-file and --tls-private-key-file arguments exist and they are set as appropriate. **Expected results:** ``` '--tls-cert-file' is present AND '--tls-private-key-file' is present ``` **Returned value:** ``` 45 root 0:01 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.25 Ensure that the --client-ca-file argument is set as appropriate[​](#1225-ensure-that-the---client-ca-file-argument-is-set-as-appropriate "Direct link to 1.2.25 Ensure that the --client-ca-file argument is set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--client-ca-file` argument exists and it is set as appropriate. **Expected results:** ``` '--client-ca-file' is present ``` **Returned value:** ``` 45 root 0:01 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.26 Ensure that the --etcd-cafile argument is set as appropriate[​](#1226-ensure-that-the---etcd-cafile-argument-is-set-as-appropriate "Direct link to 1.2.26 Ensure that the --etcd-cafile argument is set as appropriate") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true backingStore: etcd: embedded: enabled: true ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--etcd-cafile` argument exists and it is set as appropriate. **Expected results:** ``` '--etcd-cafile' is present ``` **Returned value:** ``` 47 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/data/pki/etcd/ca.crt --etcd-certfile=/data/pki/apiserver-etcd-client.crt --etcd-keyfile=/data/pki/apiserver-etcd-client.key --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.27 Ensure that the --encryption-provider-config argument is set as appropriate[​](#1227-ensure-that-the---encryption-provider-config-argument-is-set-as-appropriate "Direct link to 1.2.27 Ensure that the --encryption-provider-config argument is set as appropriate") **Result:** PASS 1. Follow the Kubernetes documentation and configure an EncryptionConfig file. Generate a 32-byte key using the following command: ``` head -c 32 /dev/urandom | base64 ``` 2. Create an encryption configuration file with base64 encoded key created previously: encryption-config.yaml ``` apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: - identity: {} ``` 3. Create a secret in the vCluster namespace from the configuration file: ``` kubectl create secret generic encryption-config --from-file=encryption-config.yaml -n vcluster-my-vcluster ``` 4. Create the vCluster referring to the secret as follows: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --encryption-provider-config=/etc/encryption/encryption-config.yaml statefulSet: persistence: addVolumes: - name: encryption-config secret: secretName: encryption-config addVolumeMounts: - name: encryption-config mountPath: /etc/encryption readOnly: true ``` 5. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 6. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--encryption-provider-config` argument is set to an EncryptionConfig file. Additionally, ensure that the EncryptionConfig file has all the desired resources covered especially any secrets. **Expected results:** ``` '--encryption-provider-config' is present ``` **Returned value:** ``` 45 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --encryption-provider-config=/etc/encryption/encryption-config.yaml ``` ### 1.2.28 Ensure that encryption providers are appropriately configured[​](#1228-ensure-that-encryption-providers-are-appropriately-configured "Direct link to 1.2.28 Ensure that encryption providers are appropriately configured") **Result:** PASS 1. Follow the same configuration steps as in control 1.2.27 to set up encryption providers. 2. Create the vCluster using the values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- cat /etc/encryption/encryption-config.yaml ``` Verify that `aescbc`, `kms`, or `secretbox` is set as the encryption provider for all the desired resources. **Expected results:** ``` aescbc is set as the encryption provider for the configured resources ``` **Returned value:** ``` apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: - identity: {} ``` ### 1.2.29 Ensure that the API Server only makes use of Strong Cryptographic Ciphers[​](#1229-ensure-that-the-api-server-only-makes-use-of-strong-cryptographic-ciphers "Direct link to 1.2.29 Ensure that the API Server only makes use of Strong Cryptographic Ciphers") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--tls-cipher-suites` argument is set with approved cipher suites. **Expected results:** ``` '--tls-cipher-suites' contains valid elements from approved cipher suite list ``` **Returned value:** ``` 43 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 ``` ## 1.3 Controller Manager[​](#13-controller-manager "Direct link to 1.3 Controller Manager") ### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate[​](#131-ensure-that-the---terminated-pod-gc-threshold-argument-is-set-as-appropriate "Direct link to 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate") **Result:** PASS 1. Pass the following configuration while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true controllerManager: extraArgs: - --terminated-pod-gc-threshold=12500 ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` Verify that the `--terminated-pod-gc-threshold` argument is set as appropriate. **Expected results:** ``` '--terminated-pod-gc-threshold' is present ``` **Returned value:** ``` 98 root 0:01 /binaries/kube-controller-manager --service-cluster-ip-range=10.96.0.0/12 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/data/pki/client-ca.crt --cluster-name=kubernetes --cluster-signing-cert-file=/data/pki/server-ca.crt --cluster-signing-key-file=/data/pki/server-ca.key --horizontal-pod-autoscaler-sync-period=60s --kubeconfig=/data/pki/controller-manager.conf --node-monitor-grace-period=180s --node-monitor-period=30s --pvclaimbinder-sync-period=60s --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --root-ca-file=/data/pki/server-ca.crt --service-account-private-key-file=/data/pki/sa.key --use-service-account-credentials=true --leader-elect=false --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl --terminated-pod-gc-threshold=12500 ``` ### 1.3.2 Ensure that the --profiling argument is set to false[​](#132-ensure-that-the---profiling-argument-is-set-to-false "Direct link to 1.3.2 Ensure that the --profiling argument is set to false") **Result:** PASS 1. Pass the following configuration while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true controllerManager: extraArgs: - --profiling=false ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` Verify that the `--profiling` argument is set to false. **Expected results:** ``` '--profiling' is equal to 'false' ``` **Returned value:** ``` 98 root 0:00 /binaries/kube-controller-manager --service-cluster-ip-range=10.96.0.0/12 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/data/pki/client-ca.crt --cluster-name=kubernetes --cluster-signing-cert-file=/data/pki/server-ca.crt --cluster-signing-key-file=/data/pki/server-ca.key --horizontal-pod-autoscaler-sync-period=60s --kubeconfig=/data/pki/controller-manager.conf --node-monitor-grace-period=180s --node-monitor-period=30s --pvclaimbinder-sync-period=60s --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --root-ca-file=/data/pki/server-ca.crt --service-account-private-key-file=/data/pki/sa.key --use-service-account-credentials=true --leader-elect=false --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl --profiling=false ``` ### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true[​](#133-ensure-that-the---use-service-account-credentials-argument-is-set-to-true "Direct link to 1.3.3 Ensure that the --use-service-account-credentials argument is set to true") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` Verify that the `--use-service-account-credentials` argument is set to true. **Expected results:** ``` '--use-service-account-credentials' is not equal to 'false' ``` **Returned value:** ``` 102 root 0:00 /binaries/kube-controller-manager --service-cluster-ip-range=10.96.0.0/12 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/data/pki/client-ca.crt --cluster-name=kubernetes --cluster-signing-cert-file=/data/pki/server-ca.crt --cluster-signing-key-file=/data/pki/server-ca.key --horizontal-pod-autoscaler-sync-period=60s --kubeconfig=/data/pki/controller-manager.conf --node-monitor-grace-period=180s --node-monitor-period=30s --pvclaimbinder-sync-period=60s --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --root-ca-file=/data/pki/server-ca.crt --service-account-private-key-file=/data/pki/sa.key --use-service-account-credentials=true --leader-elect=false --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl ``` ### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate[​](#134-ensure-that-the---service-account-private-key-file-argument-is-set-as-appropriate "Direct link to 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` Verify that the `--service-account-private-key-file` argument is set as appropriate. **Expected results:** ``` '--service-account-private-key-file' is present ``` **Returned value:** ``` 102 root 0:00 /binaries/kube-controller-manager --service-cluster-ip-range=10.96.0.0/12 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/data/pki/client-ca.crt --cluster-name=kubernetes --cluster-signing-cert-file=/data/pki/server-ca.crt --cluster-signing-key-file=/data/pki/server-ca.key --horizontal-pod-autoscaler-sync-period=60s --kubeconfig=/data/pki/controller-manager.conf --node-monitor-grace-period=180s --node-monitor-period=30s --pvclaimbinder-sync-period=60s --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --root-ca-file=/data/pki/server-ca.crt --service-account-private-key-file=/data/pki/sa.key --use-service-account-credentials=true --leader-elect=false --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl ``` ### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate[​](#135-ensure-that-the---root-ca-file-argument-is-set-as-appropriate "Direct link to 1.3.5 Ensure that the --root-ca-file argument is set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` Verify that the `--root-ca-file` argument exists and is set to a certificate bundle file containing the root certificate for the API server's serving certificate. **Expected results:** ``` '--root-ca-file' is present ``` **Returned value:** ``` 102 root 0:00 /binaries/kube-controller-manager --service-cluster-ip-range=10.96.0.0/12 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/data/pki/client-ca.crt --cluster-name=kubernetes --cluster-signing-cert-file=/data/pki/server-ca.crt --cluster-signing-key-file=/data/pki/server-ca.key --horizontal-pod-autoscaler-sync-period=60s --kubeconfig=/data/pki/controller-manager.conf --node-monitor-grace-period=180s --node-monitor-period=30s --pvclaimbinder-sync-period=60s --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --root-ca-file=/data/pki/server-ca.crt --service-account-private-key-file=/data/pki/sa.key --use-service-account-credentials=true --leader-elect=false --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl ``` ### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true[​](#136-ensure-that-the-rotatekubeletservercertificate-argument-is-set-to-true "Direct link to 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true") **Result:** NOT APPLICABLE The control recommends enabling the `RotateKubeletServerCertificate` feature-gate which ensures that kubelet server certificates are automatically rotated by the Kubernetes control plane. However, vCluster does not run real kubelets and operates entirely within the control plane cluster, abstracting away node-level operations. Since vCluster has no control over kubelet configuration or certificate rotation, enforcing this setting must be done at the control plane cluster level, where the actual kubelets are running. The check defined in this control is not applicable in the context of vCluster. ### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1[​](#137-ensure-that-the---bind-address-argument-is-set-to-127001 "Direct link to 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` Verify that the `--bind-address` argument is set to 127.0.0.1 **Expected results:** ``` '--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present ``` **Returned value:** ``` 102 root 0:00 /binaries/kube-controller-manager --service-cluster-ip-range=10.96.0.0/12 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/data/pki/client-ca.crt --cluster-name=kubernetes --cluster-signing-cert-file=/data/pki/server-ca.crt --cluster-signing-key-file=/data/pki/server-ca.key --horizontal-pod-autoscaler-sync-period=60s --kubeconfig=/data/pki/controller-manager.conf --node-monitor-grace-period=180s --node-monitor-period=30s --pvclaimbinder-sync-period=60s --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --root-ca-file=/data/pki/server-ca.crt --service-account-private-key-file=/data/pki/sa.key --use-service-account-credentials=true --leader-elect=false --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl ``` ## 1.4 Scheduler[​](#14-scheduler "Direct link to 1.4 Scheduler") ### 1.4.1 Ensure that the --profiling argument is set to false[​](#141-ensure-that-the---profiling-argument-is-set-to-false "Direct link to 1.4.1 Ensure that the --profiling argument is set to false") **Result:** PASS 1. Pass the following configuration as arguments to the scheduler while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true scheduler: enabled: true extraArgs: - --profiling=false sync: fromHost: nodes: enabled: true ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-scheduler ``` Verify that the `--profiling` argument is set to false. **Expected results:** ``` '--profiling' is equal to 'false' ``` **Returned value:** ``` 98 root 0:01 /binaries/kube-scheduler --authentication-kubeconfig=/data/pki/scheduler.conf --authorization-kubeconfig=/data/pki/scheduler.conf --bind-address=127.0.0.1 --kubeconfig=/data/pki/scheduler.conf --leader-elect=false --profiling=false ``` ### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1[​](#142-ensure-that-the---bind-address-argument-is-set-to-127001 "Direct link to 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1") **Result:** PASS 1. Pass the following configuration while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true scheduler: enabled: true sync: fromHost: nodes: enabled: true ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-scheduler ``` Verify that the --bind-address argument is set to 127.0.0.1 **Expected results:** ``` '--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present ``` **Returned value:** ``` 88 root 0:00 /binaries/kube-scheduler --authentication-kubeconfig=/data/pki/scheduler.conf --authorization-kubeconfig=/data/pki/scheduler.conf --bind-address=127.0.0.1 --kubeconfig=/data/pki/scheduler.conf --leader-elect=false ``` --- # etcd configuration Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** The following section covers security areas related to etcd configuration. The assessment focuses on encryption of sensitive data at rest on server, applications and in transit. Assessment focus for vCluster involves verifying correct authentication mechanisms are used and safeguarding the data at rest and in transit via TLS encryption. Key areas include validating that certificate-based authentication is properly configured and that automatic TLS generation is disabled in favor of explicit certificate management. Control numbering The control numbers used throughout this guide (2.1, 2.2, etc.) correlate directly to the official CIS Kubernetes Benchmark control numbers. This allows you to cross-reference with the official CIS documentation and maintain consistency with standard security frameworks. ## 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate[​](#21-ensure-that-the---cert-file-and---key-file-arguments-are-set-as-appropriate "Direct link to 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` Verify that the `--cert-file` and the `--key-file` arguments are set as appropriate. **Expected Result:** ``` '--cert-file' and '--key-file' arguments are appropriately set ``` **Returned Value:** ``` 31 root 2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster ``` ## 2.2 Ensure that the --client-cert-auth argument is set to true[​](#22-ensure-that-the---client-cert-auth-argument-is-set-to-true "Direct link to 2.2 Ensure that the --client-cert-auth argument is set to true") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` Verify that the `--client-cert-auth` argument is set to true. **Expected Result:** ``` '--client-cert-auth' is set to 'true' ``` **Returned Value:** ``` 31 root 2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster ``` ## 2.3 Ensure that the --auto-tls argument is not set to true[​](#23-ensure-that-the---auto-tls-argument-is-not-set-to-true "Direct link to 2.3 Ensure that the --auto-tls argument is not set to true") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` Verify that if the `--auto-tls` argument exists, it is not set to true. **Expected Result:** ``` '--auto-tls' argument does not exist ``` **Returned Value:** ``` 31 root 2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster ``` ## 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate[​](#24-ensure-that-the---peer-cert-file-and---peer-key-file-arguments-are-set-as-appropriate "Direct link to 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` Verify that the `--peer-cert-file` and `--peer-key-file` arguments are set as appropriate. **Expected Result:** ``` '--peer-cert-file' and '--peer-key-file' arguments are appropriately set ``` **Returned Value:** ``` 31 root 2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster ``` ## 2.5 Ensure that the --peer-client-cert-auth argument is set to true[​](#25-ensure-that-the---peer-client-cert-auth-argument-is-set-to-true "Direct link to 2.5 Ensure that the --peer-client-cert-auth argument is set to true") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` Verify that the `--peer-client-cert-auth` argument is set to true. **Expected Result:** ``` '--peer-client-cert-auth' is set to 'true' ``` **Returned Value:** ``` 31 root 2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster ``` ## 2.6 Ensure that the --peer-auto-tls argument is not set to true[​](#26-ensure-that-the---peer-auto-tls-argument-is-not-set-to-true "Direct link to 2.6 Ensure that the --peer-auto-tls argument is not set to true") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` Verify that if the `--peer-auto-tls` argument exists, it is not set to true. **Expected Result:** ``` '--peer-auto-tls' argument does not exist ``` **Returned Value:** ``` 31 root 2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster ``` ## 2.7 Ensure that a unique Certificate Authority is used for etcd[​](#27-ensure-that-a-unique-certificate-authority-is-used-for-etcd "Direct link to 2.7 Ensure that a unique Certificate Authority is used for etcd") **Result:** PASS 1. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` Note the file referenced by the `--trusted-ca-file` argument. 2. Run the following command and note the file referenced by `--client-ca-file`: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` 3. Verify that the file referenced by the `--client-ca-file` for apiserver is different from the `--trusted-ca-file` used by etcd. **Expected Result:** ``` The file referenced by the --client-ca-file for api-server is different from the --trusted-ca-file ``` **Returned Value:** ``` 31 root 2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster ``` ``` 47 root 6:43 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/16 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/data/pki/etcd/ca.crt --etcd-certfile=/data/pki/apiserver-etcd-client.crt --etcd-keyfile=/data/pki/apiserver-etcd-client.key --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer= https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file= /data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` --- # Kubernetes policies Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** The following section covers specific security policies for Kubernetes elements. The assessment focuses on RBAC configuration and least privilege access, Pod Security Standards and security contexts, and network policies and CNI security. This section is directly applicable to vCluster environments. Verification involves ensuring proper RBAC configurations are in place, Pod Security Standards are enforced, and network policies are appropriately configured within the tenant cluster. Control numbering The control numbers used throughout this guide (5.1.1, 5.2.1, etc.) correlate directly to the official CIS Kubernetes Benchmark control numbers. This allows you to cross-reference with the official CIS documentation and maintain consistency with standard security frameworks. ## Validation using kube-bench[​](#validation-using-kube-bench "Direct link to Validation using kube-bench") While most CIS Kubernetes Benchmark checks are not directly verifiable via kube-bench due to vCluster's virtualized control plane architecture, the Policies section (Section 5) includes controls that are relevant and testable within the tenant cluster environment. These include namespace-level restrictions, pod security standards, and security contexts that can be enforced from within vCluster. To validate your vCluster's compliance with this section, you can run kube-bench with a customized kubeconfig pointing to the vCluster API server. The following steps outline the procedure to perform the compliance check: 1. Create the vCluster but don't connect to it yet: ``` vcluster create my-vcluster --connect=false ``` 2. Wait for the vCluster to be ready, get the kubeconfig, then connect to it: ``` vcluster connect my-vcluster --server `kubectl get svc -n vcluster-my-vcluster my-vcluster -o jsonpath='{.spec.clusterIP}'` --print > kubeconfig.yaml && vcluster connect my-vcluster ``` 3. Create a dedicated namespace for the kube-bench job: ``` kubectl create namespace kube-bench ``` 4. Load the vCluster kubeconfig into a secret: ``` kubectl create secret generic my-kubeconfig-secret \ --from-file=kubeconfig=./kubeconfig.yaml \ -n kube-bench ``` 5. Harden service accounts (*Recommended*): To maintain compliance with service account token mounting policies, disable automatic token mounting for all service accounts in the cluster: ``` for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do for sa in $(kubectl get sa -n $ns -o jsonpath='{.items[*].metadata.name}'); do kubectl patch serviceaccount "$sa" \ -p '{"automountServiceAccountToken": false}' \ -n "$ns" done done ``` This is aligned with CIS to ensure that default service accounts are not actively used and ensures that tokens are not unnecessarily exposed. 6. Run the kube-bench job. Deploy a short-lived job to run kube-bench using the kubeconfig you mounted: ``` kubectl apply -f - < | Assessment sections | PASS | WARN | NOT APPLICABLE | FAIL | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---- | ---- | -------------- | ---- | | [Control plane security configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/control-plane-components.md) | 45 | 0 | 14 | 0 | | [etcd node configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/etcd.md) | 7 | 0 | 0 | 0 | | [Control plane configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/control-plane.md) | 1 | 4 | 0 | 0 | | [Worker node security configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/worker-node.md) | 0 | 0 | 24 | 0 | | [Kubernetes Policies](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/policies.md) | 9 | 24 | 0 | 2 | Review the assessment sections and map them to your vCluster configuration to ensure that your hardened deployment follows security best practices, as well as is auditable and defensible in a compliance context. --- # Worker node configuration Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** The following section provides security recommendations for components running on Kubernetes worker nodes. The assessment focuses on Kubelet configuration and security and file system permissions. Assessment focus for vCluster involves understanding that since vCluster uses the control plane cluster's nodes, these section requirements are primarily inherited from the control plane cluster's configuration. Verification should focus on ensuring the control plane cluster meets these requirements. Most controls in this section are marked as NOT APPLICABLE for vCluster deployments since the underlying node security is managed by the control plane cluster administrators. Control numbering The control numbers used throughout this guide (4.1.1, 4.2.1, etc.) correlate directly to the official CIS Kubernetes Benchmark control numbers. This allows you to cross-reference with the official CIS documentation and maintain consistency with standard security frameworks. ## 4.1 Worker node configuration files[​](#41-worker-node-configuration-files "Direct link to 4.1 Worker node configuration files") ### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive[​](#411-ensure-that-the-kubelet-service-file-permissions-are-set-to-600-or-more-restrictive "Direct link to 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive") **Result:** NOT APPLICABLE vCluster runs as pods within the control plane cluster and does not manage kubelet service files directly. The control plane cluster administrator is responsible for securing kubelet service files. Run the following command based on the file location on your system on each worker node: ``` chmod 600 /etc/systemd/system/kubelet.service.d/kubeadm.conf ``` ### 4.1.2 Ensure that the kubelet service file ownership is set to root:root[​](#412-ensure-that-the-kubelet-service-file-ownership-is-set-to-root "Direct link to 412-ensure-that-the-kubelet-service-file-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster runs as pods within the control plane cluster and does not manage kubelet service file ownership. The control plane cluster administrator is responsible for securing kubelet service file ownership. Run the following command based on the file location on your system on each worker node: ``` chown root:root /etc/systemd/system/kubelet.service.d/kubeadm.conf ``` ### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive[​](#413-if-proxy-kubeconfig-file-exists-ensure-permissions-are-set-to-600-or-more-restrictive "Direct link to 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive") **Result:** NOT APPLICABLE vCluster does not manage proxy kubeconfig files as these are part of the control plane cluster configuration. The control plane cluster administrator should ensure appropriate permissions. Run the following command based on the file location on your system on each worker node: ``` chmod 600 ``` ### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root[​](#414-if-proxy-kubeconfig-file-exists-ensure-ownership-is-set-to-root "Direct link to 414-if-proxy-kubeconfig-file-exists-ensure-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster does not manage proxy kubeconfig file ownership as these are part of the control plane cluster configuration. The control plane cluster administrator should ensure appropriate ownership. Run the following command based on the file location on your system on each worker node: ``` chown root:root ``` ### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive[​](#415-ensure-that-the---kubeconfig-kubeletconf-file-permissions-are-set-to-600-or-more-restrictive "Direct link to 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive") **Result:** NOT APPLICABLE vCluster does not manage kubelet configuration files as these are part of the control plane cluster. The control plane cluster administrator is responsible for securing kubelet configuration file permissions. Run the following command based on the file location on your system on each worker node: ``` chmod 600 /etc/kubernetes/kubelet.conf ``` ### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root[​](#416-ensure-that-the---kubeconfig-kubeletconf-file-ownership-is-set-to-root "Direct link to 416-ensure-that-the---kubeconfig-kubeletconf-file-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster does not manage kubelet configuration file ownership as these are part of the control plane cluster. The control plane cluster administrator is responsible for securing kubelet configuration file ownership. Run the following command based on the file location on your system on each worker node: ``` chown root:root /etc/kubernetes/kubelet.conf ``` ### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive[​](#417-ensure-that-the-certificate-authorities-file-permissions-are-set-to-600-or-more-restrictive "Direct link to 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive") **Result:** NOT APPLICABLE vCluster does not manage certificate authority files on worker nodes as these are part of the control plane cluster infrastructure. The control plane cluster administrator should ensure appropriate file permissions for certificate authorities. Run the following command to modify the file permissions of the `--client-ca-file`: ``` chmod 600 ``` ### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root[​](#418-ensure-that-the-client-certificate-authorities-file-ownership-is-set-to-root "Direct link to 418-ensure-that-the-client-certificate-authorities-file-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster does not manage certificate authority file ownership on worker nodes as these are part of the control plane cluster infrastructure. The control plane cluster administrator should ensure appropriate file ownership. Run the following command to modify the ownership of the `--client-ca-file`: ``` chown root:root ``` ### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 600 or more restrictive[​](#419-ensure-that-the-kubelet---config-configuration-file-has-permissions-set-to-600-or-more-restrictive "Direct link to 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 600 or more restrictive") **Result:** NOT APPLICABLE vCluster does not manage kubelet configuration files as these are controlled by the control plane cluster. The control plane cluster administrator is responsible for securing kubelet configuration file permissions. Run the following command using the config file location identified in the Audit step: ``` chmod 600 /var/lib/kubelet/config.yaml ``` ### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root[​](#4110-ensure-that-the-kubelet---config-configuration-file-ownership-is-set-to-root "Direct link to 4110-ensure-that-the-kubelet---config-configuration-file-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster does not manage kubelet configuration file ownership as these are controlled by the control plane cluster. The control plane cluster administrator is responsible for securing kubelet configuration file ownership. Run the following command using the config file location identified in the Audit step: ``` chown root:root /etc/kubernetes/kubelet.conf ``` ## 4.2 Kubelet[​](#42-kubelet "Direct link to 4.2 Kubelet") ### 4.2.1 Ensure that the --anonymous-auth argument is set to false[​](#421-ensure-that-the---anonymous-auth-argument-is-set-to-false "Direct link to 4.2.1 Ensure that the --anonymous-auth argument is set to false") **Result:** NOT APPLICABLE vCluster does not control kubelet configuration parameters as kubelet runs on the control plane cluster nodes. The control plane cluster administrator is responsible for disabling anonymous authentication on kubelets. If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to `false`. If using executable arguments, edit the kubelet service file `/etc/kubernetes/kubelet.conf` on each worker node and set the following parameter in `KUBELET_SYSTEM_PODS_ARGS` variable: ``` --anonymous-auth=false ``` Based on your system, restart the kubelet service: ``` systemctl daemon-reload systemctl restart kubelet.service ``` ### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow[​](#422-ensure-that-the---authorization-mode-argument-is-not-set-to-alwaysallow "Direct link to 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow") **Result:** NOT APPLICABLE vCluster does not control kubelet authorization modes as kubelet runs on the control plane cluster nodes. The control plane cluster administrator is responsible for configuring appropriate authorization modes. If using a Kubelet config file, edit the file to set `authorization: mode` to `Webhook`. If using executable arguments, edit the kubelet service file `/etc/kubernetes/kubelet.conf` on each worker node and set the following parameter in `KUBELET_AUTHZ_ARGS` variable: ``` --authorization-mode=Webhook ``` Based on your system, restart the `kubelet` service: ``` systemctl daemon-reload systemctl restart kubelet.service ``` ### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate[​](#423-ensure-that-the---client-ca-file-argument-is-set-as-appropriate "Direct link to 4.2.3 Ensure that the --client-ca-file argument is set as appropriate") **Result:** NOT APPLICABLE vCluster does not manage kubelet client CA configuration as kubelet runs on the control plane cluster nodes. The control plane cluster administrator is responsible for configuring client CA files appropriately. If using a Kubelet config file, edit the file to set `authentication: x509: clientCAFile` to the location of the client CA file. If using command line arguments, edit the kubelet service file `/etc/kubernetes/kubelet.conf` on each worker node and set the following parameter in `KUBELET_AUTHZ_ARGS` variable: ``` --client-ca-file= ``` Based on your system, restart the kubelet service: ``` systemctl daemon-reload systemctl restart kubelet.service ``` ### 4.2.4 Ensure that the --read-only-port argument is set to 0[​](#424-ensure-that-the---read-only-port-argument-is-set-to-0 "Direct link to 4.2.4 Ensure that the --read-only-port argument is set to 0") **Result:** NOT APPLICABLE vCluster does not control kubelet read-only port configuration as kubelet runs on the control plane cluster nodes. The control plane cluster administrator is responsible for disabling the read-only port. If using a Kubelet config file, edit the file to set `readOnlyPort` to `0`. If using command line arguments, edit the kubelet service file `/etc/kubernetes/kubelet.conf` on each worker node and set the following parameter in `KUBELET_SYSTEM_PODS_ARGS` variable: ``` --read-only-port=0 ``` Based on your system, restart the `kubelet` service: ``` systemctl daemon-reload systemctl restart kubelet.service ``` ### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0[​](#425-ensure-that-the---streaming-connection-idle-timeout-argument-is-not-set-to-0 "Direct link to 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0") **Result:** NOT APPLICABLE vCluster does not control kubelet streaming connection timeout configuration as kubelet runs on the control plane cluster nodes. The control plane cluster administrator is responsible for configuring appropriate timeout values. If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a value other than 0. If using command line arguments, edit the kubelet service file `/etc/kubernetes/kubelet.conf` on each worker node and set the following parameter in `KUBELET_SYSTEM_PODS_ARGS` variable: ``` --streaming-connection-idle-timeout=5m ``` Based on your system, restart the `kubelet` service: ``` systemctl daemon-reload systemctl restart kubelet.service ``` ### 4.2.6 Ensure that the --make-iptables-util-chains argument is set to true[​](#426-ensure-that-the---make-iptables-util-chains-argument-is-set-to-true "Direct link to 4.2.6 Ensure that the --make-iptables-util-chains argument is set to true") **Result:** NOT APPLICABLE vCluster does not control kubelet iptables configuration as kubelet runs on the control plane cluster nodes. The control plane cluster administrator is responsible for configuring iptables utility chains. If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains: true`. If using command line arguments, edit the kubelet service file `/etc/kubernetes/kubelet.conf` on each worker node and remove the `--make-iptables-util-chains` argument from the `KUBELET_SYSTEM_PODS_ARGS` variable. Based on your system, restart the `kubelet` service: ``` systemctl daemon-reload systemctl restart kubelet.service ``` ### 4.2.7 Ensure that the --hostname-override argument is not set[​](#427-ensure-that-the---hostname-override-argument-is-not-set "Direct link to 4.2.7 Ensure that the --hostname-override argument is not set") **Result:** NOT APPLICABLE vCluster does not control kubelet hostname configuration as kubelet runs on the control plane cluster nodes. The control plane cluster administrator is responsible for hostname configuration. Edit the kubelet service file `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and remove the `--hostname-override` argument from the `KUBELET_SYSTEM_PODS_ARGS` variable. Based on your system, restart the `kubelet` service: ``` systemctl daemon-reload systemctl restart kubelet.service ``` ### 4.2.8 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture[​](#428-ensure-that-the-eventrecordqps-argument-is-set-to-a-level-which-ensures-appropriate-event-capture "Direct link to 4.2.8 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture") **Result:** NOT APPLICABLE vCluster does not control kubelet event recording configuration as kubelet runs on the control plane cluster nodes. The control plane cluster administrator is responsible for configuring appropriate event recording rates. If using a Kubelet config file, edit the file to set `eventRecordQPS:` to an appropriate level. If using command line arguments, edit the kubelet service file `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` on each worker node and set the following parameter in `KUBELET_ARGS` variable. Based on your system, restart the `kubelet` service: ``` systemctl daemon-reload systemctl restart kubelet.service ``` ### 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate[​](#429-ensure-that-the---tls-cert-file-and---tls-private-key-file-arguments-are-set-as-appropriate "Direct link to 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate") **Result:** NOT APPLICABLE vCluster does not control kubelet TLS configuration as kubelet runs on the control plane cluster nodes. The control plane cluster administrator is responsible for configuring TLS certificates appropriately. If using a Kubelet config file, edit the file to set `tlsCertFile` to the location of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` to the location of the corresponding private key file. If using command line arguments, edit the kubelet service file `/etc/kubernetes/kubelet.conf` on each worker node and set the following parameters in `KUBELET_CERTIFICATE_ARGS` variable: ``` --tls-cert-file= --tls-private-key-file= ``` Based on your system, restart the kubelet service: ``` systemctl daemon-reload systemctl restart kubelet.service ``` ### 4.2.10 Ensure that the --rotate-certificates argument is not set to false[​](#4210-ensure-that-the---rotate-certificates-argument-is-not-set-to-false "Direct link to 4.2.10 Ensure that the --rotate-certificates argument is not set to false") **Result:** NOT APPLICABLE vCluster does not control kubelet certificate rotation configuration as kubelet runs on the control plane cluster nodes. The control plane cluster administrator is responsible for enabling certificate rotation. If using a Kubelet config file, edit the file to add the line `rotateCertificates: true` or remove it altogether to use the default value. If using command line arguments, edit the kubelet service file `/etc/kubernetes/kubelet.conf` on each worker node and remove `--rotate-certificates=false` argument from the `KUBELET_CERTIFICATE_ARGS` variable or set `--rotate-certificates=true`. Based on your system, restart the `kubelet` service: ``` systemctl daemon-reload systemctl restart kubelet.service ``` ### 4.2.11 Verify that the RotateKubeletServerCertificate argument is set to true[​](#4211-verify-that-the-rotatekubeletservercertificate-argument-is-set-to-true "Direct link to 4.2.11 Verify that the RotateKubeletServerCertificate argument is set to true") **Result:** NOT APPLICABLE vCluster does not control kubelet server certificate rotation as kubelet runs on the control plane cluster nodes. The control plane cluster administrator is responsible for enabling server certificate rotation. Edit the kubelet service file `/etc/kubernetes/kubelet.conf` on each worker node and set the following parameter in `KUBELET_CERTIFICATE_ARGS` variable: ``` --feature-gates=RotateKubeletServerCertificate=true ``` Based on your system, restart the `kubelet` service: ``` systemctl daemon-reload systemctl restart kubelet.service ``` ### 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers[​](#4212-ensure-that-the-kubelet-only-makes-use-of-strong-cryptographic-ciphers "Direct link to 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers") **Result:** NOT APPLICABLE vCluster does not control kubelet TLS cipher configuration as kubelet runs on the control plane cluster nodes. The control plane cluster administrator is responsible for configuring strong cryptographic ciphers. If using a Kubelet config file, edit the file to set `TLSCipherSuites:` to: ``` TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 ``` or to a subset of these values. If using executable arguments, edit the kubelet service file `/etc/kubernetes/kubelet.conf` on each worker node and set the `--tls-cipher-suites` parameter as follows, or to a subset of these values: ``` --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 ``` Based on your system, restart the `kubelet` service: ``` systemctl daemon-reload systemctl restart kubelet.service ``` ### 4.2.13 Ensure that a limit is set on pod PIDs[​](#4213-ensure-that-a-limit-is-set-on-pod-pids "Direct link to 4.2.13 Ensure that a limit is set on pod PIDs") **Result:** NOT APPLICABLE vCluster does not control kubelet PID limits as kubelet runs on the control plane cluster nodes. The control plane cluster administrator is responsible for setting appropriate PID limits. Decide on an appropriate level for this parameter and set it, either via the `--pod-max-pids` command line parameter or the `PodPidsLimit` configuration file setting. ## 4.3 kube-proxy[​](#43-kube-proxy "Direct link to 4.3 kube-proxy") ### 4.3.1 Ensure that the kube-proxy metrics service is bound to localhost[​](#431-ensure-that-the-kube-proxy-metrics-service-is-bound-to-localhost "Direct link to 4.3.1 Ensure that the kube-proxy metrics service is bound to localhost") **Result:** NOT APPLICABLE vCluster does not control kube-proxy configuration as kube-proxy runs on the control plane cluster nodes. The control plane cluster administrator is responsible for securing kube-proxy metrics endpoints. Modify or remove any values which bind the metrics service to a non-localhost address. --- # CIS hardening overview This guide provides comprehensive CIS Kubernetes Benchmark implementation and assessment for vCluster deployments according to the [Center for Internet Security (CIS) Kubernetes Benchmark](https://www.cisecurity.org/benchmark/kubernetes). The CIS Kubernetes Benchmark is an industry-standard security framework that defines best practices for securing Kubernetes environments. This guide is designed for platform engineers managing virtual Kubernetes clusters, DevOps teams deploying vCluster in production environments, security teams responsible for compliance and cluster hardening, and anyone running vCluster in multi-tenant or production environments. vCluster creates a lightweight, virtualized Kubernetes control plane that runs inside pods on your Control Plane Cluster. The architecture creates a unique security situation where virtualized components such as the vCluster API server, scheduler, and controller manager run as containers, while some security controls still depend on the underlying Control Plane Cluster, and namespace isolation ensures each vCluster operates within its own namespace. Because of the hybrid approach, traditional CIS benchmark controls need adaptation specifically for vCluster environments. Some controls apply directly, others require modifications, and some may not apply at all. By following this guide, you apply relevant CIS controls specifically adapted for vCluster's architecture and improve the security posture of your tenant clusters without breaking functionality. You'll also understand responsibility boundaries between your Control Plane Cluster and vCluster security, implement auditable hardening that can be validated for compliance purposes, and maintain operational efficiency while meeting security requirements. note This guide assumes that the Control Plane Cluster is already hardened in accordance with the CIS benchmark. The focus here is on securing the vCluster environment itself, including its API server, control plane components, and access controls. This guide is intended to be used with the following versions of CIS Kubernetes Benchmark, Kubernetes, and vCluster:
| vCluster version | CIS benchmark version | Kubernetes version | | ---------------- | --------------------- | ------------------ | | v0.26.0 | v1.10.0 | v1.31 | important This guide currently supports the K8s distribution for the vCluster API Server and embedded etcd as the backing store. Other Kubernetes distributions or backing store setups are not covered and might require different hardening approaches. ## vCluster runtime requirements[​](#vcluster-runtime-requirements "Direct link to vCluster runtime requirements") ### Configure default service account[​](#configure-default-service-account "Direct link to Configure default service account") Kubernetes automatically creates a default service account in every namespace and mounts its token into every pod by default. Automatic token mounting creates unnecessary security exposure. The CIS benchmark recommends workloads should only access specific service accounts with minimal required permissions. The configuration follows the principle of least privilege and ensures pods only get the API access they explicitly need. You can set `automountServiceAccountToken: false` for 'default' service accounts in all namespaces. Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod. Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account. The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments. The following configuration prevents unauthorized API access by disabling automatic service account token mounting for default service accounts. The default service account in all the namespaces must include the following value: ``` automountServiceAccountToken: false ``` Instead of doing this manually in every namespace, you can run this command once to apply the setting to all service accounts in all namespaces: ``` for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do for sa in $(kubectl get sa -n $ns -o jsonpath='{.items[*].metadata.name}'); do kubectl patch serviceaccount "$sa" \ -p '{"automountServiceAccountToken": false}' \ -n "$ns" done done ``` ### API Server audit configuration[​](#api-server-audit-configuration "Direct link to API Server audit configuration") Auditing creates a security trail that records who did what and when in your cluster. Security teams need audit trails for incident investigation to track unauthorized access attempts. Many compliance standards require audit trails. Audit logs provide operational visibility to understand how your cluster is being used and help detect unusual API access patterns. The following process creates a minimal audit policy that captures metadata about all API requests and configures the vCluster API server to use it. You can enable auditing on the Kubernetes API Server and set the desired audit log path. Control 1.2.16 of the benchmark recommends enabling auditing on the Kubernetes API Server. Auditing the Kubernetes API Server provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators, or other components of the system. Additionally, a minimal audit policy should be in place for the auditing to be carried out. The following configuration enables comprehensive logging of all API server requests for security monitoring and compliance. Create a config map with a minimal audit policy: audit-config-configmap.yaml ``` apiVersion: v1 kind: ConfigMap metadata: name: audit-config namespace: vcluster-my-vcluster data: audit-policy.yaml: | apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata ``` Pass both configurations as arguments to the API Server while creating the vCluster using the following: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --audit-policy-file=/etc/kubernetes/audit-policy.yaml - --audit-log-path=/var/log/audit.log - --audit-log-maxage=30 - --audit-log-maxbackup=10 - --audit-log-maxsize=100 statefulSet: persistence: addVolumes: - name: audit-log emptyDir: {} - name: audit-policy configMap: name: audit-config addVolumeMounts: - name: audit-log mountPath: /var/log - name: audit-policy mountPath: /etc/kubernetes ``` ### API Server encryption configuration[​](#api-server-encryption-configuration "Direct link to API Server encryption configuration") Kubernetes stores data in etcd in plain text by default. Anyone with etcd access can read all your secrets, including database passwords, API keys, TLS certificates, and OAuth tokens. Even if someone gains unauthorized access to your etcd data, they cannot read encrypted secrets without the encryption key. The following process generates a strong encryption key and configures the API server to encrypt specific resource types before storing them in etcd. You can ensure that encryption providers are appropriately configured. Where etcd encryption is used, it is important to ensure that the appropriate set of encryption providers are utilized. Currently, the `aescbc`, `kms`, and `secretbox` are usually appropriate options. The following configuration encrypts sensitive data such as Secrets at rest in etcd using strong encryption. 1. Generate a 32-byte key using the following command: ``` head -c 32 /dev/urandom | base64 ``` 2. Create an encryption configuration file, using the base64 encoded key created previously: encryption-config.yaml ``` apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: - identity: {} ``` 3. Create a secret in the vCluster namespace from the configuration file. ``` kubectl create secret generic encryption-config --from-file=encryption-config.yaml -n vcluster-my-vcluster ``` 4. Create the vCluster referring the Secret as follows: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --encryption-provider-config=/etc/encryption/encryption-config.yaml statefulSet: persistence: addVolumes: - name: encryption-config secret: secretName: encryption-config addVolumeMounts: - name: encryption-config mountPath: /etc/encryption readOnly: true ``` ### API Server setting EventRateLimit[​](#api-server-setting-eventratelimit "Direct link to API Server setting EventRateLimit") In environments with multiple tenant clusters or in production, a misbehaving application could overwhelm your API server by generating excessive events. Event flooding could make the entire cluster unresponsive, prevent legitimate workloads from functioning, and create cascading failures across your infrastructure. EventRateLimit admission control ensures no single source can flood the API server with too many requests. The following example configuration sets a server-wide limit of 50 events per second with a burst allowance of 100 events. You can ensure that the admission control plugin EventRateLimit is set. Using EventRateLimit admission control enforces a limit on the number of events that the API Server accepts in a given time slice. A misbehaving workload could overwhelm and DoS the API Server and make it unavailable. This particularly applies when multiple tenants share infrastructure, where a small percentage of misbehaving tenants could have a significant impact on the performance of the cluster overall. It is recommended to limit the rate of events that the API server accepts. The following configuration prevents denial-of-service attacks by limiting how many events the API server processes. 1. Create a config map in the vCluster namespace that contains the configuration file. admission-control.yaml ``` apiVersion: v1 kind: ConfigMap metadata: name: admission-control namespace: vcluster-my-vcluster data: admission-control.yaml: | apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: EventRateLimit configuration: apiVersion: eventratelimit.admission.k8s.io/v1alpha1 kind: Configuration limits: - type: Server qps: 50 burst: 100 ``` 2. Create the vCluster referring the ConfigMap as follows: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --enable-admission-plugins=EventRateLimit - --admission-control-config-file=/etc/kubernetes/admission-control.yaml statefulSet: persistence: addVolumes: - name: admission-control configMap: name: admission-control addVolumeMounts: - name: admission-control mountPath: /etc/kubernetes ``` ## Reference hardened vCluster configuration[​](#reference-hardened-vcluster-configuration "Direct link to Reference hardened vCluster configuration") Since the worker nodes of a vCluster can be either shared nodes from the Control Plane Cluster or dedicated private nodes, there are different hardened vCluster configuration files and assessment guides. vCluster with shared nodes The configuration combines all the individual security settings into a single vCluster values file. You need to create the supporting ConfigMaps and Secrets (shown in the sections above) before applying the configuration. Key security features include disabled anonymous authentication, strong TLS ciphers only, multiple admission controllers for enhanced security, disabled profiling on control plane components, tuned garbage collection, and enabled virtual scheduler for improved workload isolation. The following is a reference vCluster values file with minimum required configuration for a hardened installation. This needs to be used in combination with the above runtime level settings and other measures to achieve full compliance. vcluster.yaml ``` controlPlane: distro: k8s: image: tag: v1.31.2 # Kubernetes version you want to use enabled: true apiServer: extraArgs: - --admission-control-config-file=/etc/kubernetes/admission-control.yaml - --anonymous-auth=false - --audit-policy-file=/etc/kubernetes/audit-policy.yaml - --audit-log-path=/var/log/audit.log - --audit-log-maxage=30 - --audit-log-maxbackup=10 - --audit-log-maxsize=100 - --enable-admission-plugins=AlwaysPullImages,DenyServiceExternalIPs,EventRateLimit,NodeRestriction - --encryption-provider-config=/etc/encryption/encryption-config.yaml - --request-timeout=300s - --service-account-lookup=true - --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 controllerManager: extraArgs: - --terminated-pod-gc-threshold=12500 - --profiling=false scheduler: enabled: true extraArgs: - --profiling=false backingStore: etcd: embedded: enabled: true statefulSet: persistence: addVolumes: - name: audit-log emptyDir: {} - name: audit-policy configMap: name: audit-config - name: encryption-config secret: secretName: encryption-config - name: admission-control configMap: name: admission-control addVolumeMounts: - name: audit-log mountPath: /var/log - name: audit-policy mountPath: /etc/kubernetes/audit-policy.yaml subPath: audit-policy.yaml - name: encryption-config mountPath: /etc/encryption readOnly: true - name: admission-control mountPath: /etc/kubernetes/admission-control.yaml subPath: admission-control.yaml sync: fromHost: nodes: enabled: true ``` The hardening guide helps you configure your tenant cluster to meet the CIS Kubernetes Benchmark. Review the [assessment guide for shared nodes](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/self-assessment.md) to validate the configuration by testing it against specific CIS controls. Not all controls apply directly because of the namespaced and virtualized nature of vCluster. vCluster with private nodes The configuration combines all the individual security settings into a single vCluster values file. You need to create the supporting ConfigMaps and Secrets (shown in the sections above) before applying the configuration. Key security features include disabled anonymous authentication, strong TLS ciphers only, multiple admission controllers for enhanced security, disabled profiling on control plane components, tuned garbage collection, and enabled virtual scheduler for improved workload isolation. The following is a reference vCluster values file with minimum required configuration for a hardened installation. This needs to be used in combination with the above runtime level settings and other measures to achieve full compliance. vcluster.yaml ``` controlPlane: distro: k8s: image: tag: v1.31.2 # Kubernetes version you want to use enabled: true apiServer: extraArgs: - --admission-control-config-file=/etc/kubernetes/admission-control.yaml - --anonymous-auth=false - --audit-policy-file=/etc/kubernetes/audit-policy.yaml - --audit-log-path=/var/log/audit.log - --audit-log-maxage=30 - --audit-log-maxbackup=10 - --audit-log-maxsize=100 - --enable-admission-plugins=AlwaysPullImages,DenyServiceExternalIPs,EventRateLimit,NodeRestriction - --encryption-provider-config=/etc/encryption/encryption-config.yaml - --feature-gates=RotateKubeletServerCertificate=true - --request-timeout=300s - --service-account-lookup=true - --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 - --kubelet-certificate-authority=/data/pki/ca.crt controllerManager: extraArgs: - --terminated-pod-gc-threshold=12500 - --profiling=false scheduler: extraArgs: - --profiling=false service: spec: type: LoadBalancer backingStore: etcd: embedded: enabled: true statefulSet: persistence: addVolumes: - name: audit-log emptyDir: {} - name: audit-policy configMap: name: audit-config - name: encryption-config secret: secretName: encryption-config - name: admission-control configMap: name: admission-control addVolumeMounts: - name: audit-log mountPath: /var/log - name: audit-policy mountPath: /etc/kubernetes/audit-policy.yaml subPath: audit-policy.yaml - name: encryption-config mountPath: /etc/encryption readOnly: true - name: admission-control mountPath: /etc/kubernetes/admission-control.yaml subPath: admission-control.yaml privateNodes: enabled: true kubelet: config: serverTLSBootstrap: true featureGates: RotateKubeletServerCertificate: true podPidsLimit: 1000000 tlsCipherSuites: - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384" - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256" networking: podCIDR: 10.64.0.0/16 serviceCIDR: 10.128.0.0/16 ``` The hardening guide helps you configure your tenant cluster to meet the CIS Kubernetes Benchmark. Review the [assessment guide for private nodes](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/private-nodes/self-assessment.md) to validate the configuration by testing it against specific CIS controls. Not all controls apply directly because of the namespaced and virtualized nature of vCluster. --- # Control plane configuration Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** The following section covers cluster-wide security areas. The assessment focuses on authentication and authorization mechanisms and audit logging configuration. Assessment focus for vCluster involves verifying audit logging is enabled and alternative mechanisms such as OIDC are used. Key areas include ensuring that traditional authentication methods like client certificates, service account tokens, and bootstrap tokens are replaced with more secure alternatives for user authentication. Control numbering The control numbers used throughout this guide (3.1.1, 3.2.1, etc.) correlate directly to the official CIS Kubernetes Benchmark control numbers. This allows you to cross-reference with the official CIS documentation and maintain consistency with standard security frameworks. ## 3.1 Authentication and authorization[​](#31-authentication-and-authorization "Direct link to 3.1 Authentication and authorization") ### 3.1.1 Client certificate authentication should not be used for users[​](#311-client-certificate-authentication-should-not-be-used-for-users "Direct link to 3.1.1 Client certificate authentication should not be used for users") **Result:** WARN Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented in place of client certificates. Client certificate authentication, while functional, creates operational challenges around certificate lifecycle management and lacks the advanced features provided by modern authentication systems. OIDC integration allows for centralized identity management, multi-factor authentication, and easier user provisioning and deprovisioning. ### 3.1.2 Service account token authentication should not be used for users[​](#312-service-account-token-authentication-should-not-be-used-for-users "Direct link to 3.1.2 Service account token authentication should not be used for users") **Result:** WARN Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented in place of service account tokens. Service account tokens are designed for workload authentication rather than human users. Using service account tokens for user authentication bypasses important security controls and audit trails that are essential for compliance and security monitoring. ### 3.1.3 Bootstrap token authentication should not be used for users[​](#313-bootstrap-token-authentication-should-not-be-used-for-users "Direct link to 3.1.3 Bootstrap token authentication should not be used for users") **Result:** WARN Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented in place of bootstrap tokens. Bootstrap tokens are intended for node bootstrapping and initial cluster setup, not for ongoing user authentication. Using bootstrap tokens for user access creates security risks and prevents proper user identity tracking and access control. ## 3.2 Logging[​](#32-logging "Direct link to 3.2 Logging") ### 3.2.1 Ensure that a minimal audit policy is created[​](#321-ensure-that-a-minimal-audit-policy-is-created "Direct link to 3.2.1 Ensure that a minimal audit policy is created") **Result:** PASS 1. Follow the [Kubernetes documentation](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/) and create a config map with a minimal audit policy: audit-config-configmap.yaml ``` apiVersion: v1 kind: ConfigMap metadata: name: audit-config namespace: vcluster-my-vcluster data: audit-policy.yaml: | apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata ``` 2. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --audit-policy-file=/etc/kubernetes/audit-policy.yaml statefulSet: persistence: addVolumes: - name: audit-policy configMap: name: audit-config addVolumeMounts: - name: audit-policy mountPath: /etc/kubernetes ``` 3. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 4. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--audit-policy-file` is set. **Expected Result:** ``` '--audit-policy-file' is present ``` **Returned Value:** ``` 26 root 0:08 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --audit-policy-file=/etc/kubernetes/audit-policy.yaml ``` ### 3.2.2 Ensure that the audit policy covers key security concerns[​](#322-ensure-that-the-audit-policy-covers-key-security-concerns "Direct link to 3.2.2 Ensure that the audit policy covers key security concerns") **Result:** WARN Review the audit policy provided for the cluster and ensure that it covers key security areas. The audit policy should include access to Secrets managed by the cluster, taking care to only log Metadata for requests to Secrets, ConfigMaps, and TokenReviews to avoid the risk of logging sensitive data. Additionally, ensure coverage of modification of pod and deployment objects and use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. For most requests, minimally logging at the Metadata level is recommended as the most basic level of logging. Consider modification of the audit policy in use on the cluster to include these items at a minimum. The following sample policy satisfies these criteria: audit-config-configmap.yaml ``` apiVersion: v1 kind: ConfigMap metadata: name: audit-config namespace: vcluster-my-vcluster data: audit-policy.yaml: | apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata resources: - group: "" resources: ["secrets", "configmaps"] - group: "authentication.k8s.io" resources: ["tokenreviews"] - level: RequestResponse resources: - group: "" resources: ["pods"] - group: "apps" resources: ["deployments", "replicasets"] verbs: ["create", "update", "patch", "delete"] - level: RequestResponse resources: - group: "" resources: ["pods/exec", "pods/portforward", "pods/proxy", "services/proxy"] - level: Metadata omitStages: - RequestReceived ``` --- # Control plane security configuration Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** The following section covers security recommendations for the direct configuration of Kubernetes control plane processes. The assessment focuses on API Server configuration and security settings, Controller Manager security parameters, Scheduler security configurations, and general control plane security practices. Assessment focus for vCluster involves checking the extraArgs configurations in your `vcluster.yaml` file and ensuring proper security parameters are set for the virtualized API server, controller manager, and scheduler. Since vCluster virtualizes the control plane components, verification requires examining container processes rather than traditional system-level checks. Control numbering The control numbers used throughout this guide (1.1.1, 1.2.1, etc.) correlate directly to the official CIS Kubernetes Benchmark control numbers. This allows you to cross-reference with the official CIS documentation and maintain consistency with standard security frameworks. note For auditing each control, create the vCluster using default values as shown below, unless specified otherwise. ``` vcluster create my-vcluster --connect=false ``` ## 1.1 Master node configuration files[​](#11-master-node-configuration-files "Direct link to 1.1 Master node configuration files") ### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive[​](#111-ensure-that-the-api-server-pod-specification-file-permissions-are-set-to-644-or-more-restrictive "Direct link to 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/kube-apiserver.yaml` as found in kubeadm-based clusters. Instead, the vCluster API server runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file permissions check defined in this control not applicable in the context of vCluster. ### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root[​](#112-ensure-that-the-api-server-pod-specification-file-ownership-is-set-to-root "Direct link to 112-ensure-that-the-api-server-pod-specification-file-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/kube-apiserver.yaml` as found in kubeadm-based clusters. Instead, the vCluster API server runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file ownership check defined in this control not applicable in the context of vCluster. ### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive[​](#113-ensure-that-the-controller-manager-pod-specification-file-permissions-are-set-to-644-or-more-restrictive "Direct link to 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/kube-controller-manager.yaml` as found in kubeadm-based clusters. Instead, the vCluster controller manager runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file permissions check defined in this control not applicable in the context of vCluster. ### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root[​](#114-ensure-that-the-controller-manager-pod-specification-file-ownership-is-set-to-root "Direct link to 114-ensure-that-the-controller-manager-pod-specification-file-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/kube-controller-manager.yaml` as found in kubeadm-based clusters. Instead, the vCluster controller manager runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file ownership check defined in this control not applicable in the context of vCluster. ### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive[​](#115-ensure-that-the-scheduler-pod-specification-file-permissions-are-set-to-644-or-more-restrictive "Direct link to 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/kube-scheduler.yaml` as found in kubeadm-based clusters. Instead, the vCluster scheduler runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file permissions check defined in this control not applicable in the context of vCluster. ### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root[​](#116-ensure-that-the-scheduler-pod-specification-file-ownership-is-set-to-root "Direct link to 116-ensure-that-the-scheduler-pod-specification-file-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/kube-scheduler.yaml` as found in kubeadm-based clusters. Instead, the vCluster scheduler runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file ownership check defined in this control not applicable in the context of vCluster. ### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive[​](#117-ensure-that-the-etcd-pod-specification-file-permissions-are-set-to-644-or-more-restrictive "Direct link to 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/etcd.yaml` as found in kubeadm-based clusters. In case of embedded etcd, the etcd runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file permissions check defined in this control not applicable in the context of vCluster. ### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root[​](#118-ensure-that-the-etcd-pod-specification-file-ownership-is-set-to-root "Direct link to 118-ensure-that-the-etcd-pod-specification-file-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster does not rely on static pod manifests stored on the host such as `/etc/kubernetes/manifests/etcd.yaml` as found in kubeadm-based clusters. In case of embedded etcd, the etcd runs as a separate binary embedded within the same container as the syncer pod. The architecture does not use the static pod mechanism, making the file ownership check defined in this control not applicable in the context of vCluster. ### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive[​](#119-ensure-that-the-container-network-interface-file-permissions-are-set-to-644-or-more-restrictive "Direct link to 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive") **Result:** NOT APPLICABLE vCluster does not configure or manage Container Network Interface (CNI) settings. Networking is handled entirely by the control plane cluster's CNI plugin. As a result, there are no CNI configuration files such as `/etc/cni/net.d/*.conf` present within the vCluster container. You should evaluate this control on the underlying control plane cluster as it is not applicable in vCluster environments. ### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root[​](#1110-ensure-that-the-container-network-interface-file-ownership-is-set-to-root "Direct link to 1110-ensure-that-the-container-network-interface-file-ownership-is-set-to-root") **Result:** NOT APPLICABLE vCluster does not configure or manage Container Network Interface (CNI) settings. Networking is handled entirely by the control plane cluster's CNI plugin. As a result, there are no CNI configuration files such as `/etc/cni/net.d/*.conf` present within the vCluster container. You should evaluate this control on the underlying control plane cluster as it is not applicable in vCluster environments. ### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive[​](#1111-ensure-that-the-etcd-data-directory-permissions-are-set-to-700-or-more-restrictive "Direct link to 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive") **Result:** PASS 1. Get the etcd data directory, passed as an argument to `--data-dir`, from the following command: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` 2. Run the audit command to verify the permissions on the data directory. If they do not match the expected result, run the following command to set the appropriate permissions: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- chmod 700 /data/etcd ``` **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- stat -c permissions=%a /data/etcd ``` Verify that the etcd data directory permissions are set to 700 or more restrictive. **Expected results:** ``` permissions has value 700, expected 700 or more restrictive ``` **Returned value:** ``` permissions=700 ``` ### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd[​](#1112-ensure-that-the-etcd-data-directory-ownership-is-set-to-etcd "Direct link to 1112-ensure-that-the-etcd-data-directory-ownership-is-set-to-etcd") **Result:** NOT APPLICABLE The control recommends that the etcd data directory be owned by the etcd user and group (etcd:etcd) to follow least privilege principles. However, in vCluster, etcd is embedded and runs as `root` within the syncer container. There is no separate etcd user present in the container. The directory ownership check defined in this control is not applicable in the context of vCluster. ### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive[​](#1113-ensure-that-the-adminconf-file-permissions-are-set-to-600-or-more-restrictive "Direct link to 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- stat -c permissions=%a /data/pki/admin.conf ``` Verify that the admin.conf file permissions are 600 or more restrictive. **Expected results:** ``` permissions has value 600, expected 600 or more restrictive ``` **Returned value:** ``` permissions=600 ``` ### 1.1.14 Ensure that the admin.conf file ownership is set to root:root[​](#1114-ensure-that-the-adminconf-file-ownership-is-set-to-root "Direct link to 1114-ensure-that-the-adminconf-file-ownership-is-set-to-root") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- stat -c %U:%G /data/pki/admin.conf ``` Verify that the admin.conf file ownership is set to root:root. **Expected results:** ``` root:root ``` **Returned value:** ``` root:root ``` ### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive[​](#1115-ensure-that-the-schedulerconf-file-permissions-are-set-to-600-or-more-restrictive "Direct link to 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive") **Result:** PASS Get the scheduler kubeconfig file, passed as an argument to `--kubeconfig`, from the following command: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-scheduler ``` **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- stat -c permissions=%a /data/pki/scheduler.conf ``` Verify that the scheduler kubeconfig file permissions are set to 600 or more restrictive. **Expected results:** ``` permissions has value 600, expected 600 or more restrictive ``` **Returned value:** ``` permissions=600 ``` ### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root[​](#1116-ensure-that-the-schedulerconf-file-ownership-is-set-to-root "Direct link to 1116-ensure-that-the-schedulerconf-file-ownership-is-set-to-root") **Result:** PASS Get the scheduler kubeconfig file, passed as an argument to `--kubeconfig`, from the following command: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-scheduler ``` **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- stat -c %U:%G /data/pki/scheduler.conf ``` Verify that the scheduler kubeconfig file ownership is set to root:root. **Expected results:** ``` root:root ``` **Returned value:** ``` root:root ``` ### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive[​](#1117-ensure-that-the-controller-managerconf-file-permissions-are-set-to-600-or-more-restrictive "Direct link to 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive") **Result:** PASS Get the controller-manager kubeconfig file, passed as an argument to `--kubeconfig`, from the following command: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- stat -c permissions=%a /data/pki/controller-manager.conf ``` Verify that the controller-manager kubeconfig file permissions are set to 600 or more restrictive. **Expected results:** ``` permissions has value 600, expected 600 or more restrictive ``` **Returned value:** ``` permissions=600 ``` ### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root[​](#1118-ensure-that-the-controller-managerconf-file-ownership-is-set-to-root "Direct link to 1118-ensure-that-the-controller-managerconf-file-ownership-is-set-to-root") **Result:** PASS Get the controller-manager kubeconfig file, passed as an argument to `--kubeconfig`, from the following command: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- stat -c %U:%G /data/pki/controller-manager.conf ``` Verify that the controller-manager kubeconfig file ownership is set to root:root. **Expected results:** ``` root:root ``` **Returned value:** ``` root:root ``` ### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root[​](#1119-ensure-that-the-kubernetes-pki-directory-and-file-ownership-is-set-to-root "Direct link to 1119-ensure-that-the-kubernetes-pki-directory-and-file-ownership-is-set-to-root") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- sh -c "find /data/pki -not -user root -o -not -group root | wc -l | grep -q '^0$' && echo 'All files owned by root' || echo 'Some files not owned by root'" ``` Verify that the ownership of all files and directories in this hierarchy is set to root:root. **Expected results:** ``` All files owned by root ``` **Returned value:** ``` All files owned by root ``` ### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive[​](#1120-ensure-that-the-kubernetes-pki-certificate-file-permissions-are-set-to-600-or-more-restrictive "Direct link to 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive") **Result:** PASS Run the audit command to verify the permissions on the certificate files. If they do not match the expected result, run the following command to set the appropriate permissions: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- sh -c "find /data/pki -iname '*.crt' -exec chmod 600 {} \;" ``` **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- sh -c "find /data/pki -iname '*.crt' -exec stat -c permissions=%a {} \;" ``` Verify that the permissions on all the certificate files are 600 or more restrictive. **Expected results:** ``` permissions on all the certificate files are 600 or more restrictive ``` **Returned value:** ``` permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 ``` ### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600[​](#1121-ensure-that-the-kubernetes-pki-key-file-permissions-are-set-to-600 "Direct link to 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600") **Result:** PASS Run the audit command to verify the permissions on the key files. If they do not match the expected result, run the following command to set the appropriate permissions: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- sh -c "find /data/pki -iname '*.key' -exec chmod 600 {} \;" ``` **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- sh -c "find /data/pki -iname '*.key' -exec stat -c permissions=%a {} \;" ``` Verify that the permissions on all the key files are 600 or more restrictive. **Expected results:** ``` permissions on all the key files are 600 or more restrictive ``` **Returned value:** ``` permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 ``` ## 1.2 API Server[​](#12-api-server "Direct link to 1.2 API Server") ### 1.2.1 Ensure that the --anonymous-auth argument is set to false[​](#121-ensure-that-the---anonymous-auth-argument-is-set-to-false "Direct link to 1.2.1 Ensure that the --anonymous-auth argument is set to false") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --anonymous-auth=false ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--anonymous-auth` argument is set to false. **Expected results:** ``` '--anonymous-auth' is equal to 'false' ``` **Returned value:** ``` 41 root 0:07 /binaries/kube-apiserver --service-cluster-ip-range=10.96.0.0/16 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --profiling=false --advertise-address=127.0.0.1 --endpoint-reconciler-type=none --anonymous-auth=false ``` ### 1.2.2 Ensure that the --token-auth-file parameter is not set[​](#122-ensure-that-the---token-auth-file-parameter-is-not-set "Direct link to 1.2.2 Ensure that the --token-auth-file parameter is not set") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the --token-auth-file parameter is not set. **Expected results:** ``` '--token-auth-file' is not present ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --enable-admission-plugins=AlwaysPullImages,DenyServiceExternalIPs,NodeRestriction --request-timeout=300s --encryption-provider-config=/etc/encryption/encryption-config.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 ``` ### 1.2.3 Ensure that the --DenyServiceExternalIPs is set[​](#123-ensure-that-the---denyserviceexternalips-is-set "Direct link to 1.2.3 Ensure that the --DenyServiceExternalIPs is set") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --enable-admission-plugins=DenyServiceExternalIPs ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `DenyServiceExternalIPs` argument exists as a string value in --enable-admission-plugins. **Expected results:** ``` 'DenyServiceExternalIPs' argument exist as a string value in the --enable-admission-plugins list. ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --enable-admission-plugins=AlwaysPullImages,DenyServiceExternalIPs,NodeRestriction --request-timeout=300s --encryption-provider-config=/etc/encryption/encryption-config.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 ``` ### 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate[​](#124-ensure-that-the---kubelet-client-certificate-and---kubelet-client-key-arguments-are-set-as-appropriate "Direct link to 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the --kubelet-client-certificate and --kubelet-client-key arguments exist and they are set as appropriate. **Expected results:** ``` '--kubelet-client-certificate' is present AND '--kubelet-client-key' is present ``` **Returned value:** ``` 12 root 0:32 /binaries/kube-apiserver --service-cluster-ip-range=10.128.0.0/16 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=https://my-vcluster-etcd:2379 --etcd-cafile=/data/pki/etcd/ca.crt --etcd-certfile=/data/pki/apiserver-etcd-client.crt --etcd-keyfile=/data/pki/apiserver-etcd-client.key --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --profiling=false --advertise-address=127.0.0.1 --endpoint-reconciler-type=none --kubelet-client-certificate=/data/pki/apiserver-kubelet-client.crt --kubelet-client-key=/data/pki/apiserver-kubelet-client.key --endpoint-reconciler-type=none --egress-selector-config-file=/data/konnectivity/egress.yaml --admission-control-config-file=/etc/kubernetes/admission-control.yaml --anonymous-auth=false --audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --enable-admission-plugins=AlwaysPullImages,DenyServiceExternalIPs,EventRateLimit,NodeRestriction --encryption-provider-config=/etc/encryption/encryption-config.yaml --request-timeout=300s --service-account-lookup=true --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 ``` ### 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate[​](#125-ensure-that-the---kubelet-certificate-authority-argument-is-set-as-appropriate "Direct link to 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- sh -c "pgrep -f 'kube-apiserver' | head -1 | xargs -I {} cat /proc/{}/cmdline | tr '\0' ' '" ``` Verify that the --kubelet-certificate-authority argument exists and is set as appropriate. **Expected results:** ``` '--kubelet-certificate-authority' is present ``` **Returned value:** ``` /binaries/kube-apiserver --service-cluster-ip-range=10.128.0.0/16 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/data/pki/etcd/ca.crt --etcd-certfile=/data/pki/apiserver-etcd-client.crt --etcd-keyfile=/data/pki/apiserver-etcd-client.key --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --profiling=false --advertise-address=127.0.0.1 --endpoint-reconciler-type=none --kubelet-client-certificate=/data/pki/apiserver-kubelet-client.crt --kubelet-client-key=/data/pki/apiserver-kubelet-client.key --endpoint-reconciler-type=none --egress-selector-config-file=/data/konnectivity/egress.yaml --admission-control-config-file=/etc/kubernetes/admission-control.yaml --anonymous-auth=false --audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --enable-admission-plugins=AlwaysPullImages,DenyServiceExternalIPs,EventRateLimit,NodeRestriction --encryption-provider-config=/etc/encryption/encryption-config.yaml --feature-gates=RotateKubeletServerCertificate=true --request-timeout=300s --service-account-lookup=true --kubelet-certificate-authority=/data/pki/ca.crt --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 ``` ### 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow[​](#126-ensure-that-the---authorization-mode-argument-is-not-set-to-alwaysallow "Direct link to 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--authorization-mode` argument exists and is not set to AlwaysAllow. **Expected results:** ``` 'AlwaysAllow' argument does not exist as a string value in the --authorization-mode list. ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --enable-admission-plugins=AlwaysPullImages,DenyServiceExternalIPs,NodeRestriction --request-timeout=300s --encryption-provider-config=/etc/encryption/encryption-config.yaml --audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 ``` ### 1.2.7 Ensure that the --authorization-mode argument includes Node[​](#127-ensure-that-the---authorization-mode-argument-includes-node "Direct link to 1.2.7 Ensure that the --authorization-mode argument includes Node") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --authorization-mode=Node ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--authorization-mode` argument exists and is set to a value that includes Node. **Expected results:** ``` 'Node' argument exists as a string value in the --authorization-mode list. ``` **Returned value:** ``` 47 root 0:10 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/data/pki/etcd/ca.crt --etcd-certfile=/data/pki/apiserver-etcd-client.crt --etcd-keyfile=/data/pki/apiserver-etcd-client.key --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --authorization-mode=Node ``` ### 1.2.8 Ensure that the --authorization-mode argument includes RBAC[​](#128-ensure-that-the---authorization-mode-argument-includes-rbac "Direct link to 1.2.8 Ensure that the --authorization-mode argument includes RBAC") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--authorization-mode` argument exists and is set to a value that includes RBAC. **Expected results:** ``` 'RBAC' argument exists as a string value in the --authorization-mode list. ``` **Returned value:** ``` 47 root 0:10 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/data/pki/etcd/ca.crt --etcd-certfile=/data/pki/apiserver-etcd-client.crt --etcd-keyfile=/data/pki/apiserver-etcd-client.key --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --authorization-mode=Node ``` ### 1.2.9 Ensure that the admission control plugin EventRateLimit is set[​](#129-ensure-that-the-admission-control-plugin-eventratelimit-is-set "Direct link to 1.2.9 Ensure that the admission control plugin EventRateLimit is set") **Result:** PASS 1. Follow the Kubernetes documentation and set the desired limits in a configuration file. Create a config map in the vCluster namespace that contains the configuration file: admission-control.yaml ``` apiVersion: v1 kind: ConfigMap metadata: name: admission-control namespace: vcluster-my-vcluster data: admission-control.yaml: | apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: EventRateLimit configuration: apiVersion: eventratelimit.admission.k8s.io/v1alpha1 kind: Configuration limits: - type: Server qps: 50 burst: 100 ``` 2. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --enable-admission-plugins=EventRateLimit - --admission-control-config-file=/etc/kubernetes/admission-control.yaml statefulSet: persistence: addVolumes: - name: admission-control configMap: name: admission-control addVolumeMounts: - name: admission-control mountPath: /etc/kubernetes ``` 3. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 4. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--enable-admission-plugins` argument is set to a value that includes EventRateLimit. **Expected results:** ``` 'EventRateLimit' argument exist as a string value in the --enable-admission-plugins list. ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --enable-admission-plugins=EventRateLimit --admission-control-config-file=/etc/kubernetes/admission-control.yaml ``` ### 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set[​](#1210-ensure-that-the-admission-control-plugin-alwaysadmit-is-not-set "Direct link to 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that if the `--enable-admission-plugins` argument is set, its value does not include AlwaysAdmit. **Expected results:** ``` 'AlwaysAdmit' argument does not exist as a string value in the --enable-admission-plugins list. ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --enable-admission-plugins=EventRateLimit --admission-control-config-file=/etc/kubernetes/admission-control.yaml ``` ### 1.2.11 Ensure that the admission control plugin AlwaysPullImages is set[​](#1211-ensure-that-the-admission-control-plugin-alwayspullimages-is-set "Direct link to 1.2.11 Ensure that the admission control plugin AlwaysPullImages is set") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --enable-admission-plugins=AlwaysPullImages ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--enable-admission-plugins` argument is set to a value that includes AlwaysPullImages. **Expected results:** ``` 'AlwaysPullImages' argument exist as a string value in the --enable-admission-plugins list. ``` **Returned value:** ``` 45 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --enable-admission-plugins=AlwaysPullImages ``` ### 1.2.12 Ensure that the admission control plugin ServiceAccount is set[​](#1212-ensure-that-the-admission-control-plugin-serviceaccount-is-set "Direct link to 1.2.12 Ensure that the admission control plugin ServiceAccount is set") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--disable-admission-plugins` argument is set to a value that does not include ServiceAccount. **Expected results:** ``` --disable-admission-plugins is not set. ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.13 Ensure that the admission control plugin NamespaceLifecycle is set[​](#1213-ensure-that-the-admission-control-plugin-namespacelifecycle-is-set "Direct link to 1.2.13 Ensure that the admission control plugin NamespaceLifecycle is set") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--disable-admission-plugins` argument is set to a value that does not include NamespaceLifecycle. **Expected results:** ``` --disable-admission-plugins is not set. ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.14 Ensure that the admission control plugin NodeRestriction is set[​](#1214-ensure-that-the-admission-control-plugin-noderestriction-is-set "Direct link to 1.2.14 Ensure that the admission control plugin NodeRestriction is set") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --enable-admission-plugins=NodeRestriction ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--enable-admission-plugins` argument is set to a value that includes NodeRestriction. **Expected results:** ``` 'NodeRestriction' argument exist as a string value in the --enable-admission-plugins list. ``` **Returned value:** ``` 44 root 0:03 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --enable-admission-plugins=NodeRestriction ``` ### 1.2.15 Ensure that the --profiling argument is set to false[​](#1215-ensure-that-the---profiling-argument-is-set-to-false "Direct link to 1.2.15 Ensure that the --profiling argument is set to false") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--profiling` argument is set to false. **Expected results:** ``` '--profiling' is equal to 'false' ``` **Returned value:** ``` 45 root 0:04 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.16 Ensure that the --audit-log-path argument is set[​](#1216-ensure-that-the---audit-log-path-argument-is-set "Direct link to 1.2.16 Ensure that the --audit-log-path argument is set") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --audit-log-path=/var/log/audit.log statefulSet: persistence: addVolumes: - name: audit-log emptyDir: {} addVolumeMounts: - name: audit-log mountPath: /var/log ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--audit-log-path` argument is set as appropriate. **Expected results:** ``` '--audit-log-path' is present ``` **Returned value:** ``` 45 root 0:03 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --audit-log-path=/var/log/audit.log ``` ### 1.2.17 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate[​](#1217-ensure-that-the---audit-log-maxage-argument-is-set-to-30-or-as-appropriate "Direct link to 1.2.17 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --audit-log-path=/var/log/audit.log - --audit-log-maxage=30 statefulSet: persistence: addVolumes: - name: audit-log emptyDir: {} addVolumeMounts: - name: audit-log mountPath: /var/log ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--audit-log-maxage` argument is set to 30 or as appropriate. **Expected results:** ``` '--audit-log-maxage' is greater or equal to 30 ``` **Returned value:** ``` 45 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --audit-log-path=/var/log/audit.log --audit-log-maxage=30 ``` ### 1.2.18 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate[​](#1218-ensure-that-the---audit-log-maxbackup-argument-is-set-to-10-or-as-appropriate "Direct link to 1.2.18 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --audit-log-path=/var/log/audit.log - --audit-log-maxbackup=10 statefulSet: persistence: addVolumes: - name: audit-log emptyDir: {} addVolumeMounts: - name: audit-log mountPath: /var/log ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--audit-log-maxbackup` argument is set to 10 or as appropriate. **Expected results:** ``` '--audit-log-maxbackup' is greater or equal to 10 ``` **Returned value:** ``` 44 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --audit-log-path=/var/log/audit.log --audit-log-maxbackup=10 ``` ### 1.2.19 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate[​](#1219-ensure-that-the---audit-log-maxsize-argument-is-set-to-100-or-as-appropriate "Direct link to 1.2.19 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --audit-log-path=/var/log/audit.log - --audit-log-maxsize=100 statefulSet: persistence: addVolumes: - name: audit-log emptyDir: {} addVolumeMounts: - name: audit-log mountPath: /var/log ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--audit-log-maxsize` argument is set to 100 or as appropriate. **Expected results:** ``` '--audit-log-maxsize' is greater or equal to 100 ``` **Returned value:** ``` 43 root 0:01 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --audit-log-path=/var/log/audit.log --audit-log-maxsize=100 ``` ### 1.2.20 Ensure that the --request-timeout argument is set as appropriate[​](#1220-ensure-that-the---request-timeout-argument-is-set-as-appropriate "Direct link to 1.2.20 Ensure that the --request-timeout argument is set as appropriate") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --request-timeout=300s ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--request-timeout` argument is either not set or set to an appropriate value. **Expected results:** ``` '--request-timeout' is set to 300s ``` **Returned value:** ``` 43 root 0:03 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --request-timeout=300s ``` ### 1.2.21 Ensure that the --service-account-lookup argument is set to true[​](#1221-ensure-that-the---service-account-lookup-argument-is-set-to-true "Direct link to 1.2.21 Ensure that the --service-account-lookup argument is set to true") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --service-account-lookup=true ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that if the `--service-account-lookup` argument exists it is set to true. **Expected results:** ``` '--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' ``` **Returned value:** ``` 43 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --service-account-lookup=true ``` ### 1.2.22 Ensure that the --service-account-key-file argument is set as appropriate[​](#1222-ensure-that-the---service-account-key-file-argument-is-set-as-appropriate "Direct link to 1.2.22 Ensure that the --service-account-key-file argument is set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--service-account-key-file` argument exists and is set as appropriate. **Expected results:** ``` '--service-account-key-file' argument exists and is set appropriately ``` **Returned value:** ``` 45 root 0:01 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.23 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate[​](#1223-ensure-that-the---etcd-certfile-and---etcd-keyfile-arguments-are-set-as-appropriate "Direct link to 1.2.23 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true backingStore: etcd: embedded: enabled: true ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--etcd-certfile` and `--etcd-keyfile` arguments exist and they are set as appropriate. **Expected results:** ``` '--etcd-certfile' is present AND '--etcd-keyfile' is present ``` **Returned value:** ``` 47 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/data/pki/etcd/ca.crt --etcd-certfile=/data/pki/apiserver-etcd-client.crt --etcd-keyfile=/data/pki/apiserver-etcd-client.key --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.24 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate[​](#1224-ensure-that-the---tls-cert-file-and---tls-private-key-file-arguments-are-set-as-appropriate "Direct link to 1.2.24 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the --tls-cert-file and --tls-private-key-file arguments exist and they are set as appropriate. **Expected results:** ``` '--tls-cert-file' is present AND '--tls-private-key-file' is present ``` **Returned value:** ``` 45 root 0:01 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.25 Ensure that the --client-ca-file argument is set as appropriate[​](#1225-ensure-that-the---client-ca-file-argument-is-set-as-appropriate "Direct link to 1.2.25 Ensure that the --client-ca-file argument is set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--client-ca-file` argument exists and it is set as appropriate. **Expected results:** ``` '--client-ca-file' is present ``` **Returned value:** ``` 45 root 0:01 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.26 Ensure that the --etcd-cafile argument is set as appropriate[​](#1226-ensure-that-the---etcd-cafile-argument-is-set-as-appropriate "Direct link to 1.2.26 Ensure that the --etcd-cafile argument is set as appropriate") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true backingStore: etcd: embedded: enabled: true ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--etcd-cafile` argument exists and it is set as appropriate. **Expected results:** ``` '--etcd-cafile' is present ``` **Returned value:** ``` 47 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/data/pki/etcd/ca.crt --etcd-certfile=/data/pki/apiserver-etcd-client.crt --etcd-keyfile=/data/pki/apiserver-etcd-client.key --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` ### 1.2.27 Ensure that the --encryption-provider-config argument is set as appropriate[​](#1227-ensure-that-the---encryption-provider-config-argument-is-set-as-appropriate "Direct link to 1.2.27 Ensure that the --encryption-provider-config argument is set as appropriate") **Result:** PASS 1. Follow the Kubernetes documentation and configure an EncryptionConfig file. Generate a 32-byte key using the following command: ``` head -c 32 /dev/urandom | base64 ``` 2. Create an encryption configuration file with base64 encoded key created previously: encryption-config.yaml ``` apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: - identity: {} ``` 3. Create a secret in the vCluster namespace from the configuration file: ``` kubectl create secret generic encryption-config --from-file=encryption-config.yaml -n vcluster-my-vcluster ``` 4. Create the vCluster referring to the secret as follows: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --encryption-provider-config=/etc/encryption/encryption-config.yaml statefulSet: persistence: addVolumes: - name: encryption-config secret: secretName: encryption-config addVolumeMounts: - name: encryption-config mountPath: /etc/encryption readOnly: true ``` 5. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 6. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` Verify that the `--encryption-provider-config` argument is set to an EncryptionConfig file. Additionally, ensure that the EncryptionConfig file has all the desired resources covered especially any secrets. **Expected results:** ``` '--encryption-provider-config' is present ``` **Returned value:** ``` 45 root 0:02 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/12 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=unix:///data/kine.sock --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false --encryption-provider-config=/etc/encryption/encryption-config.yaml ``` ### 1.2.28 Ensure that encryption providers are appropriately configured[​](#1228-ensure-that-encryption-providers-are-appropriately-configured "Direct link to 1.2.28 Ensure that encryption providers are appropriately configured") **Result:** PASS 1. Follow the same configuration steps as in control 1.2.27 to set up encryption providers. 2. Create the vCluster using the values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- cat /etc/encryption/encryption-config.yaml ``` Verify that `aescbc`, `kms`, or `secretbox` is set as the encryption provider for all the desired resources. **Expected results:** ``` aescbc is set as the encryption provider for the configured resources ``` **Returned value:** ``` apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: - identity: {} ``` ### 1.2.29 Ensure that the API Server only makes use of Strong Cryptographic Ciphers[​](#1229-ensure-that-the-api-server-only-makes-use-of-strong-cryptographic-ciphers "Direct link to 1.2.29 Ensure that the API Server only makes use of Strong Cryptographic Ciphers") **Result:** PASS 1. Pass the following configuration as arguments to the API Server while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: - --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- sh -c "pgrep -f 'kube-apiserver' | head -1 | xargs -I {} cat /proc/{}/cmdline | tr '\0' ' '" ``` Verify that the `--tls-cipher-suites` argument is set with approved cipher suites. **Expected results:** ``` '--tls-cipher-suites' contains valid elements from approved cipher suite list ``` **Returned value:** ``` /binaries/kube-apiserver --service-cluster-ip-range=10.128.0.0/16 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/data/pki/etcd/ca.crt --etcd-certfile=/data/pki/apiserver-etcd-client.crt --etcd-keyfile=/data/pki/apiserver-etcd-client.key --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file=/data/pki/apiserver.key --profiling=false --advertise-address=127.0.0.1 --endpoint-reconciler-type=none --kubelet-client-certificate=/data/pki/apiserver-kubelet-client.crt --kubelet-client-key=/data/pki/apiserver-kubelet-client.key --endpoint-reconciler-type=none --egress-selector-config-file=/data/konnectivity/egress.yaml --admission-control-config-file=/etc/kubernetes/admission-control.yaml --anonymous-auth=false --audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/var/log/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --enable-admission-plugins=AlwaysPullImages,DenyServiceExternalIPs,EventRateLimit,NodeRestriction --encryption-provider-config=/etc/encryption/encryption-config.yaml --feature-gates=RotateKubeletServerCertificate=true --request-timeout=300s --service-account-lookup=true --kubelet-certificate-authority=/data/pki/ca.crt --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 ``` ## 1.3 Controller Manager[​](#13-controller-manager "Direct link to 1.3 Controller Manager") ### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate[​](#131-ensure-that-the---terminated-pod-gc-threshold-argument-is-set-as-appropriate "Direct link to 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate") **Result:** PASS 1. Pass the following configuration while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true controllerManager: extraArgs: - --terminated-pod-gc-threshold=12500 ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` Verify that the `--terminated-pod-gc-threshold` argument is set as appropriate. **Expected results:** ``` '--terminated-pod-gc-threshold' is present ``` **Returned value:** ``` 98 root 0:01 /binaries/kube-controller-manager --service-cluster-ip-range=10.96.0.0/12 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/data/pki/client-ca.crt --cluster-name=kubernetes --cluster-signing-cert-file=/data/pki/server-ca.crt --cluster-signing-key-file=/data/pki/server-ca.key --horizontal-pod-autoscaler-sync-period=60s --kubeconfig=/data/pki/controller-manager.conf --node-monitor-grace-period=180s --node-monitor-period=30s --pvclaimbinder-sync-period=60s --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --root-ca-file=/data/pki/server-ca.crt --service-account-private-key-file=/data/pki/sa.key --use-service-account-credentials=true --leader-elect=false --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl --terminated-pod-gc-threshold=12500 ``` ### 1.3.2 Ensure that the --profiling argument is set to false[​](#132-ensure-that-the---profiling-argument-is-set-to-false "Direct link to 1.3.2 Ensure that the --profiling argument is set to false") **Result:** PASS 1. Pass the following configuration while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true controllerManager: extraArgs: - --profiling=false ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` Verify that the `--profiling` argument is set to false. **Expected results:** ``` '--profiling' is equal to 'false' ``` **Returned value:** ``` 98 root 0:00 /binaries/kube-controller-manager --service-cluster-ip-range=10.96.0.0/12 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/data/pki/client-ca.crt --cluster-name=kubernetes --cluster-signing-cert-file=/data/pki/server-ca.crt --cluster-signing-key-file=/data/pki/server-ca.key --horizontal-pod-autoscaler-sync-period=60s --kubeconfig=/data/pki/controller-manager.conf --node-monitor-grace-period=180s --node-monitor-period=30s --pvclaimbinder-sync-period=60s --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --root-ca-file=/data/pki/server-ca.crt --service-account-private-key-file=/data/pki/sa.key --use-service-account-credentials=true --leader-elect=false --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl --profiling=false ``` ### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true[​](#133-ensure-that-the---use-service-account-credentials-argument-is-set-to-true "Direct link to 1.3.3 Ensure that the --use-service-account-credentials argument is set to true") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` Verify that the `--use-service-account-credentials` argument is set to true. **Expected results:** ``` '--use-service-account-credentials' is not equal to 'false' ``` **Returned value:** ``` 102 root 0:00 /binaries/kube-controller-manager --service-cluster-ip-range=10.96.0.0/12 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/data/pki/client-ca.crt --cluster-name=kubernetes --cluster-signing-cert-file=/data/pki/server-ca.crt --cluster-signing-key-file=/data/pki/server-ca.key --horizontal-pod-autoscaler-sync-period=60s --kubeconfig=/data/pki/controller-manager.conf --node-monitor-grace-period=180s --node-monitor-period=30s --pvclaimbinder-sync-period=60s --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --root-ca-file=/data/pki/server-ca.crt --service-account-private-key-file=/data/pki/sa.key --use-service-account-credentials=true --leader-elect=false --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl ``` ### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate[​](#134-ensure-that-the---service-account-private-key-file-argument-is-set-as-appropriate "Direct link to 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` Verify that the `--service-account-private-key-file` argument is set as appropriate. **Expected results:** ``` '--service-account-private-key-file' is present ``` **Returned value:** ``` 102 root 0:00 /binaries/kube-controller-manager --service-cluster-ip-range=10.96.0.0/12 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/data/pki/client-ca.crt --cluster-name=kubernetes --cluster-signing-cert-file=/data/pki/server-ca.crt --cluster-signing-key-file=/data/pki/server-ca.key --horizontal-pod-autoscaler-sync-period=60s --kubeconfig=/data/pki/controller-manager.conf --node-monitor-grace-period=180s --node-monitor-period=30s --pvclaimbinder-sync-period=60s --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --root-ca-file=/data/pki/server-ca.crt --service-account-private-key-file=/data/pki/sa.key --use-service-account-credentials=true --leader-elect=false --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl ``` ### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate[​](#135-ensure-that-the---root-ca-file-argument-is-set-as-appropriate "Direct link to 1.3.5 Ensure that the --root-ca-file argument is set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` Verify that the `--root-ca-file` argument exists and is set to a certificate bundle file containing the root certificate for the API server's serving certificate. **Expected results:** ``` '--root-ca-file' is present ``` **Returned value:** ``` 102 root 0:00 /binaries/kube-controller-manager --service-cluster-ip-range=10.96.0.0/12 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/data/pki/client-ca.crt --cluster-name=kubernetes --cluster-signing-cert-file=/data/pki/server-ca.crt --cluster-signing-key-file=/data/pki/server-ca.key --horizontal-pod-autoscaler-sync-period=60s --kubeconfig=/data/pki/controller-manager.conf --node-monitor-grace-period=180s --node-monitor-period=30s --pvclaimbinder-sync-period=60s --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --root-ca-file=/data/pki/server-ca.crt --service-account-private-key-file=/data/pki/sa.key --use-service-account-credentials=true --leader-elect=false --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl ``` ### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true[​](#136-ensure-that-the-rotatekubeletservercertificate-argument-is-set-to-true "Direct link to 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` By default, RotateKubeletServerCertificate is set to "true". Verify that it has not been disabled. **Expected results:** ``` --feature-gates=RotateKubeletServerCertificate=false is not present ``` **Returned value:** ``` 102 root 0:00 /binaries/kube-controller-manager --service-cluster-ip-range=10.96.0.0/12 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/data/pki/client-ca.crt --cluster-name=kubernetes --cluster-signing-cert-file=/data/pki/server-ca.crt --cluster-signing-key-file=/data/pki/server-ca.key --horizontal-pod-autoscaler-sync-period=60s --kubeconfig=/data/pki/controller-manager.conf --node-monitor-grace-period=180s --node-monitor-period=30s --pvclaimbinder-sync-period=60s --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --root-ca-file=/data/pki/server-ca.crt --service-account-private-key-file=/data/pki/sa.key --use-service-account-credentials=true --leader-elect=false --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl ``` ### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1[​](#137-ensure-that-the---bind-address-argument-is-set-to-127001 "Direct link to 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-controller-manager ``` Verify that the `--bind-address` argument is set to 127.0.0.1 **Expected results:** ``` '--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present ``` **Returned value:** ``` 102 root 0:00 /binaries/kube-controller-manager --service-cluster-ip-range=10.96.0.0/12 --authentication-kubeconfig=/data/pki/controller-manager.conf --authorization-kubeconfig=/data/pki/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/data/pki/client-ca.crt --cluster-name=kubernetes --cluster-signing-cert-file=/data/pki/server-ca.crt --cluster-signing-key-file=/data/pki/server-ca.key --horizontal-pod-autoscaler-sync-period=60s --kubeconfig=/data/pki/controller-manager.conf --node-monitor-grace-period=180s --node-monitor-period=30s --pvclaimbinder-sync-period=60s --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --root-ca-file=/data/pki/server-ca.crt --service-account-private-key-file=/data/pki/sa.key --use-service-account-credentials=true --leader-elect=false --controllers=*,-nodeipam,-nodelifecycle,-persistentvolume-binder,-attachdetach,-persistentvolume-expander,-cloud-node-lifecycle,-ttl ``` ## 1.4 Scheduler[​](#14-scheduler "Direct link to 1.4 Scheduler") ### 1.4.1 Ensure that the --profiling argument is set to false[​](#141-ensure-that-the---profiling-argument-is-set-to-false "Direct link to 1.4.1 Ensure that the --profiling argument is set to false") **Result:** PASS 1. Pass the following configuration as arguments to the scheduler while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true scheduler: enabled: true extraArgs: - --profiling=false sync: fromHost: nodes: enabled: true ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-scheduler ``` Verify that the `--profiling` argument is set to false. **Expected results:** ``` '--profiling' is equal to 'false' ``` **Returned value:** ``` 98 root 0:01 /binaries/kube-scheduler --authentication-kubeconfig=/data/pki/scheduler.conf --authorization-kubeconfig=/data/pki/scheduler.conf --bind-address=127.0.0.1 --kubeconfig=/data/pki/scheduler.conf --leader-elect=false --profiling=false ``` ### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1[​](#142-ensure-that-the---bind-address-argument-is-set-to-127001 "Direct link to 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1") **Result:** PASS 1. Pass the following configuration while creating the vCluster: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true scheduler: enabled: true sync: fromHost: nodes: enabled: true ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-scheduler ``` Verify that the --bind-address argument is set to 127.0.0.1 **Expected results:** ``` '--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present ``` **Returned value:** ``` 88 root 0:00 /binaries/kube-scheduler --authentication-kubeconfig=/data/pki/scheduler.conf --authorization-kubeconfig=/data/pki/scheduler.conf --bind-address=127.0.0.1 --kubeconfig=/data/pki/scheduler.conf --leader-elect=false ``` --- # etcd configuration Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** The following section covers security areas related to etcd configuration. The assessment focuses on encryption of sensitive data at rest on server, applications and in transit. Assessment focus for vCluster involves verifying correct authentication mechanisms are used and safeguarding the data at rest and in transit via TLS encryption. Key areas include validating that certificate-based authentication is properly configured and that automatic TLS generation is disabled in favor of explicit certificate management. Control numbering The control numbers used throughout this guide (2.1, 2.2, etc.) correlate directly to the official CIS Kubernetes Benchmark control numbers. This allows you to cross-reference with the official CIS documentation and maintain consistency with standard security frameworks. ## 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate[​](#21-ensure-that-the---cert-file-and---key-file-arguments-are-set-as-appropriate "Direct link to 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` Verify that the `--cert-file` and the `--key-file` arguments are set as appropriate. **Expected Result:** ``` '--cert-file' and '--key-file' arguments are appropriately set ``` **Returned Value:** ``` 31 root 2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster ``` ## 2.2 Ensure that the --client-cert-auth argument is set to true[​](#22-ensure-that-the---client-cert-auth-argument-is-set-to-true "Direct link to 2.2 Ensure that the --client-cert-auth argument is set to true") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` Verify that the `--client-cert-auth` argument is set to true. **Expected Result:** ``` '--client-cert-auth' is set to 'true' ``` **Returned Value:** ``` 31 root 2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster ``` ## 2.3 Ensure that the --auto-tls argument is not set to true[​](#23-ensure-that-the---auto-tls-argument-is-not-set-to-true "Direct link to 2.3 Ensure that the --auto-tls argument is not set to true") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` Verify that if the `--auto-tls` argument exists, it is not set to true. **Expected Result:** ``` '--auto-tls' argument does not exist ``` **Returned Value:** ``` 31 root 2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster ``` ## 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate[​](#24-ensure-that-the---peer-cert-file-and---peer-key-file-arguments-are-set-as-appropriate "Direct link to 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` Verify that the `--peer-cert-file` and `--peer-key-file` arguments are set as appropriate. **Expected Result:** ``` '--peer-cert-file' and '--peer-key-file' arguments are appropriately set ``` **Returned Value:** ``` 31 root 2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster ``` ## 2.5 Ensure that the --peer-client-cert-auth argument is set to true[​](#25-ensure-that-the---peer-client-cert-auth-argument-is-set-to-true "Direct link to 2.5 Ensure that the --peer-client-cert-auth argument is set to true") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` Verify that the `--peer-client-cert-auth` argument is set to true. **Expected Result:** ``` '--peer-client-cert-auth' is set to 'true' ``` **Returned Value:** ``` 31 root 2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster ``` ## 2.6 Ensure that the --peer-auto-tls argument is not set to true[​](#26-ensure-that-the---peer-auto-tls-argument-is-not-set-to-true "Direct link to 2.6 Ensure that the --peer-auto-tls argument is not set to true") **Result:** PASS **Audit:** Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` Verify that if the `--peer-auto-tls` argument exists, it is not set to true. **Expected Result:** ``` '--peer-auto-tls' argument does not exist ``` **Returned Value:** ``` 31 root 2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster ``` ## 2.7 Ensure that a unique Certificate Authority is used for etcd[​](#27-ensure-that-a-unique-certificate-authority-is-used-for-etcd "Direct link to 2.7 Ensure that a unique Certificate Authority is used for etcd") **Result:** PASS 1. Run the following command against the vCluster pod: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep etcd ``` Note the file referenced by the `--trusted-ca-file` argument. 2. Run the following command and note the file referenced by `--client-ca-file`: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- ps -ef | grep kube-apiserver ``` 3. Verify that the file referenced by the `--client-ca-file` for apiserver is different from the `--trusted-ca-file` used by etcd. **Expected Result:** ``` The file referenced by the --client-ca-file for api-server is different from the --trusted-ca-file ``` **Returned Value:** ``` 31 root 2:41 etcd --data-dir=/data/etcd --advertise-client-urls=https://emb-0.emb-headless.emb:2379 --initial-advertise-peer-urls=https://emb-0.emb-headless.emb:2380 --initial-cluster-token=vcluster --listen-client-urls=https://0.0.0.0:2379 --listen-metrics-urls=http://0.0.0.0:2381 --listen-peer-urls=https://0.0.0.0:2380 --name=emb-0 --heartbeat-interval=500 --election-timeout=5000 --experimental-watch-progress-notify-interval=5s --experimental-peer-skip-client-san-verification --log-level=info --snapshot-count=10000 --log-outputs=stderr --logger=zap --client-cert-auth=true --cert-file=/data/pki/etcd/server.crt --key-file=/data/pki/etcd/server.key --peer-client-cert-auth=true --peer-key-file=/data/pki/etcd/peer.key --peer-cert-file=/data/pki/etcd/peer.crt --peer-trusted-ca-file=/data/pki/etcd/ca.crt --trusted-ca-file=/data/pki/etcd/ca.crt --initial-cluster=emb-0= https://emb-0.emb-headless.emb:2380 --initial-cluster-state=new --force-new-cluster ``` ``` 47 root 6:43 /binaries/kube-apiserver --advertise-address=127.0.0.1 --service-cluster-ip-range=10.96.0.0/16 --bind-address=127.0.0.1 --allow-privileged=true --authorization-mode=RBAC --client-ca-file=/data/pki/client-ca.crt --enable-bootstrap-token-auth=true --etcd-servers=https://127.0.0.1:2379 --etcd-cafile=/data/pki/etcd/ca.crt --etcd-certfile=/data/pki/apiserver-etcd-client.crt --etcd-keyfile=/data/pki/apiserver-etcd-client.key --proxy-client-cert-file=/data/pki/front-proxy-client.crt --proxy-client-key-file=/data/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/data/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer= https://kubernetes.default.svc.cluster.local --service-account-key-file=/data/pki/sa.pub --service-account-signing-key-file=/data/pki/sa.key --tls-cert-file=/data/pki/apiserver.crt --tls-private-key-file= /data/pki/apiserver.key --endpoint-reconciler-type=none --profiling=false ``` --- # Kubernetes policies Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** The following section covers specific security policies for Kubernetes elements. The assessment focuses on RBAC configuration and least privilege access, Pod Security Standards and security contexts, and network policies and CNI security. This section is directly applicable to vCluster environments. Verification involves ensuring proper RBAC configurations are in place, Pod Security Standards are enforced, and network policies are appropriately configured within the tenant cluster. Control numbering The control numbers used throughout this guide (5.1.1, 5.2.1, etc.) correlate directly to the official CIS Kubernetes Benchmark control numbers. This allows you to cross-reference with the official CIS documentation and maintain consistency with standard security frameworks. ## Validation using kube-bench[​](#validation-using-kube-bench "Direct link to Validation using kube-bench") While most CIS Kubernetes Benchmark checks are not directly verifiable via kube-bench due to vCluster's virtualized control plane architecture, the Policies section (Section 5) includes controls that are relevant and testable within the tenant cluster environment. These include namespace-level restrictions, pod security standards, and security contexts that can be enforced from within vCluster. To validate your vCluster's compliance with this section, you can run kube-bench with a customized kubeconfig pointing to the vCluster API server. The following steps outline the procedure to perform the compliance check: 1. Create the vCluster but don't connect to it yet: ``` vcluster create my-vcluster --connect=false ``` 2. Wait for the vCluster to be ready, get the kubeconfig, then connect to it: ``` vcluster connect my-vcluster --server `kubectl get svc -n vcluster-my-vcluster my-vcluster -o jsonpath='{.status.loadBalancer.ingress[0].ip}'` --print > kubeconfig.yaml && vcluster connect my-vcluster ``` 3. Create a dedicated namespace for the kube-bench job: ``` kubectl create namespace kube-bench ``` 4. Load the vCluster kubeconfig into a secret: ``` kubectl create secret generic my-kubeconfig-secret \ --from-file=kubeconfig=./kubeconfig.yaml \ -n kube-bench ``` 5. Harden service accounts (*Recommended*): To maintain compliance with service account token mounting policies, disable automatic token mounting for all service accounts in the cluster: ``` for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do for sa in $(kubectl get sa -n $ns -o jsonpath='{.items[*].metadata.name}'); do kubectl patch serviceaccount "$sa" \ -p '{"automountServiceAccountToken": false}' \ -n "$ns" done done ``` This is aligned with CIS to ensure that default service accounts are not actively used and ensures that tokens are not unnecessarily exposed. 6. Run the kube-bench job. Deploy a short-lived job to run kube-bench using the kubeconfig you mounted: ``` kubectl apply -f - < | Assessment sections | PASS | WARN | NOT APPLICABLE | FAIL | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---- | ---- | -------------- | ---- | | [Control plane security configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/private-nodes/control-plane-components.md) | 48 | 0 | 11 | 0 | | [etcd node configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/private-nodes/etcd.md) | 7 | 0 | 0 | 0 | | [Control plane configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/private-nodes/control-plane.md) | 1 | 4 | 0 | 0 | | [Worker node security configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/private-nodes/worker-node.md) | 23 | 0 | 1 | 0 | | [Kubernetes Policies](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/private-nodes/policies.md) | 7 | 26 | 0 | 2 | Review the assessment sections and map them to your vCluster configuration to ensure that your hardened deployment follows security best practices, as well as is auditable and defensible in a compliance context. --- # Worker node configuration Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** The following section provides security recommendations for components running on Kubernetes worker nodes. The assessment focuses on Kubelet configuration and security and file system permissions. Assessment focus for vCluster involves verifying and ensuring the private node meets these requirements. Control numbering The control numbers used throughout this guide (4.1.1, 4.2.1, etc.) correlate directly to the official CIS Kubernetes Benchmark control numbers. This allows you to cross-reference with the official CIS documentation and maintain consistency with standard security frameworks. ## 4.1 Worker node configuration files[​](#41-worker-node-configuration-files "Direct link to 4.1 Worker node configuration files") ### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive[​](#411-ensure-that-the-kubelet-service-file-permissions-are-set-to-600-or-more-restrictive "Direct link to 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive") **Result:** PASS Run the audit command to verify the permissions on the service file. If they do not match the expected result, run the following command to set the appropriate permissions: ``` chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf ``` **Audit:** Run the following command against each node: ``` stat -c permissions=%a /etc/systemd/system/kubelet.service.d/10-kubeadm.conf ``` Verify that the permissions are 600 or more restrictive. **Expected results:** ``` permissions has value 600, expected 600 or more restrictive ``` **Returned value:** ``` permissions=600 ``` ### 4.1.2 Ensure that the kubelet service file ownership is set to root:root[​](#412-ensure-that-the-kubelet-service-file-ownership-is-set-to-root "Direct link to 412-ensure-that-the-kubelet-service-file-ownership-is-set-to-root") **Result:** PASS **Audit:** Run the following command against each node: ``` stat -c %U:%G /etc/systemd/system/kubelet.service.d/10-kubeadm.conf ``` Verify that the ownership is set to root:root. **Expected results:** ``` root:root ``` **Returned value:** ``` root:root ``` ### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive[​](#413-if-proxy-kubeconfig-file-exists-ensure-permissions-are-set-to-600-or-more-restrictive "Direct link to 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive") **Result:** PASS 1. Connect to the vCluster and run the following command to extract the kube-proxy pod name. ``` KUBE_PROXY_POD=$(kubectl get pods -n kube-system -l k8s-app=kube-proxy -o jsonpath='{.items[0].metadata.name}') ``` 2. Save the below file for use with the debug container. custom.json ``` { "volumeMounts": [ { "name": "kube-proxy", "mountPath": "/var/lib/kube-proxy" } ] } ``` **Audit:** note When both serverTLSBootstrap is true and RotateKubeletServerCertificate feature is enabled, instead of self signing a serving certificate, the Kubelet will request a certificate from the 'certificates.k8s.io' API. Ensure to approve the certificate signing requests (CSR) using `kubectl certificate approve `, for the following debug command to function. Run the following command against the kube-proxy pod: ``` kubectl debug --custom custom.json -it $KUBE_PROXY_POD --image=busybox --target=kube-proxy --namespace kube-system --profile=general -q -- stat -L -c permissions=%a /var/lib/kube-proxy/kubeconfig.conf ``` Verify that the permissions are 600 or more restrictive. **Expected results:** ``` permissions has value 600, expected 600 or more restrictive ``` **Returned value:** ``` permissions=600 ``` ### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root[​](#414-if-proxy-kubeconfig-file-exists-ensure-ownership-is-set-to-root "Direct link to 414-if-proxy-kubeconfig-file-exists-ensure-ownership-is-set-to-root") **Result:** PASS 1. Connect to the vCluster and run the following command to extract the kube-proxy pod name. ``` KUBE_PROXY_POD=$(kubectl get pods -n kube-system -l k8s-app=kube-proxy -o jsonpath='{.items[0].metadata.name}') ``` 2. Save the below file for use with the debug container. custom.json ``` { "volumeMounts": [ { "name": "kube-proxy", "mountPath": "/var/lib/kube-proxy" } ] } ``` **Audit:** note When both serverTLSBootstrap is true and RotateKubeletServerCertificate feature is enabled, instead of self signing a serving certificate, the Kubelet will request a certificate from the 'certificates.k8s.io' API. Ensure to approve the certificate signing requests (CSR) using `kubectl certificate approve `, for the following debug command to function. Run the following command against the kube-proxy pod: ``` kubectl debug --custom custom.json -it $KUBE_PROXY_POD --image=busybox --target=kube-proxy --namespace kube-system --profile=general -q -- stat -c %U:%G /var/lib/kube-proxy/kubeconfig.conf ``` Verify that the ownership is set to root:root. **Expected results:** ``` root:root ``` **Returned value:** ``` root:root ``` ### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive[​](#415-ensure-that-the---kubeconfig-kubeletconf-file-permissions-are-set-to-600-or-more-restrictive "Direct link to 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive") **Result:** PASS **Audit:** Run the following command against each node: ``` stat -c permissions=%a /etc/kubernetes/kubelet.conf ``` Verify that the permissions are 600 or more restrictive. **Expected results:** ``` permissions has value 600, expected 600 or more restrictive ``` **Returned value:** ``` permissions=600 ``` ### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root[​](#416-ensure-that-the---kubeconfig-kubeletconf-file-ownership-is-set-to-root "Direct link to 416-ensure-that-the---kubeconfig-kubeletconf-file-ownership-is-set-to-root") **Result:** PASS **Audit:** Run the following command against each node: ``` stat -c %U:%G /etc/kubernetes/kubelet.conf ``` Verify that the ownership is set to root:root. **Expected results:** ``` root:root ``` **Returned value:** ``` root:root ``` ### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive[​](#417-ensure-that-the-certificate-authorities-file-permissions-are-set-to-600-or-more-restrictive "Direct link to 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive") **Result:** PASS Run the audit command to verify the permissions on the certificate authorities file. If they do not match the expected result, run the following command to set the appropriate permissions: ``` chmod 600 /etc/kubernetes/pki/ca.crt ``` **Audit:** Run the following command against each node: ``` stat -c permissions=%a /etc/kubernetes/pki/ca.crt ``` Verify that the permissions are 600 or more restrictive. **Expected results:** ``` permissions has value 600, expected 600 or more restrictive ``` **Returned value:** ``` permissions=600 ``` ### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root[​](#418-ensure-that-the-client-certificate-authorities-file-ownership-is-set-to-root "Direct link to 418-ensure-that-the-client-certificate-authorities-file-ownership-is-set-to-root") **Result:** PASS **Audit:** Run the following command against each node: ``` stat -c %U:%G /etc/kubernetes/pki/ca.crt ``` Verify that the ownership is set to root:root. **Expected results:** ``` root:root ``` **Returned value:** ``` root:root ``` ### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 600 or more restrictive[​](#419-ensure-that-the-kubelet---config-configuration-file-has-permissions-set-to-600-or-more-restrictive "Direct link to 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 600 or more restrictive") **Result:** PASS Run the audit command to verify the permissions on the certificate authorities file. If they do not match the expected result, run the following command to set the appropriate permissions: ``` chmod 600 /var/lib/kubelet/config.yaml ``` **Audit:** Run the following command against each node: ``` stat -c permissions=%a /var/lib/kubelet/config.yaml ``` Verify that the permissions are 600 or more restrictive. **Expected results:** ``` permissions has value 600, expected 600 or more restrictive ``` **Returned value:** ``` permissions=600 ``` ### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root[​](#4110-ensure-that-the-kubelet---config-configuration-file-ownership-is-set-to-root "Direct link to 4110-ensure-that-the-kubelet---config-configuration-file-ownership-is-set-to-root") **Result:** PASS **Audit:** Run the following command against each node: ``` stat -c %U:%G /var/lib/kubelet/config.yaml ``` Verify that the ownership is set to root:root. **Expected results:** ``` root:root ``` **Returned value:** ``` root:root ``` ## 4.2 Kubelet[​](#42-kubelet "Direct link to 4.2 Kubelet") ### 4.2.1 Ensure that the --anonymous-auth argument is set to false[​](#421-ensure-that-the---anonymous-auth-argument-is-set-to-false "Direct link to 4.2.1 Ensure that the --anonymous-auth argument is set to false") **Result:** PASS **Audit:** If using a Kubelet configuration file, check that there is an entry for authentication:anonymous: enabled set to false. Run the following command against each node: ``` cat /var/lib/kubelet/config.yaml ``` Verify that the corresponding entry is set to false in the Kubelet config file. **Expected results:** ``` ... authentication: anonymous: enabled: false ... ``` **Returned value:** ``` ... authentication: anonymous: enabled: false ... ``` ### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow[​](#422-ensure-that-the---authorization-mode-argument-is-not-set-to-alwaysallow "Direct link to 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow") **Result:** PASS **Audit:** If using a Kubelet configuration file, check that there is an entry that sets authorization: mode to something other than AlwaysAllow. Run the following command against each node: ``` cat /var/lib/kubelet/config.yaml ``` Verify that the corresponding entry is set to Webhook in the Kubelet config file. **Expected results:** ``` ... authorization: mode: Webhook ... ``` **Returned value:** ``` ... authorization: mode: Webhook ... ``` ### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate[​](#423-ensure-that-the---client-ca-file-argument-is-set-as-appropriate "Direct link to 4.2.3 Ensure that the --client-ca-file argument is set as appropriate") **Result:** PASS **Audit:** If using a Kubelet configuration file, check that there is an entry that sets authentication: x509: clientCAFile to the location of the client certificate authority file. Run the following command against each node: ``` cat /var/lib/kubelet/config.yaml ``` Verify that the corresponding entry is set to the appropriate file in the Kubelet config file. **Expected results:** ``` ... authentication: x509: clientCAFile: /etc/kubernetes/pki/ca.crt ... ``` **Returned value:** ``` ... authentication: x509: clientCAFile: /etc/kubernetes/pki/ca.crt ... ``` ### 4.2.4 Ensure that the --read-only-port argument is set to 0[​](#424-ensure-that-the---read-only-port-argument-is-set-to-0 "Direct link to 4.2.4 Ensure that the --read-only-port argument is set to 0") **Result:** PASS **Audit:** If using a Kubelet configuration file, check that it does not set readOnlyPort to any value other than 0. Run the following command against each node: ``` cat /var/lib/kubelet/config.yaml ``` Verify that readOnlyPort either does not exist or is not set to a value other than 0. **Expected results:** ``` 'readOnlyPort' does not exist in the config ``` **Returned value:** ``` 'readOnlyPort' does not exist in the config ``` ### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0[​](#425-ensure-that-the---streaming-connection-idle-timeout-argument-is-not-set-to-0 "Direct link to 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0") **Result:** PASS **Audit:** If using a Kubelet configuration file, check that it does not set streamingConnectionIdleTimeout to 0. Run the following command against each node: ``` cat /var/lib/kubelet/config.yaml ``` Verify that streamingConnectionIdleTimeout is not set to 0. **Expected results:** ``` ... streamingConnectionIdleTimeout: 4h0m0s ... ``` **Returned value:** ``` ... streamingConnectionIdleTimeout: 4h0m0s ... ``` ### 4.2.6 Ensure that the --make-iptables-util-chains argument is set to true[​](#426-ensure-that-the---make-iptables-util-chains-argument-is-set-to-true "Direct link to 4.2.6 Ensure that the --make-iptables-util-chains argument is set to true") **Result:** PASS **Audit:** If using a Kubelet configuration file, check that it does not set makeIPTablesUtilChains to false. Run the following command against each node: ``` cat /var/lib/kubelet/config.yaml ``` Verify that makeIPTablesUtilChains is not set to false. **Expected results:** ``` ... makeIPTablesUtilChains: true ... ``` **Returned value:** ``` ... makeIPTablesUtilChains: true ... ``` ### 4.2.7 Ensure that the --hostname-override argument is not set[​](#427-ensure-that-the---hostname-override-argument-is-not-set "Direct link to 4.2.7 Ensure that the --hostname-override argument is not set") **Result:** PASS **Audit:** Run the following command against each node: ``` cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf ``` Verify that --hostname-override argument does not exist. **Expected results:** ``` --hostname-override argument does not exist. ``` **Returned value:** ``` --hostname-override argument does not exist. ``` ### 4.2.8 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture[​](#428-ensure-that-the-eventrecordqps-argument-is-set-to-a-level-which-ensures-appropriate-event-capture "Direct link to 4.2.8 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture") **Result:** PASS **Audit:** If using a Kubelet configuration file, check that there is an entry to set eventRecordQPS: to an appropriate level. Run the following command against each node: ``` cat /var/lib/kubelet/config.yaml ``` Verify that eventRecordQPS is set to an appropriate level for the cluster. **Expected results:** ``` ... eventRecordQPS: 50 ... ``` **Returned value:** ``` ... eventRecordQPS: 50 ... ``` ### 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate[​](#429-ensure-that-the---tls-cert-file-and---tls-private-key-file-arguments-are-set-as-appropriate "Direct link to 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate") **Result:** NOT APPLICABLE `--tls-cert-file` and `--tls-private-key-file` are mutually exclusive with the `serverTLSBootstrap: true` setting (also recommended in Control 4.2.11). These flags are required to be set when generating and distributing your own certificates to nodes. While serverTLSBootstrap: true switches to using the CSR certificate bootstrapping method, which bootstraps as well as auto-rotates the certificates. ### 4.2.10 Ensure that the --rotate-certificates argument is not set to false[​](#4210-ensure-that-the---rotate-certificates-argument-is-not-set-to-false "Direct link to 4.2.10 Ensure that the --rotate-certificates argument is not set to false") **Result:** PASS **Audit:** If using a Kubelet configuration file, check that there is an entry to set rotateCertificates: to true. Run the following command against each node: ``` cat /var/lib/kubelet/config.yaml ``` Verify that rotateCertificates is set to true. **Expected results:** ``` ... rotateCertificates: true ... ``` **Returned value:** ``` ... rotateCertificates: true ... ``` ### 4.2.11 Verify that the RotateKubeletServerCertificate argument is set to true[​](#4211-verify-that-the-rotatekubeletservercertificate-argument-is-set-to-true "Direct link to 4.2.11 Verify that the RotateKubeletServerCertificate argument is set to true") **Result:** PASS 1. This check can be ignored if serverTLSBootstrap is true in the kubelet config file. Pass the following configuration while creating the vCluster: vcluster.yaml ``` privateNodes: enabled: true kubelet: config: serverTLSBootstrap: true featureGates: RotateKubeletServerCertificate: true ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against each node: ``` cat /var/lib/kubelet/config.yaml ``` Verify that serverTLSBootstrap is set to true. **Expected results:** ``` ... serverTLSBootstrap: true ... ``` **Returned value:** ``` ... serverTLSBootstrap: true ... ``` note When both serverTLSBootstrap is true and RotateKubeletServerCertificate feature is enabled, instead of self signing a serving certificate, the Kubelet will request a certificate from the 'certificates.k8s.io' API. This would require an approver to approve the certificate signing requests (CSR). ### 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers[​](#4212-ensure-that-the-kubelet-only-makes-use-of-strong-cryptographic-ciphers "Direct link to 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers") **Result:** PASS 1. Pass the following configuration while creating the vCluster: vcluster.yaml ``` privateNodes: enabled: true kubelet: config: tlsCipherSuites: - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384" - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256" ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against each node: ``` cat /var/lib/kubelet/config.yaml ``` Verify that the tlsCipherSuites field is set with one of the cipher suites listed below ``` TLS_AES_128_GCM_SHA256 TLS_AES_256_GCM_SHA384 TLS_CHACHA20_POLY1305_SHA256 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_128_GCM_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_256_GCM_SHA384 ``` **Expected results:** ``` ... tlsCipherSuites: - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 ... ``` **Returned value:** ``` ... tlsCipherSuites: - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 ... ``` ### 4.2.13 Ensure that a limit is set on pod PIDs[​](#4213-ensure-that-a-limit-is-set-on-pod-pids "Direct link to 4.2.13 Ensure that a limit is set on pod PIDs") **Result:** PASS 1. Pass the following configuration while creating the vCluster: vcluster.yaml ``` privateNodes: enabled: true kubelet: config: podPidsLimit: 1000000 ``` 2. Create the vCluster using the above values file: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 3. Run the following command against each node: ``` cat /var/lib/kubelet/config.yaml ``` Verify that --pod-max-pids is set correctly. **Expected results:** ``` ... podPidsLimit: 1000000 ... ``` **Returned value:** ``` ... podPidsLimit: 1000000 ... ``` ## 4.3 kube-proxy[​](#43-kube-proxy "Direct link to 4.3 kube-proxy") ### 4.3.1 Ensure that the kube-proxy metrics service is bound to localhost[​](#431-ensure-that-the-kube-proxy-metrics-service-is-bound-to-localhost "Direct link to 4.3.1 Ensure that the kube-proxy metrics service is bound to localhost") **Result:** PASS **Audit:** kube-proxy runs as a pod inside the vCluster and its configuration is backed by a ConfigMap named "kube-proxy" in the kube-system namespace. Run the following command inside the tenant cluster: ``` kubectl get cm kube-proxy -n kube-system -o jsonpath='{.data.config\.conf}' | grep metricsBindAddress ``` Verify that metricsBindAddress is set to default. **Expected results:** ``` metricsBindAddress: "" ``` **Returned value:** ``` metricsBindAddress: "" ``` --- # Deploy in rootless mode By default, vCluster deploys the vCluster control plane pod as a root user on the control plane cluster, but you can adjust vCluster to deploy on a control plane cluster as a non-root user. There is no upgrade from a vCluster running as a root user to a vCluster running with a non-root user, so this needs to be enabled upon initial deployment. ## Configure to run as non-root user[​](#configure-to-run-as-non-root-user "Direct link to Configure to run as non-root user") You can set the desired UID for the control plane and deployed CoreDNS pod (if deployed). Example of non-root user configuration ``` controlPlane: statefulSet: security: podSecurityContext: fsGroup: 12345 containerSecurityContext: runAsUser: 12345 runAsNonRoot: true ``` ## Other predeployment configuration options[​](#other-predeployment-configuration-options "Direct link to Other predeployment configuration options") Before deploying, it's recommended to review the set of configuration options that cannot be updated post deployment. These options require deploying a new vCluster instead of upgrading your vCluster with new options. ### Control Plane Options[​](#control-plane-options "Direct link to Control Plane Options") Decide the various options of how you want your control plane deployed: * **[High availability](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/high-availability.md)** - Run multiple copies of vCluster components. * **[Rootless mode](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/rootless-mode.md)** - Deploy the vCluster pod without root access to the control plane cluster. * **Backing Store** - Decide how the data of your cluster is stored. Backing store options vCluster supports etcd or a relational database (using KINE) as the backend. This feature provides flexibility to vCluster operators. The available data store options allow you to select a data store that fits your use case. vCluster supports the following datastore options: * [Embedded SQLite](https://www.sqlite.org/index.html) (default with `PersistentVolume` (PV)) * [PostgreSQL](https://www.postgresql.org/) * [MySQL](https://www.mysql.com/) * [MariaDB](https://mariadb.org/) * [etcd](https://etcd.io/) warning After deploying your vCluster, there are limited migration paths to change your backing store. Review the backing store migration options before deploying. **Backing store options** * Embedded SQLite (Default) * Embedded SQLite (No PV) * Embedded etcd * Deployed etcd * MySQL / MariaDB * PostgreSQL This is the default, so you don't need to configure anything. If you want to explicitly set this option, you can use: ``` controlPlane: backingStore: database: embedded: enabled: true ``` By default, vCluster stores its data in a `PersistentVolumeClaim` (PVC). Alternatively, you can use an [`emptyDir`](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume to store tenant cluster data. To use an `emptyDir` to store the data instead of a `PersistentVolume`, create a `values.yaml` with the following contents: ``` controlPlane: statefulSet: persistence: volumeClaim: enabled: true ``` Then upgrade or recreate the vCluster with: ``` vcluster create my-vcluster -n my-vcluster --upgrade -f values.yaml ``` Potential data loss This method should only be used for testing purposes, as data is lost upon pod recreation. This is an enterprise feature that allows you to deploy etcd within each vCluster to enable high availability (HA), which isn’t supported with embedded SQLite: ``` controlPlane: backingStore: etcd: embedded: enabled: true ``` This deploys an etcd instance outside of the vCluster control plane pod that is used as a backing store: ``` controlPlane: backingStore: etcd: deploy: enabled: true ``` warning vCluster snapshots aren't supported with this backing store. See [Back up and restore](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md#back-up-and-restore). The option for MySQL and MariaDB typically has the following format: ``` controlPlane: backingStore: database: external: enabled: true dataSource: mysql://username:password@tcp(hostname:3306)/database-name ``` If you specify a database name and it does not exist, the server attempts to create it. The option for PostgreSQL typically has the following format: ``` controlPlane: backingStore: database: external: enabled: true dataSource: postgres://username:password@hostname:port/database-name ``` More advanced configuration parameters are available. For more information, see . If you specify a database name and it does not exist, the server attempts to create it. ### Worker Nodes[​](#worker-nodes "Direct link to Worker Nodes") Decide where you want your worker nodes to come from: * **[Nodes from the control plane cluster](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes.md)** - (Default) All worker nodes of the shared control plane cluster are used by the tenant cluster and all resources are synced to the single namespace that the vCluster is deployed on. * **[Syncing Namespaces](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md)** - Resources are synced to mapped namespaces on the control plane cluster. * **[Isolated workloads](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes/isolated-workloads.md)** - Different options to isolate a workload in a vCluster. * **[Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md)** - Enable adding individual nodes to the tenant cluster. ## Deploy vCluster[​](#deploy-vcluster "Direct link to Deploy vCluster") YAML configuration If you're not sure which options to configure, you can update most settings later by upgrading your vCluster with an updated `vcluster.yaml`. However, some settings — such as what type of worker nodes or the backing store — can only be set during the initial deployment and cannot be changed during an upgrade. All of the deployment options below have the following assumptions: * A `vcluster.yaml` is provided. Refer to the `vcluster.yaml` reference docs to explore all configuration options. This file is optional and can be removed from the examples. * The vCluster is called `my-vcluster`. * The vCluster is deployed into the `team-x` namespace.
* vCluster CLI * Helm * Terraform * ArgoCD * Flux * Cluster API The vCluster CLI provides the most straightforward way to deploy and manage virtual clusters. 1. Install the vCluster CLI: * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` 2. Deploy vCluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` vcluster create my-vcluster --namespace team-x --values vcluster.yaml ``` note After installation, vCluster automatically switches your Kubernetes context to the new virtual cluster. You can now run `kubectl` commands against the virtual cluster. Helm provides fine-grained control over the deployment process and integrates well with existing Helm-based workflows. 1. Deploy vCluster using the `helm upgrade` command: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CONFIG FILEvcluster.yaml NAMESPACEteam-x ``` helm upgrade --install my-vcluster vcluster \ --values vcluster.yaml \ --repo https://charts.loft.sh \ --namespace team-x \ --repository-config='' \ --create-namespace ``` You can use Terraform to deploy vCluster as code with version control and state management. 1. Create a `main.tf` file to define your vCluster deployment using the Terraform Helm provider: ``` provider "helm" { kubernetes = { config_path = "~/.kube/config" } } resource "helm_release" "my_vcluster" { name = "my-vcluster" namespace = "team-x" create_namespace = true repository = "https://charts.loft.sh" chart = "vcluster" # If you didn't create a vcluster.yaml, remove the values section. values = [ file("${path.module}/vcluster.yaml") ] } ``` Helm Provider Version This configuration uses the Terraform Helm provider v3.x syntax where `kubernetes` is defined as an argument (`kubernetes = {`). If you're using Helm provider v2.x, use the block syntax instead (`kubernetes {`). To use v3.x, ensure your provider version is at least v3.0.0. 2. Install the required [Helm provider](https://registry.terraform.io/providers/hashicorp/helm/latest) and initialize Terraform: ``` terraform init ``` 3. Generate a plan to preview the changes: ``` terraform plan ```
Review the plan output to verify connectivity and proposed changes. 4. Deploy vCluster: ``` terraform apply ``` ArgoCD deployment enables GitOps workflows for vCluster management, and provides automated deployment, drift detection, and declarative configuration management through Git repositories. To deploy vCluster using ArgoCD, you need the following files: * `vcluster.yaml` for your vCluster configuration options. * `-app.yaml` for your ArgoCD `Application` definition. Replace `` with your actual cluster name.
1. Create the ArgoCD `Application` file `-app.yaml`, which references the vCluster Helm chart: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x ``` --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-vcluster namespace: argocd spec: project: default source: chart: vcluster repoURL: https://charts.loft.sh helm: releaseName: my-vcluster valueFiles: - vcluster.yaml destination: server: https://kubernetes.default.svc namespace: team-x ``` 2. Commit and push these files to your configured ArgoCD repository. 3. Sync your ArgoCD repository with your configured cluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster ``` argocd app sync my-vcluster ``` 1) Create a `HelmRepository` source in the Git repository that Flux monitors, so Flux can fetch the vCluster Helm charts automatically. In this example, the related files are stored under the `clusters` folder. Save the following file as `clusters/sources/vcluster-repository.yaml`: Modify the following with your specific values to generate a copyable command: REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system ``` --- apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository metadata: name: vcluster namespace: flux-system spec: interval: 1h url: https://charts.loft.sh ``` 2) The vCluster will be deployed into the namespace `team-x` of the host cluster. If this namespace doesn’t already exist, create it with: Modify the following with your specific values to generate a copyable command: NAMESPACEteam-x ``` kubectl create namespace team-x ``` 3) Create a vCluster `HelmRelease` file in your Git repository. This `HelmRelease` tells Flux how to deploy a vCluster in your Kubernetes cluster using the configured Helm charts in Step 1. Save the following file as `clusters/production/vcluster-demo.yaml`: Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system SECRET NAMEvcluster-flux-kubeconfig ``` --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: vcluster-demo namespace: team-x spec: interval: 10m chart: spec: chart: vcluster version: "0.28.x" sourceRef: kind: HelmRepository name: vcluster namespace: flux-system values: # Configure TLS SAN for the certificate controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" coredns: enabled: true exportKubeConfig: # Set a meaningful context name context: default # Use a server URL that matches the TLS SAN server: https://vcluster-demo.team-x.svc.cluster.local:443 # Skip TLS verification when Flux connects to the vCluster insecure: true # Specify the secret where the KubeConfig is stored secret: name: vcluster-flux-kubeconfig sync: toHost: ingresses: enabled: true ``` The content in `vcluster.yaml` should be placed under the `values`. Unlike ArgoCD, Flux requires the KubeConfig secret to be able to access the vCluster's API server and deploy to vCluster. The `exportKubeConfig` configuration under `values`: * Exports the virtual cluster KubeConfig as a Secret in the host namespace * Makes the Secret available for Flux to use with the `spec.kubeConfig` field * Uses a server URL that is accessible from the Flux controllers (replace `vcluster-name` and `team-x` with your actual values) * Sets `insecure: true` to automatically skip TLS certificate verification * Adds a TLS SAN (Subject Alternative Name) that matches the server URL, which helps prevent certificate verification errors KubeConfig Secret key The vCluster `exportKubeConfig` configuration creates a Secret with the KubeConfig data stored under the key `config`. When referring to this Secret in Flux resources, you must specify this key in the `secretRef.key` field, as shown in the examples below. Modify the following with your specific values to generate a copyable command: SECRET NAMEvcluster-flux-kubeconfig ``` # In Flux HelmRelease spec: kubeConfig: secretRef: name: vcluster-flux-kubeconfig key: config # Must match the key used in the vCluster-generated Secret ``` Certificate verification considerations When using vCluster with Flux, proper TLS certificate configuration is essential: 1. Set `exportKubeConfig.insecure: true` in your vCluster configuration 2. Configure proper TLS SANs with the `--tls-san` flag in vCluster configuration 3. Ensure the server URL matches the certificate's SAN Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x ``` # In your vCluster configuration controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" exportKubeConfig: server: https://vcluster-demo.team-x.svc.cluster.local:443 insecure: true ``` 4) After adding the `HelmRelease` and any supporting files to your Git repository, commit and push them: Modify the following with your specific values to generate a copyable command: FOLDER NAMEclusters ``` git add clusters/ git commit -m "Add vCluster demo configuration" git push ``` Once the changes are pushed, Flux will automatically detect them and deploy the vCluster according to the configuration in your repository. Cluster API (CAPI) provides lifecycle management for Kubernetes clusters. The vCluster CAPI provider enables you to manage virtual clusters using the same declarative APIs and tooling used for physical clusters. For more details, see the [Cluster API Provider for vCluster documentation](https://github.com/loft-sh/cluster-api-provider-vcluster). 1. Install the [`clusterctl` CLI](https://cluster-api.sigs.k8s.io/user/quick-start.html#install-clusterctl). 2. Install the vCluster provider: ``` clusterctl init --infrastructure vcluster:v0.2.0 ``` 3. Export environment variables for the Cluster API provider to create the manifest. The manifest is applied to your Kubernetes cluster, which deploys a vCluster. Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` export CLUSTER_NAME=my-vcluster export CLUSTER_NAMESPACE=team-x export VCLUSTER_YAML=$(awk '{printf "%s\n", $0}' vcluster.yaml) ``` 4. Create the namespace for the vCluster using the exported variable: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x ``` kubectl create namespace team-x ``` 5. Generate the required manifests and apply them using the exported variables: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CLUSTER NAMESPACEteam-x ``` clusterctl generate cluster my-vcluster \ --infrastructure vcluster \ --target-namespace team-x \ | kubectl apply -f - ``` Kubernetes version The Kubernetes version for the vCluster is not set at the CAPI provider command. Configure it in the `vcluster.yaml` file based on your Kubernetes distribution. 6. Wait for vCluster to become ready by monitoring the vCluster custom resource status: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x CLUSTER NAMEmy-vcluster ``` kubectl wait --for=condition=ready vcluster -n team-x my-vcluster --timeout=300s ``` ## Next steps[​](#next-steps "Direct link to Next steps") Now that you have vCluster running, consider exploring the [platform UI](https://vcluster.com/docs/platform/install/quick-start-guide.md) to manage your tenant clusters. --- # Control plane sizing and performance This page explains how to evaluate and tune a Tenant Cluster's Virtual Control Plane for your environment. It covers the evaluation framework, monitoring thresholds, resource profile starting points, and validated tuning settings. For the test methodology and analysis behind the profiles, see [Benchmark methodology](https://vcluster.com/docs/vcluster/deploy/control-plane/benchmark-methodology.md). For CoreDNS, a control-plane-managed component with its own scaling path, see [Scale CoreDNS for large node counts](https://vcluster.com/docs/vcluster/deploy/control-plane/coredns-scaling.md). ## Scope and caveats[​](#scope-and-caveats "Direct link to Scope and caveats") Control plane only The numbers on this page describe the capacity of the Virtual Control Plane (API server, controller manager, scheduler, and embedded etcd) under simulated load. They don't describe data-plane capacity, networking throughput, storage backend behavior, or the runtime cost of your actual workloads. **The load is simulated, not real.** Tests use [KWOK](https://kwok.sigs.k8s.io/) fake nodes with no real container runtime or kubelet. [kube-burner](https://kube-burner.github.io/kube-burner/) drives object lifecycles to stress the control plane's bookkeeping, watch fan-out, and etcd write path. The tests don't cover real pod startup and shutdown latency, CNI throughput, container image pulls, NetworkPolicy enforcement, or application behavior. Treat these results as an upper bound. Your workload's true ceiling is lower with significant data-plane demands. **There is no published mapping from KWOK fake-node counts to real-node counts.** Real workloads have data-plane costs that KWOK doesn't model. If your customer-visible claim is a node count, validate it on real nodes at the scale that matters. ## How to evaluate your environment[​](#how-to-evaluate-your-environment "Direct link to How to evaluate your environment") The recommended approach is to run the benchmark framework against a representative subset of your real configuration. Match your own operators, CRDs, and object density. Measure the control-plane response as it runs. The same SLI checks the framework uses internally make a portable monitoring template you can apply to any Tenant Cluster. ### Run the framework[​](#run-the-framework "Direct link to Run the framework") The framework is open source: [loft-sh/vCluster-Control-Plane-Load-Testing](https://github.com/loft-sh/vcluster-control-plane-load-testing). It includes per-profile EKS Terraform modules, kube-burner workload configurations, the 13-SLI evaluation checks, and the doubling-ladder and binary-search scripts. See [Benchmark methodology](https://vcluster.com/docs/vcluster/deploy/control-plane/benchmark-methodology.md) for test setup details. Customize the workload (`kube-burner/configs/`) to match your tenant. Adjust CRD mix, operator density, and churn pattern to your own workload, then run the ladder against your candidate profile. ### What to monitor[​](#what-to-monitor "Direct link to What to monitor") These are the 13 conditions the framework evaluates after each stage. They also make a useful production monitoring set: | # | Condition | Threshold | | -- | ------------------------------ | ----------------------------- | | 1 | Control plane replica Ready | Any pod not Ready → fail | | 2a | Container restart count | > 2 → fail | | 2b | OOMKilled | Any event → fail | | 3a | Sustained 429 responses | > 0.1 req/s → fail | | 3b | Sustained 5xx responses | > 0.1 req/s → fail | | 3c | Sustained request terminations | > 0.1 req/s → fail | | 4 | API P99 latency (non-watch) | > 2 s sustained → fail | | 5a | etcd leader lost | Any interval → fail | | 5b | etcd leader changes | Any → fail | | 5c | etcd proposals failed | Any → fail | | 5d | etcd WAL fsync P99 | > 25 ms → warn | | 5e | etcd DB size | > 4 GB → warn; > 12 GB → fail | | 6 | Node convergence | < 98% of target N → fail | If your workload trips any condition against a candidate profile, scale up to the next profile or apply the tuning settings below. For production alert thresholds, see [Operational guidance](#operational-guidance). ### When to validate on real nodes[​](#when-to-validate-on-real-nodes "Direct link to When to validate on real nodes") Synthetic testing covers the control-plane portion of sizing. Real validation is needed for: * **Pod startup latency SLOs** — image pulls, scheduling under contention, and kubelet binding latency aren't modeled by KWOK. * **Network programming p99** — Service, EndpointSlice, kube-proxy, and CNI interactions aren't modeled. * **CRD-heavy operator workloads** — each CRD adds a separate watch cache and informer that KWOK doesn't exercise. * **Real container runtime behavior** — image pulls, probes, and evictions create event storms that synthetic load can't reproduce. If your customer-visible claim is a node count or pod count, validate it on real nodes at the scale that matters. ## Choose a resource profile[​](#choose-a-resource-profile "Direct link to Choose a resource profile") The following profiles are **per-replica** resource limits for the Virtual Control Plane StatefulSet. Always run 3 replicas in HA mode for any production Tenant Cluster. | Scale | Profile | CPU | Memory | Notes | | --------------------- | ------- | --- | ------ | ------------------------------------------------------------------------------------------------------------------------- | | Dev / sandbox | P1 | 1 | 2 GiB | Light, single-team CI environments | | Small team | P2 | 2 | 4 GiB | Up to a few hundred long-running services | | Standard production | P3 | 4 | 8 GiB | Most production tenants — recommended starting point | | Large production | P4 | 8 | 16 GiB | Heavy churn or dense deployments | | Enterprise / AI Cloud | P5 | 16 | 32 GiB | High-density GPU clusters; operator-rich workloads such as cert-manager, external-secrets, GPU Operator, Argo, and KServe | | Hyperscale | P6 | 32 | 64 GiB | Latency headroom above P5; same node ceiling, marginal API p99 improvement | **Key principles:** * **P3 (4 CPU / 8 GiB) is the right default** for most production Tenant Clusters. * **Below 4 CPU**, node capacity scales roughly linearly with CPU. The control plane saturates CPU before the etcd write path becomes the constraint. * **Above 4 CPU**, the ceiling flattens. The etcd write path becomes the binding constraint, not CPU. Additional resources buy latency headroom, not more nodes. * **Step up to P5** when you have operator-rich workloads (20+ controllers issuing persistent watches) or are hitting latency-sensitive SLOs under sustained churn. * **Memory scales with object count, not node count.** Pod, Deployment, and Service count drives etcd DB size. Plan memory based on expected object density, not node count alone. * **Always run 3-replica HA.** A single replica can't tolerate even a brief etcd compaction stall without impacting tenant API availability. To configure a resource profile, set `controlPlane.statefulSet.resources` in `vcluster.yaml`: vcluster.yaml — P5 resource profile, 3-replica HA ``` controlPlane: statefulSet: highAvailability: replicas: 3 resources: limits: cpu: "16" memory: 32Gi requests: cpu: "16" memory: 32Gi ``` ## Tune your control plane[​](#tune-your-control-plane "Direct link to Tune your control plane") We validated these settings across all profiles under sustained synthetic load. The same knobs apply to real workloads, though gains depend on your traffic mix. ### Go garbage collection pacing[​](#go-garbage-collection-pacing "Direct link to Go garbage collection pacing") Set on the Virtual Control Plane container: vcluster.yaml — Go GC tuning (P5 example) ``` controlPlane: statefulSet: env: - name: GOGC value: "200" # default is 100; doubling halves GC overhead - name: GOMEMLIMIT value: "24000MiB" # ~75% of the pod memory limit ``` `GOGC=200` halves GC overhead at the cost of slightly higher steady-state memory use. `GOMEMLIMIT` prevents OOM events by giving the Go runtime a hard ceiling to plan against. This combination meaningfully reduces API p99 latency under sustained churn. Set `GOMEMLIMIT` to about 75% of your profile's memory limit. For example, use `12000MiB` for a P4 16 GiB pod, or `6000MiB` for a P3 8 GiB pod. ### etcd storage class[​](#etcd-storage-class "Direct link to etcd storage class") Embedded etcd is sensitive to WAL fsync latency. Use a high-IOPS StorageClass for the etcd PersistentVolume: vcluster.yaml — etcd storage ``` controlPlane: backingStore: etcd: embedded: enabled: true statefulSet: persistence: volumeClaim: storageClass: size: 100Gi ``` Create a StorageClass with at least 16K IOPS and 500 MB/s throughput. Generic cloud defaults push WAL fsync p99 into the 25–50 ms range under heavy churn, above the 25 ms warn threshold. For example, AWS `gp3` defaults to 3K IOPS and 125 MB/s. A high-IOPS class keeps WAL fsync p99 well within the warn threshold even under sustained load. ### API server concurrency[​](#api-server-concurrency "Direct link to API server concurrency") The Kubernetes defaults (400 / 200) are sufficient for most workloads. For operator-rich Tenant Clusters with many persistent controllers, double them to absorb the extra `LIST` and `WATCH` bookkeeping during churn cycles: vcluster.yaml — API server concurrency (k8s distro) ``` controlPlane: distro: k8s: apiServer: extraArgs: - --max-requests-inflight=800 - --max-mutating-requests-inflight=400 ``` ### Controller manager and scheduler — client throttling[​](#controller-manager-and-scheduler--client-throttling "Direct link to Controller manager and scheduler — client throttling") Both components have `--kube-api-qps` and `--kube-api-burst` settings. The defaults (50 QPS / 100 burst) are too conservative under heavy churn: vcluster.yaml — controller manager and scheduler QPS ``` controlPlane: distro: k8s: controllerManager: extraArgs: - --kube-api-qps=200 - --kube-api-burst=400 scheduler: extraArgs: - --kube-api-qps=200 - --kube-api-burst=400 ``` ### Syncer outbound rate limits[​](#syncer-outbound-rate-limits "Direct link to Syncer outbound rate limits") The vCluster syncer reflects Tenant Cluster objects to the Control Plane Cluster. Under heavy churn, its outbound API rate becomes the next bottleneck after the tenant API server. Set the rate limits using environment variables on the control plane pod: vcluster.yaml — syncer outbound rate limits ``` controlPlane: statefulSet: env: - name: VCLUSTER_PHYSICAL_CLIENT_QPS value: "400" - name: VCLUSTER_PHYSICAL_CLIENT_BURST value: "800" ``` ### Rate-limit alignment[​](#rate-limit-alignment "Direct link to Rate-limit alignment") For high-density Tenant Clusters, raise all three rate-limit stacks together. Lifting one without the others shifts the bottleneck rather than removing it: | Stack | QPS | Burst | | --------------------------------------------------------- | --- | ----- | | Tenant controller manager / scheduler → tenant API server | 200 | 400 | | Syncer → Control Plane Cluster API server | 400 | 800 | | Client (kubectl, CI pipelines, and so on) | 100 | 200 | ### Bootstrap replica order[​](#bootstrap-replica-order "Direct link to Bootstrap replica order") When provisioning a new HA Tenant Cluster with embedded etcd, bring up replica 0 first. Wait for it to become Ready before scaling to 3. This avoids a race condition where all three replicas try to become the etcd cluster founder simultaneously. The vCluster CLI handles this automatically. In self-managed Helm pipelines, handle it explicitly. ## Operational guidance[​](#operational-guidance "Direct link to Operational guidance") **Plan Control Plane Cluster node capacity.** A P5 Virtual Control Plane runs comfortably on a `c6a.8xlarge` node. P6 needs `c6a.16xlarge` or larger. Karpenter with `consolidationPolicy: WhenEmpty` and a 5-minute `consolidateAfter` interval works well in steady state. | Metric | Alert threshold | | -------------------------- | ---------------------------- | | API p99 latency | > 1 s | | API 5xx rate | > 0.1 req/s | | etcd WAL fsync p99 | > 25 ms | | etcd DB size | > 4 GB (warn) / 12 GB (fail) | | Dropped watches per second | > 0 sustained | | Node convergence ratio | < 98% | There's no vCluster-published alert threshold for DNS query latency or error rate. Both depend on your own query volume and pod density, not a fixed number this table can give you. See [What to watch](https://vcluster.com/docs/vcluster/deploy/control-plane/coredns-scaling.md#what-to-watch) in [Scale CoreDNS for large node counts](https://vcluster.com/docs/vcluster/deploy/control-plane/coredns-scaling.md) for the metric expressions and how to build your own baseline. **Tenant Isolation is per-Tenant-Cluster.** Each Tenant Cluster is fully isolated. Control-plane sizing is per-tenant, not aggregate. A single Control Plane Cluster can host many Tenant Clusters concurrently, within its own capacity limits. When planning, multiply the per-tenant profile by your expected tenant count. --- # Nodes from the Control Plane Cluster Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Shared nodes are for trusted tenants Shared nodes are a supported, common model for trusted tenants. They give each tenant control-plane, API, and namespace isolation, but tenant workloads share the same kernel and physical nodes. They aren't a security boundary for untrusted tenants with Kubernetes access or arbitrary workload execution. Good fits are development, testing, CI/CD, and internal engineering teams. Use [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for external, resale, regulated, or otherwise untrusted tenant offerings, optionally with [vNode](https://www.vnode.com/docs/) for runtime isolation. NetworkPolicy is an added isolation layer worth enabling even for trusted tenants. vCluster can create the policies for you through `policies.networkPolicy`, and your control plane cluster's CNI enforces them. Confirm your CNI supports enforcement, since some accept NetworkPolicy resources without acting on them. See the [security baseline](https://vcluster.com/docs/vcluster/security/.md). If you accept shared nodes for a trusted-tenant use case, follow the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) as the required next step. With shared nodes, a tenant cluster's worker nodes are the worker nodes of the control plane cluster it's deployed on. These nodes are called host nodes because they belong to the control plane cluster. A tenant cluster uses shared nodes unless you enable [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) with `privateNodes.enabled`. Shared nodes are the vCluster OSS default and the only worker node model OSS provides. Private nodes require a vCluster license. warning If you start with host nodes, you can't change the worker node pool to use [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) after the tenant cluster is deployed. ## Understand how host nodes are reported in vCluster[​](#understand-how-host-nodes-are-reported-in-vcluster "Direct link to Understand how host nodes are reported in vCluster") When running `kubectl get nodes` in the tenant cluster kubecontext, host nodes only show up as nodes in the tenant cluster when there are pods from the tenant cluster deployed onto those nodes. There is one exception where you can view all of the host nodes regardless if pods are scheduled on to them, but only if you sync them and select all of them. ### Pseudo nodes (default)[​](#pseudo-nodes-default "Direct link to Pseudo nodes (default)") By default, the host nodes are not synced from the control plane cluster to the tenant cluster and are considered pseudo nodes. Pseudo nodes have real values for name, but everything else is randomly generated. When you run `kubectl get nodes`, the only nodes that show up are the ones that have pods scheduled on them. Any information besides the name is randomly generated. If there are no more pods on that host node, vCluster deletes the pseudo node in the tenant cluster. ### Real nodes (Synced from the control plane cluster)[​](#real-nodes-synced-from-the-control-plane-cluster "Direct link to Real nodes (Synced from the control plane cluster)") If you want accurate node information, you can sync the host nodes from the control plane cluster to the tenant cluster, but syncing real nodes requires a cluster role to access node information. vCluster automatically deploys this cluster role when nodes are synced. Node IP obfuscation By default, vCluster obfuscates node IP addresses when syncing real nodes to protect sensitive information. Learn how to [control node IP visibility](https://vcluster.com/docs/vcluster/learn-how-to/worker-nodes/host-nodes/control-node-ip-visibility.md) for your use case. When you run `kubectl get nodes`, the only nodes that show up are the ones that have pods scheduled on them. All information about the nodes is correct except for IP addresses. If there are no more pods on a node, vCluster deletes the node in the tenant cluster. Sync node information from the control plane cluster ``` sync: fromHost: nodes: enabled: true ``` Read more about [other options when syncing nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md). ## Determine which host nodes to use[​](#determine-which-host-nodes-to-use "Direct link to Determine which host nodes to use") ### Shared nodes[​](#shared-nodes "Direct link to Shared nodes") By default, vCluster can use **any** node in the control plane cluster to deploy pods and resources on. This means that all the nodes in the control plane cluster can be used, but they will only show up as a node in the vCluster if a pod is deployed on it. #### View all host nodes in the tenant cluster kubecontext[​](#view-all-host-nodes-in-the-tenant-cluster-kubecontext "Direct link to View all host nodes in the tenant cluster kubecontext") If you want to see all available nodes in the control plane cluster regardless if a pod is scheduled onto it, then you sync all the host nodes as well as specify that you want all of them selected. ``` sync: fromHost: nodes: enabled: true selector: all: true ``` ### Dedicated nodes[​](#dedicated-nodes "Direct link to Dedicated nodes") Instead of being able to schedule onto any host node in the vCluster, you can specify a set of worker nodes that are available for the vCluster to schedule pods and resources to. This means that even though vCluster is deployed on a control plane cluster, there can be some isolation achieved by only allowing specific worker nodes to be used by a vCluster. Use label selectors to choose which nodes on the control plane cluster vCluster workloads can be scheduled onto. Set labels on the worker nodes on the shared control plane cluster before using label selectors to select those nodes. Use cases to only select a set of nodes include: * Restrict vCluster to nodes in a specific region. * Target nodes with a particular architecture. * Using only spot or preemptible instances. * Prevent vCluster workloads from running on critical infrastructure nodes. #### Select nodes with a label selector[​](#select-nodes-with-a-label-selector "Direct link to Select nodes with a label selector") By adding a label selector in the `vcluster.yaml`, nodes from the control plane cluster that have the matching labels can be used to schedule pods and resources. This can be used for pseudo nodes or real nodes. **Pseudo Nodes** Selecting a set of pseudo nodes ``` sync: fromHost: nodes: # Pseudo Nodes enabled: false # Select which nodes the vCluster can see selector: labels: environment: production team: backend ``` **Real Nodes** Selecting a set of real nodes ``` sync: fromHost: nodes: # Sync real node information enabled: true # Select which nodes the vCluster can see selector: labels: environment: production team: backend ``` --- # Deploy with isolated workloads Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** vCluster offers several different policies that reduce risk for tenant workloads sharing the control plane cluster's nodes. On shared nodes, these policies are risk-reduction controls, not a security boundary against an untrusted tenant. Review the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) for the full threat model before onboarding trusted tenants, and the [policies documentation](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies.md) for all options within each policy below. To reduce risk for workloads on a vCluster, enable a couple of configuration options: 1. Set a [Pod Security Standard](https://kubernetes.io/docs/concepts/security/pod-security-standards/). If you enable either `baseline` or `restricted` policy, it follows the standards outlined in Kubernetes. For example, with `baseline` as a Pod Security Standard, pods that try to run as a privileged container are not synced to the control plane cluster. The same applies to pods that try to mount a host path directly. Pod Security Standards evaluate the Pod spec, so they don't inspect what a PersistentVolumeClaim's backing volume points at. If [`sync.toHost.persistentVolumes`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/persistent-volumes.md) is enabled, a `hostPath` PersistentVolume can still reach a Pod through an ordinary PVC mount. Though Pod Security Standard is a Kubernetes concept and only applicable to certain versions of Kubernetes, vCluster supports this regardless of Kubernetes version as this is directly implemented in vCluster. Rejected pods stay `Pending` in the vCluster and in newer Kubernetes version they are denied by the admission controller. 2. Enable a [resource quota](https://kubernetes.io/docs/concepts/policy/resource-quotas/) as well as a [limit range](https://kubernetes.io/docs/concepts/policy/limit-range/). This allows restricting resource consumption of vCluster workloads. If enabled, sane defaults for those 2 resources are chosen. 3. Enable a [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) that restricts access of vCluster workloads as well as the vCluster control plane to other pods in the control plane cluster. This only works if your control plane [cluster CNI supports network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/#prerequisites). Example of Workload Isolation YAML ``` policies: # empty, baseline, restricted can be used here podSecurityStandard: baseline resourceQuota: enabled: true limitRange: enabled: true networkPolicy: enabled: true ``` info When enabling resource quotas locally, add a `--expose-local=false` flag to your `vcluster create [...]` command, as by default the vCluster CLI tries to automatically expose the vCluster using NodePorts, when interacting with a local Kubernetes cluster. ## Network only isolation[​](#network-only-isolation "Direct link to Network only isolation") By default, workloads created by vCluster are able to communicate with other workloads in the control plane cluster through their cluster IPs. This can be beneficial if you want to purposely access a control plane cluster service, which is a good method to share services between tenant clusters. If you do not want pods running inside one vCluster to have access to other workloads in the control plane cluster, then deploy a [network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) for the specific namespace where vCluster is installed in. vCluster can automatically deploy this network policy in your control plane cluster. Example of enabling network policy ``` policies: networkPolicy: enabled: true ``` warning Network policies do not work in all Kubernetes clusters and need to be supported by the underlying CNI plugin. ## Advanced Isolation[​](#advanced-isolation "Direct link to Advanced Isolation") Besides the basic workload isolation using Pod Security Standard and Resource Quotas, you could always set more advanced isolation methods. ### Isolation with dedicated nodes[​](#isolation-with-dedicated-nodes "Direct link to Isolation with dedicated nodes") You can use `sync.fromHost.nodes.selector.labels` to restrict a tenant cluster to a specific set of control plane cluster nodes. A node selector is a placement mechanism, not a security boundary. It only reduces cross-tenant exposure if the label it selects on is a dedicated tenant-workload label. A general-purpose label also applied to ingress, metrics, storage, VPN, or other infrastructure nodes doesn't provide that isolation. * Give tenant-workload nodes a label and taint that's dedicated to tenant compute, distinct from any label infrastructure nodes carry. * Taint infrastructure and control-plane nodes so they reject tenant workloads outright, rather than relying on tenants not targeting them. * Don't let a tenant's own node selectors or affinity rules broaden placement beyond the reserved node pool. See the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#node-scheduling-and-isolation) for the full scheduling and node isolation model. ### Isolation with private nodes[​](#isolation-with-private-nodes "Direct link to Isolation with private nodes") Instead of using nodes from the shared control plane cluster, you can use [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for the tenant cluster. ## Workload & Network Isolation within the vCluster[​](#workload--network-isolation-within-the-vcluster "Direct link to Workload & Network Isolation within the vCluster") Besides isolating workloads from one tenant cluster to another tenant cluster, normal workload isolation from within the vCluster can also be achieved by deploying resource quotas, limit ranges, admission controllers and network policies in the tenant cluster. To allow network policies to function correctly, you'll need to enable the same configuration options in the deployment of the tenant cluster through the `vcluster.yaml`. ### Secret based Service Account tokens[​](#secret-based-service-account-tokens "Direct link to Secret based Service Account tokens") By default, vCluster creates a service account token for each pod and injects them as an annotation in the respective pod's metadata. If this doesn't comply with your security practices, this can be mitigated by enabling an option in the `vcluster.yaml` which creates separate secrets for each pod's service account token and mounts it accordingly using projected volumes. ``` sync: toHost: pods: useSecretsForSATokens: true ``` --- # Private Nodes Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. Using private nodes is a tenancy model for vCluster where, instead of sharing the control plane cluster's worker nodes, individual worker nodes are joined to a vCluster. These private nodes act as the vCluster's worker nodes and are treated as worker nodes for the vCluster. Because these nodes are real Kubernetes nodes, vCluster does not sync any resources to the control plane cluster as no control plane cluster worker nodes are used. All workloads run directly on the attached nodes as if they were native to the tenant cluster. This approach can be understood as follows: the control plane creates a virtual Kubernetes context, while the workloads can run on separate physical nodes that exist entirely within that virtual context. One key benefit of using private nodes is that vCluster can automatically manage the lifecycle of the worker nodes. To allow nodes to join the tenant cluster, the cluster must be exposed and accessible. One option is to expose a `LoadBalancer` service and use its endpoint for the worker nodes. vCluster does not need direct access to the nodes; instead, it uses [Konnectivity](https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/) to tunnel requests—such as `kubectl logs`—from the control plane to the nodes.
![Private nodes architecture showing dedicated worker nodes per vCluster](/docs/media/diagrams/private-nodes.png) ## How private nodes can be provisioned[​](#how-private-nodes-can-be-provisioned "Direct link to How private nodes can be provisioned") Private nodes can be provisioned in two different ways: * **[Manually provisioned](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md)** - Nodes that were provisioned outside of vCluster. These nodes are joined to vCluster using a vCluster CLI command. * **[Automatically provisioned](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/auto-nodes.md)** - Nodes that are provisioned on-demand based on the vCluster configuration and resource requirements. vCluster is connected to vCluster Platform and references a node provider defined in vCluster Platform. ## vcluster.yaml configuration[​](#vclusteryaml-configuration "Direct link to vcluster.yaml configuration") To create a vCluster with private nodes, you'll need to enable the feature as well as set some other configuration options in preparation for worker nodes to join. Enable private nodes ``` # Enable private nodes privateNodes: enabled: true # vCluster control plane options controlPlane: service: spec: type: LoadBalancer # If you want to expose vCluster using LoadBalancer (Recommended option) # Networking configuration networking: # Specify the pod CIDR podCIDR: 10.64.0.0/16 # Specify the service CIDR serviceCIDR: 10.128.0.0/16 ``` ### Scheduling enabled by default[​](#scheduling-enabled-by-default "Direct link to Scheduling enabled by default") When private nodes are enabled, scheduling is handled by the tenant cluster and the virtual scheduler is automatically enabled. You do not need to manually configure this in your `vcluster.yaml`. Virtual scheduler automatically enabled with private nodes ``` # Enabling private nodes enables the virtual scheduler # Disabling the virtual scheduler is not possible controlPlane: distro: k8s: scheduler: enabled: true ``` ### Other vCluster feature limitations[​](#other-vcluster-feature-limitations "Direct link to Other vCluster feature limitations") Certain vCluster features are automatically disabled or unavailable. If you include these options in your `vcluster.yaml`, they are ignored or might cause configuration errors. The following features are not available: * `sync.*` - No resource syncing between tenant and control plane clusters * `integrations.*` - Integrations depend on syncing functionality * `networking.replicateServices` - Services are not replicated to host * `controlPlane.coredns.embedded: true` - Embedded CoreDNS conflicts with custom CNI * `controlPlane.distro.k8s.scheduler.enabled: false` - Virtual scheduler cannot be disabled * `sleep.*` - No ability to use auto sleep for workloads or control plane Unsupported vcluster.yaml options with private nodes ``` # These configurations are NOT supported with private nodes # Resource syncing between tenant and control plane clusters is disabled sync: toHost: services: enabled: false # Services cannot be synced to the control plane cluster secrets: enabled: false # Secrets cannot be synced to the control plane cluster # All other sync options (pods, configmaps, etc.) are also disabled # Platform integrations require syncing functionality integrations: metricsServer: enabled: false # Metrics server integration not supported # All other integrations are disabled due to sync dependency controlPlane: # DNS configuration limitations coredns: embedded: false # Embedded CoreDNS conflicts with custom CNI options distro: k8s: # Virtual scheduler is required for workload placement scheduler: enabled: true # Always enabled (cannot be disabled) # Host Path Mapper is not supported hostPathMapper: enabled: false # Service replication (networking.replicateServices) and sleep mode # (sleep.*) are also unavailable; see the list above. ``` ### Network policies[​](#network-policies "Direct link to Network policies") If you are using [network policies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md), private nodes traffic into the virtual cluster control plane must be allowed. vcluster.yaml ``` privateNodes: enabled: true controlPlane: service: spec: type: LoadBalancer policies: networkPolicy: enabled: true controlPlane: ingress: - from: # Allow incoming traffic from the load balancer internal IP address. # This example is allowing incoming traffic from any address. Load balancer internal CIDR should be used. - ipBlock: cidr: 0.0.0.0/0 ``` ## Features available with private nodes[​](#features-available-with-private-nodes "Direct link to Features available with private nodes") The following vCluster features are available when using private nodes as your tenancy model: ## Create a vCluster with private nodes[​](#create-a-vcluster-with-private-nodes "Direct link to Create a vCluster with private nodes") ### Create the vCluster control plane with private nodes enabled[​](#create-the-vcluster-control-plane-with-private-nodes-enabled "Direct link to Create the vCluster control plane with private nodes enabled") The vCluster control plane needs to be created with private nodes feature enabled. Private Nodes must be enabled at the start Private nodes need to be enabled from the initial installation of the vCluster. An existing vCluster cannot upgrade from using the control plane cluster's nodes to private nodes. vcluster.yaml for private nodes ``` # Enable private nodes privateNodes: enabled: true # vCluster control plane options controlPlane: service: spec: type: LoadBalancer # If you want to expose vCluster via LoadBalancer (recommended option) # Networking configuration networking: # Specify the pod cidr podCIDR: 10.64.0.0/16 # Specify the service cidr serviceCIDR: 10.128.0.0/16 ``` Connecting to the platform Follow [this guide](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/platform.md) to create an access key and set up the vCluster for deployment using Helm or other tools. ### Join worker nodes[​](#join-worker-nodes "Direct link to Join worker nodes") Ensure your [nodes are ready to join](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/node-requirements.md) the vCluster, before [joining each worker node](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md). Once a worker node has been attached to a vCluster, pods can be scheduled and start to run on these nodes. ### Manage vCluster[​](#manage-vcluster "Direct link to Manage vCluster") Finally, after vCluster is up and running, [manage](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/manage.md) the lifecycle of the vCluster and worker nodes. --- # Automatically Provision and Join Nodes Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** You can configure vCluster to automatically provision and join worker nodes based on the node and resource requirements. To use auto nodes, you need vCluster Platform installed and vCluster needs to be connected to it. The feature is based on [Karpenter](https://karpenter.sh/), which is a cluster autoscaler for Kubernetes that chooses the best node for the requested amount of pods and resources. Karpenter is built into vCluster and does not need to be installed. This allows vCluster to use Karpenter to do the node management and scheduling. The provisioning of the nodes is handled by vCluster Platform.
vCluster provides multiple [node providers](https://vcluster.com/docs/platform/administer/node-providers/overview.md) to automatically create worker nodes: * Terraform * KubeVirt * Nvidia Base Command Manager ## Install vCluster Platform[​](#install-vcluster-platform "Direct link to Install vCluster Platform") vCluster needs to be connected to a running vCluster Platform to enable configuring for nodes to automatically provision based on resource needs. Refer to the [vCluster Platform installation guide](https://www.vcluster.com/docs/platform/install/quick-start-guide). ## Create a node provider[​](#create-a-node-provider "Direct link to Create a node provider") After vCluster Platform is installed, a node provider needs to be created in order for tenant clusters to reference in the `vcluster.yaml` configuration. Read more about [node providers](https://www.vcluster.com/docs/platform/administer/node-providers/overview) and create one. ## Create a vCluster[​](#create-a-vcluster "Direct link to Create a vCluster") Now that vCluster Platform is created with a node provider, you can start creating tenant clusters that are configured with to have automatic nodes join. The tenant clusters must either be created through the vCluster Platform or connected to the vCluster Platform using an [API key](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/platform.md). Read through the [auto nodes configuration details](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/auto-nodes.md) to learn about how to configure your node pools and how to define which nodes from the node provider should be configured. --- # Deploy auto nodes with quick start templates Quick start templates provide a ready-to-go infrastructure-as-code deployment for Kubernetes clusters across the three major public clouds: **AWS, GCP, and Azure**. They are built to: * **Provision essential cloud resources** such as VPCs, networks, subnets, NAT gateways, and node instances. * **Enable full cloud provider integration** by deploying the [Cloud Controller Manager (CCM)](https://kubernetes.io/docs/concepts/architecture/cloud-controller/) and [Container Storage Interface (CSI)](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/), ensuring control-plane, storage, and networking features work seamlessly. * **Speed up onboarding** by getting clusters running with minimal manual effort. ### Get started immediately[​](#get-started-immediately "Direct link to Get started immediately") vCluster config.yaml ``` privateNodes: enabled: true autoNodes: - provider: aws dynamic: - name: aws-1 nodeTypeSelector: - property: instance-type operator: In values: - e2-medium controlPlane: service: spec: type: LoadBalancer networking: podCIDR: 10.64.0.0/16 serviceCIDR: 10.128.0.0/16 ``` ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before deploying a vCluster with auto-nodes, you’ll need to prepare two things: 1. A cloud-specific service account or role that Terraform can use. 2. A NodeProvider configuration that sources one of the quick start templates. ### Terraform service account / role[​](#terraform-service-account--role "Direct link to Terraform service account / role") Terraform requires cloud credentials with sufficient permissions to provision infrastructure.
You can review the required permissions for each provider here: * [AWS permissions](https://github.com/loft-sh/vcluster-auto-nodes-aws/blob/v0.1.0/docs/auto_nodes_policy.json) * [Azure permissions](https://github.com/loft-sh/vcluster-auto-nodes-azure/blob/v0.1.0/docs/auto_nodes_role.json) * [GCP permissions](https://github.com/loft-sh/vcluster-auto-nodes-gcp/blob/v0.1.0/docs/auto_nodes_role.yaml) ### NodeProvider[​](#nodeprovider "Direct link to NodeProvider") It is recommended to use the predefined NodeProvider, which already reference the quick start templates and is ready to use out of the box. They can be selected directly from the vCluster Platform UI. If you prefer to create your own NodeProvider, you can source one of the quick start templates directly: * [AWS](https://github.com/loft-sh/vcluster-auto-nodes-aws/tree/v0.1.0) * [Azure](https://github.com/loft-sh/vcluster-auto-nodes-azure/tree/v0.1.0) * [GCP](https://github.com/loft-sh/vcluster-auto-nodes-gcp/tree/v0.1.0) See the following example: * AWS * Azure * GCP ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: auto-nodes-aws spec: properties: vcluster.com/ccm-enabled: "true" region: "us-east-1" terraform: nodeEnvironmentTemplate: infrastructure: git: repository: https://github.com/loft-sh/vcluster-auto-nodes-aws.git tag: v0.1.0 subPath: environment/infrastructure kubernetes: git: repository: https://github.com/loft-sh/vcluster-auto-nodes-aws.git tag: v0.1.0 subPath: environment/kubernetes nodeTemplate: git: repository: https://github.com/loft-sh/vcluster-auto-nodes-aws.git tag: v0.1.0 subPath: node nodeTypes: - name: t3.small resources: cpu: "2" memory: 2Gi properties: instance-type: t3.small - name: t3.medium resources: cpu: "2" memory: 4Gi properties: instance-type: t3.medium - name: t3.large resources: cpu: "2" memory: 8Gi properties: instance-type: t3.large - name: t3.xlarge resources: cpu: "4" memory: 16Gi properties: instance-type: t3.xlarge - name: t3.2xlarge resources: cpu: "8" memory: 32Gi properties: instance-type: t3.2xlarge ``` ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: auto-nodes-azure spec: properties: vcluster.com/ccm-enabled: "true" location: "eastus" resource-group: "dev" subscription-id: "f1b3f662-ec84-4c23-b590-681997a0f751" terraform: nodeEnvironmentTemplate: infrastructure: git: repository: https://github.com/loft-sh/vcluster-auto-nodes-azure.git tag: v0.1.0 subPath: environment/infrastructure kubernetes: git: repository: https://github.com/loft-sh/vcluster-auto-nodes-azure.git tag: v0.1.0 subPath: environment/kubernetes nodeTemplate: git: repository: https://github.com/loft-sh/vcluster-auto-nodes-azure.git tag: v0.1.0 subPath: node nodeTypes: - name: standard-d2s-v5 resources: cpu: "2" memory: 8Gi properties: instance-type: Standard_D2s_v5 - name: standard-d4s-v5 resources: cpu: "4" memory: 16Gi properties: instance-type: Standard_D4s_v5 - name: standard-d8s-v5 resources: cpu: "8" memory: 32Gi properties: instance-type: Standard_D8s_v5 - name: standard-d16s-v5 resources: cpu: "16" memory: 64Gi properties: instance-type: Standard_D16s_v5 ``` ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: auto-nodes-gcp spec: properties: vcluster.com/ccm-enabled: "true" region: "europe-west1" project: "my-project" terraform: nodeEnvironmentTemplate: infrastructure: git: repository: https://github.com/loft-sh/vcluster-auto-nodes-gcp.git tag: v0.1.0 subPath: environment/infrastructure kubernetes: git: repository: https://github.com/loft-sh/vcluster-auto-nodes-gcp.git tag: v0.1.0 subPath: environment/kubernetes nodeTemplate: git: repository: https://github.com/loft-sh/vcluster-auto-nodes-gcp.git tag: v0.1.0 subPath: node nodeTypes: - name: e2-small resources: cpu: "2" memory: 2Gi properties: instance-type: e2-small - name: e2-medium resources: cpu: "2" memory: 4Gi properties: instance-type: e2-medium - name: e2-standard-2 resources: cpu: "2" memory: 8Gi properties: instance-type: e2-standard-2 - name: e2-standard-4 resources: cpu: "4" memory: 16Gi properties: instance-type: e2-standard-4 - name: e2-standard-8 resources: cpu: "8" memory: 32Gi properties: instance-type: e2-standard-8 - name: e2-standard-16 resources: cpu: "16" memory: 64Gi properties: instance-type: e2-standard-16 ``` ## Configure[​](#configure "Direct link to Configure") You can configure the **NodeProvider** with the following options: | Option | Default Value | Description | | ----------------------------- | --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | `vcluster.com/ccm-enabled` | `true` | Enables deployment of the Cloud Controller Manager (CCM). | | `vcluster.com/ccm-lb-enabled` | `true` | Enables the CCM service controller. If disabled, CCM will not create LoadBalancer services. | | `vcluster.com/csi-enabled` | `true` | Deploys the CSI driver and configures a default storage class (AWS EBS, GCP PD, Azure Disk). | | `vcluster.com/vpc-cidr` | AWS: `10.0.0.0/16`
Azure: `10.5.0.0/16`
GCP: `10.10.0.0/16` | Sets the VPC CIDR range. Useful in multi-cloud scenarios to avoid CIDR conflicts. | Example vCluster config.yaml vCluster config.yaml ``` privateNodes: enabled: true autoNodes: - provider: aws properties: vcluster.com/ccm-enabled: "false" vcluster.com/csi-enabled: "false" vcluster.com/vpc-cidr: "10.30.0.0/16" dynamic: - name: aws-1 nodeTypeSelector: - property: instance-type operator: In values: - e2-medium controlPlane: service: spec: type: LoadBalancer networking: podCIDR: 10.64.0.0/16 serviceCIDR: 10.128.0.0/16 ``` ## Deploy[​](#deploy "Direct link to Deploy") * Cloud * Hybrid Cloud * Multi Cloud Deploy on a cloud provider vCluster config.yaml ``` privateNodes: enabled: true autoNodes: - provider: aws dynamic: - name: aws-1 nodeTypeSelector: - property: instance-type operator: In values: - e2-medium controlPlane: service: spec: type: LoadBalancer networking: podCIDR: 10.64.0.0/16 serviceCIDR: 10.128.0.0/16 ``` Integrate your on-premises data center with the cloud vCluster config.yaml ``` privateNodes: enabled: true vpn: enabled: true nodeToNode: enabled: true autoNodes: - provider: aws properties: vcluster.com/csi-enabled: "false" dynamic: - name: aws-1 nodeTypeSelector: - property: instance-type operator: In values: - t3.medium - provider: bare-metal # your own bare-metal provider static: - name: bare-metal nodeTypeSelector: - property: instance-type operator: In values: - server-1 networking: podCIDR: 10.64.0.0/16 serviceCIDR: 10.128.0.0/16 ``` Combine multiple cloud providers within one vCluster vCluster config.yaml ``` privateNodes: enabled: true vpn: enabled: true nodeToNode: enabled: true autoNodes: - provider: aws properties: vcluster.com/csi-enabled: "false" dynamic: - name: aws-1 nodeLabels: topology.ebs.csi.aws.com/zone: us-east-1a nodeTypeSelector: - property: instance-type operator: In values: - t3.medium - provider: gcp properties: vcluster.com/ccm-lb-enabled: "false" vcluster.com/csi-enabled: "false" dynamic: - name: gcp-1 nodeTypeSelector: - property: instance-type operator: In values: - e2-medium networking: podCIDR: 10.64.0.0/16 serviceCIDR: 10.128.0.0/16 ``` ## Security considerations[​](#security-considerations "Direct link to Security considerations") important Currently, when deploying [Cloud Controller Manager (CCM)](https://kubernetes.io/docs/concepts/architecture/cloud-controller/) and [Container Storage Interface (CSI)](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/) with Auto Nodes, permissions are granted through instance profiles (AWS, Azure) or service accounts/roles (GCP). **This means all worker nodes inherit the same permissions as CCM and CSI**. As a result, **any pod running in the cluster could potentially access the same cloud permissions**. Refer to the full list of permissions below for details. Cluster administrators should be aware of the following: * **Shared permissions** – All pods may gain the same access level as CCM and CSI. * **Mitigation** – To avoid this, cluster admins can disable CCM and CSI deployment. In that case, instance profiles will not receive additional permissions. However, responsibility for deploying and configuring CCM and CSI securely then falls to the cluster admin. * **AWS-specific note** – On AWS, pods run with host networking disabled by default. This prevents them from directly requesting the same credentials as CCM and CSI. * **Review granted permissions** – The full list of permissions can be found in the Terraform IAM definitions for each provider: * [AWS IAM policies](https://github.com/loft-sh/vcluster-auto-nodes-aws/blob/v0.1.0/environment/infrastructure/worker_iam.tf) * [GCP IAM roles](https://github.com/loft-sh/vcluster-auto-nodes-gcp/blob/v0.1.0/environment/infrastructure/iam.tf) * [Azure identity roles](https://github.com/loft-sh/vcluster-auto-nodes-azure/blob/v0.1.0/environment/infrastructure/identity.tf) important Security-sensitive environments should carefully review which permissions are granted to clusters and consider whether CCM/CSI should be disabled and managed manually. ## Limitations[​](#limitations "Direct link to Limitations") ### Hybrid-cloud and multi-cloud[​](#hybrid-cloud-and-multi-cloud "Direct link to Hybrid-cloud and multi-cloud") When running a vCluster across multiple providers, some additional configuration is required: * **CSI drivers** – Install and configure the appropriate CSI driver for each cloud provider. * **StorageClasses** – Use `allowedTopologies` to restrict provisioning to valid zones/regions. * **NodePools** – Add topology-specific labels (e.g., `topology.ebs.csi.aws.com/zone` on AWS or `topology.gke.io/zone` on GCP) so workloads are scheduled on nodes with matching storage availability. - NodePool - AWS EBS StorageClass - GCP PD StorageClass vCluster config.yaml ``` privateNodes: enabled: true vpn: enabled: true nodeToNode: enabled: true autoNodes: - provider: aws properties: vcluster.com/csi-enabled: "false" dynamic: - name: aws-1 nodeLabels: topology.ebs.csi.aws.com/zone: us-east-1a nodeTypeSelector: - property: instance-type operator: In values: - t3.medium - provider: gcp properties: vcluster.com/ccm-lb-enabled: "false" vcluster.com/csi-enabled: "false" dynamic: - name: gcp-1 nodeLabels: topology.gke.io/zone: us-central1-a nodeTypeSelector: - property: instance-type operator: In values: - e2-medium ``` ``` apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: aws-gp3 provisioner: ebs.csi.aws.com volumeBindingMode: WaitForFirstConsumer parameters: type: gp3 allowedTopologies: - matchLabelExpressions: - key: topology.ebs.csi.aws.com/zone values: - us-east-1a ``` ``` apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gcp-standard provisioner: pd.csi.storage.gke.io volumeBindingMode: WaitForFirstConsumer parameters: type: pd-standard allowedTopologies: - matchLabelExpressions: - key: topology.gke.io/zone values: - us-central1-a ``` ### Region changes[​](#region-changes "Direct link to Region changes") Changing the region of an existing node pool is not supported.
To switch regions, create a new vCluster and migrate your workloads. ### Dynamic nodes `Limit`[​](#dynamic-nodes-limit "Direct link to dynamic-nodes-limit") When editing the limits property of dynamic nodes, any nodes that already exceed the new limit will **not** be removed automatically.
Administrators are responsible for manually scaling down or deleting the excess nodes. --- # Join Manually Provisioned Nodes Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * vCluster CLI installed on your local machine * vCluster control plane has to be running and in `Ready` state * A node that satisfies the [node requirements](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/node-requirements.md) ## Create token[​](#create-token "Direct link to Create token") To join worker nodes, a token from the vCluster must be created to provide access and permissions. A single token can be used for any nodes to join, or if you wanted to, you could create a token for each node. By default, the token expires within 1 hour. The token is stored as a secret prefixed with `bootstrap-token-` in the `kube-system` namespace. The expiry timestamp is stored under the `expiration` key in the secret. * Control-Plane As Container * Control-Plane As Binary Make sure your kube context is set to vCluster Standalone. Create a token for worker nodes ``` # Create a token vcluster token create --expires=1h ``` Create a token for worker nodes ``` export VCLUSTER_NAME=my-vcluster # Connect to your vcluster vcluster connect $VCLUSTER_NAME # Create a token vcluster token create --expires=1h ``` The output provides a command to run on your worker node: Example output from creating a token ``` curl -sfLk https:///node/join?token= | sh - ``` ## Join each worker node[​](#join-each-worker-node "Direct link to Join each worker node") For each worker node that you want to join vCluster, run the command on the node. Example output on worker node ``` Preparing node for Kubernetes installation... Kubernetes version: v1.36.0 Installing Kubernetes binaries... Enabling containerd and kubelet... Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /etc/systemd/system/containerd.service. Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /etc/systemd/system/kubelet.service. Starting containerd and kubelet... Installation successful! Joining node into cluster... [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ``` ## Join with a node profile[​](#join-with-a-node-profile "Direct link to Join with a node profile") If the tenant cluster is connected to vCluster Platform, you can attach a [node profile](https://vcluster.com/docs/platform/administer/node-profiles/overview.md) to a node as part of the join. This saves you from joining the node bare and configuring it by hand afterward. Node profiles require vCluster 0.36 or later. 1. Open the tenant cluster instance in the platform UI, go to the Status tab, and click Join Node.... 2. In the dialog that opens, optionally select a profile from the Node Profile dropdown. It lists the profiles allowed in the project. If you don't select one, the node falls back to the tenant cluster's or project's default profile, when configured. 3. Copy the returned curl command and run it on the target node, as described in [Join each worker node](#join-each-worker-node). Changing the selected profile regenerates the command. The node joins carrying the profile's labels and taints, plus a `vcluster.com/node-profile=` back-reference label. From then on, the platform continuously reconciles the profile onto the node, so later edits to the profile converge automatically without rejoining the node. See [Assignment and precedence](https://vcluster.com/docs/platform/administer/node-profiles/overview.md#assignment-and-precedence) for how the fallback chain resolves. ## Available flags to use in the script to join nodes[​](#available-flags-to-use-in-the-script-to-join-nodes "Direct link to Available flags to use in the script to join nodes") The output of creating the token outputs a script to join the worker node to the vCluster. There are several flags available that can be added to the script. | Flag | Description | Default | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | `--kubernetes-version` | Version of kubernetes node components | optional, defaults to the version of vCluster [k8s](https://kubernetes.io/) | | `--repository-url` | Specific vCluster version to install | optional, defaults to `/usr/local/bin` | | `--binaries-dir` | allows to customize target directory where k8s node components binaries are installed | optional, defaults to `/usr/local/bin` | | `--cni-binaries-dir` | allows to customize target directory where CNI binaries are installed | optional, defaults to `/opt/cni/bin` | | `--skip-reset` | Skips uninstalling existing node components | optional, default to false (reset happens before install by default) | | `--bundle-path` | If set, skips downloading Kubernetes binaries and uses a local path. Expects a path to bundle in `.tar.gz` format | optional, empty by default | | `--reset-only` | only uninstall previously installed node components. Use this to clean up the node. Doesn't stop the `vcluster-vpn` service. See [Clean up orphaned node](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/manage.md#clean-up-orphaned-node) | optional, defaults to `false` | | `--skip-join` | Installs all required components but skips joining node to the cluster. This can be useful if you want to update the kubeadm join config before joining the node to the cluster manually by executing `kubeadm join` | optional, defaults to `false` | | `--node-name` | Node name in the kubernetes cluster | optional, defaults to hostname | | `--force-join` | Force joining even when there is existing kubelet service running on the node. | optional, defaults to `false` | | `--skip-selinux-rpm` | Skips installing the `vcluster-selinux` RPM on RHEL-family hosts with SELinux `Enforcing` or `Permissive`. Use when the RPM is already installed or SELinux is disabled. See [SELinux support](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/selinux.md#install-offline) | optional, defaults to `false` | | `--selinux-rpm-url` | Overrides the default `vcluster-selinux` RPM source. Also settable with the `VCLUSTER_SELINUX_RPM_URL` environment variable. See [SELinux support](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/selinux.md#install-offline) | optional, defaults to `rpm.vcluster.com/stable/el/noarch` | | `--containerd-selinux` | Enables `enable_selinux = true` in containerd's config so tenant pods get per-pod MCS labels. Also settable with the `VCLUSTER_CONTAINERD_SELINUX` environment variable. See [SELinux support](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/selinux.md#containerd-selinux) | optional, defaults to `false` | These flags can be passed to the script by appending `--` and then the flag: Run join script with flags ``` curl -sfLk https:///node/join?token= | sh -s -- --force-join ``` --- # kube-ovn integration Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. vCluster ships an embedded cloud-controller-manager (CCM). Its kube-ovn integration lets a private-nodes tenant cluster route traffic through the tenant's kube-ovn VPC. This works when the tenant's nodes run on a kube-ovn network, for example KubeVirt VM nodes on the control plane cluster. It layers two capabilities onto that VPC: * **Direct pod routing**, so the tenant VPC router reaches pod IPs directly and preserves the client source IP. * **LoadBalancer services**, so `type: LoadBalancer` Services get an externally reachable VIP without an external load balancer, cloud provider, or MetalLB. The direct-to-pod LoadBalancer builds on direct pod routing. The integration only writes into kube-ovn objects the operator has already created, namely the VPC and its subnets. It never creates or manages the VPC or the subnets themselves. See [Prerequisites](#prerequisites) for the kube-ovn topology that must already exist. ## Configuration[​](#configuration "Direct link to Configuration") The entire user-facing surface lives under `controlPlane.advanced.cloudControllerManager` in `vcluster.yaml`. It's an Enterprise feature. vcluster.yaml ``` controlPlane: advanced: cloudControllerManager: # The embedded CCM must be running enabled: true kubeOvn: # Turn on the kube-ovn integration enabled: true # The custom VPC the integration operates on vpc: vmlb-vpc loadBalancers: # Subnet the LoadBalancer VIPs are allocated from externalSubnet: vmlb-external # A real subnet in the VPC (see the table below) backendSubnet: vmlb-vms podRoutes: # Add direct pod routing and route LoadBalancer VIPs straight to pods enabled: false ``` Two switches control the integration: * `kubeOvn.enabled` turns the integration on. This is also what enables LoadBalancer services. * `kubeOvn.podRoutes.enabled` adds direct pod routing and switches LoadBalancer services to route directly to pods. Left at its default `false`, LoadBalancer services route through nodes instead. Each field: | Key | Meaning | | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cloudControllerManager.enabled` | Gates the embedded CCM. Must be `true`, because the kube-ovn integration runs inside it. | | `kubeOvn.enabled` | Enables the kube-ovn integration. Requires `privateNodes.enabled: true` (see [Prerequisites](#prerequisites)) or the config is rejected. | | `kubeOvn.vpc` | Name of the existing kube-ovn `Vpc` to operate on. Shared by both `loadBalancers` and `podRoutes`. | | `kubeOvn.loadBalancers.externalSubnet` | Name of the kube-ovn `Subnet` the VIPs are allocated from. Each LoadBalancer VIP is an `OvnEip` on this subnet. | | `kubeOvn.loadBalancers.backendSubnet` | Name of a real `Subnet` inside `vpc`. It selects which kube-ovn load balancer the VIP is placed on, and it's stamped as the generated Service's `ovn.kubernetes.io/logical_switch`. The value just needs to be a real subnet in the VPC. Only required (and only used) when `podRoutes.enabled` is set; node routing ignores it. | | `kubeOvn.podRoutes.enabled` | Adds direct pod routing and routes LoadBalancer VIPs straight to pods. Defaults to `false`, which routes through nodes. When set, it additionally requires `networking.podCIDR` and `loadBalancers.backendSubnet`, or the config is rejected. See [Direct pod routing](#direct-pod-routing). | Configure from the Platform UI The same fields are available in vCluster Platform. On a tenant cluster in private nodes mode, the networking settings expose the kube-ovn integration and write the same `controlPlane.advanced.cloudControllerManager.kubeOvn` values. The form appears only for tenant clusters running vCluster 0.37 or later. ### RBAC[​](#rbac "Direct link to RBAC") When `kubeOvn.enabled` is set, vCluster deploys the RBAC the integration needs in the control plane cluster. ## Direct pod routing[​](#direct-pod-routing "Direct link to Direct pod routing") Set `podRoutes.enabled: true`. The integration's route controller programs one `Vpc` static route per node podCIDR, read from `Node.Spec.PodCIDRs`, so the tenant VPC router can reach pod IPs directly. This removes the extra kube-proxy hop and preserves the client source IP end to end. These static routes are the foundation that the direct-to-pod LoadBalancer builds on. The routes don't depend on any `type: LoadBalancer` Service, or on `externalSubnet` and `backendSubnet`. Direct pod routing runs inside the kube-ovn integration, reached through `podRoutes.enabled` with `kubeOvn.enabled`, not as a separate component. Enabling `podRoutes.enabled` also requires `networking.podCIDR` and `loadBalancers.backendSubnet` in `vcluster.yaml`. The pod CIDR is the cluster CIDR that kube-controller-manager splits into the per-node `Node.Spec.PodCIDRs` the route controller reads. ## LoadBalancer services[​](#loadbalancer-services "Direct link to LoadBalancer services") When a tenant creates a `type: LoadBalancer` Service, the integration allocates a VIP from a kube-ovn external subnet (an `OvnEip`). It programs the tenant VPC's router to forward that VIP to the Service backends (a `RouterLBRule`), then publishes the allocated VIP back onto the Service as its ingress address. Deleting the Service, or changing it away from `type: LoadBalancer`, tears the kube-ovn resources back down. The `OvnEip` and `RouterLBRule` share a Service-UID-derived name (`vlb-`). How the VIP reaches its backends depends on `podRoutes.enabled`. ### Node routing (default)[​](#node-routing-default "Direct link to Node routing (default)") With `podRoutes.enabled: false`, the CCM service controller drives the LoadBalancer. For each `type: LoadBalancer` Service the integration creates: * an `OvnEip` on `externalSubnet` whose allocated IP becomes the VIP, and * a `RouterLBRule` that DNATs the VIP to the tenant node internal IPs, targeting the Service `NodePort`. Traffic path: 1. The client sends traffic to the VIP. 2. The tenant VPC router DNATs the VIP to a node IP and `NodePort`. 3. kube-proxy on that node performs the second hop to a backend pod. Because kube-proxy does the final hop, the client source IP is lost. The pod sees the node's address. This mode requires each Service port to have a `NodePort` allocated, which is the default for `type: LoadBalancer`. Use it when you don't need the client source IP. ### Direct-to-pod routing[​](#direct-to-pod-routing "Direct link to Direct-to-pod routing") With `podRoutes.enabled: true`, the LoadBalancer builds on [direct pod routing](#direct-pod-routing) and sends traffic straight to pods, preserving the client source IP. It replaces the node-routing service load balancer with two cooperating pieces: * The PodLBReconciler reconciles each Service into an `OvnEip` and a pod-mode `RouterLBRule` with empty endpoints and selector. It stamps the generated Service with `backendSubnet` as its `logical_switch`, so kube-ovn places the VIP on the VPC's load balancer. * The EndpointSliceReplicator copies the tenant Service's EndpointSlices into the control plane cluster namespace to supply the pod-IP backends. Traffic path: 1. The client sends traffic to the VIP. 2. The tenant VPC router DNATs the VIP straight to a pod IP, using the per-node podCIDR routes to find the next hop. 3. The backend pod receives the traffic and sees the real client source IP. Node routing and direct-to-pod routing don't run at the same time. Enabling `podRoutes.enabled` switches the LoadBalancer from one to the other. Use direct-to-pod routing when the workload needs the real client source IP, for example source-IP allowlists, logging, geo-routing, or mTLS by IP. In-place changes don't self-converge Switching an already-created LoadBalancer Service to direct-to-pod routing, by flipping `podRoutes.enabled` while the Service exists, doesn't converge on its own. Node routing leaves a stale v1 Endpoints object, created by kube-ovn's own RouterLBRule controller, that emptying `RouterLBRule.spec.endpoints` doesn't clear. kube-ovn unions it with the pod backends, giving a mixed datapath for a few minutes until the stale Endpoints is removed. Recreate the LoadBalancer Service after changing `podRoutes.enabled` rather than relying on an in-place change. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") This feature sits on top of a specific kube-ovn topology on the control plane cluster. Everything below must already exist and be healthy before you enable the integration. The integration writes into these objects but doesn't create them. ### Configure kube-ovn[​](#configure-kube-ovn "Direct link to Configure kube-ovn") The cluster administrator must have the following in place on the control plane cluster. These are configured directly on kube-ovn and are outside the vCluster config surface. 1. **kube-ovn deployed in secondary (non-primary CNI) mode.** The primary CNI, for example flannel, keeps pod networking, services, and DNS. kube-ovn only serves the tenant VPC overlay subnets and the external segment through Multus secondary interfaces. On the kube-ovn install this requires `NON_PRIMARY_CNI: true`, and `CNI_CONFIG_PRIORITY: "20"` so kube-ovn's conflist sorts after `00-multus` and `10-flannel` rather than becoming the cluster CNI. The `install-cni` step writes a conflist unconditionally with no non-primary branch, so the priority is what keeps it out of the primary datapath. 2. **Custom VPCs enabled.** Set `ENABLE_EXTERNAL_VPC: true` on the kube-ovn install. Without it, the controller silently refuses to reconcile any `Vpc` other than the default, and the tenant VPC objects reconcile to nothing. 3. **The VPC and its subnets exist.** The `Vpc` named in `kubeOvn.vpc` and the subnets named in `loadBalancers.externalSubnet` and `loadBalancers.backendSubnet` must already exist. The KubeVirt node provider creates these through NodeEnvironment properties. See [kube-ovn networking](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md#integrate-with-kube-ovn). 4. **Provider network, uplink, and the external segment.** Configure a `ProviderNetwork` with the node uplink NIC, a `Vlan`, and the external `Subnet` that the VIPs are allocated from. The external subnet's name must match kube-ovn's `--external-gateway-switch` flag, which defaults to `external`. The VPC must also have `enableExternal: true` so it gets an LRP and gateway chassis onto that segment. 5. **Egress out of the custom VPC configured.** Tenant nodes need egress from the VPC to reach external networks, for example to pull images. Enable it through the NodeEnvironment's `kube-ovn.vcluster.com/egress-enabled` property. See [kube-ovn networking](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md#integrate-with-kube-ovn). 6. **A kube-ovn build that has the `RouterLBRule` CRD.** The integration drives the `RouterLBRule` custom resource, which kube-ovn versions up to and including 1.16 don't ship. Use a kube-ovn build that includes it. ## Related[​](#related "Direct link to Related") * [Private nodes overview](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) * [Join manually provisioned nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md) * [Node requirements](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/node-requirements.md) --- # Manage Private Nodes Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. ## Upgrade vCluster[​](#upgrade-vcluster "Direct link to Upgrade vCluster") ### Upgrade vCluster control plane[​](#upgrade-vcluster-control-plane "Direct link to Upgrade vCluster control plane") Kubernetes Version Skew Policy applies Do not upgrade multiple minor versions at the same time for the control plane as outlined in the [Kubernetes Version Skew Policy](https://kubernetes.io/releases/version-skew-policy/). Instead always upgrade a single minor version, wait until the cluster becomes healthy and then upgrade to the next minor version. For example: `v1.26` -> `v1.27` -> `v1.28`. To upgrade the vCluster control plane, update the Kubernetes distro image tag in your `vcluster.yaml` file. After the change, vCluster upgrades the control plane automatically. If automatic worker node upgrades are configured, those nodes are also upgraded. To change the Kubernetes version, add the following to your `vcluster.yaml`: Upgrade control plane Kubernetes version ``` ... controlPlane: distro: k8s: image: tag: v1.35.6 # Or any other Kubernetes version ... ``` ### Upgrade vCluster workers[​](#upgrade-vcluster-workers "Direct link to Upgrade vCluster workers") Kubernetes version skew policy apply Use the [Kubernetes Version Skew Policy](https://kubernetes.io/releases/version-skew-policy/) for worker upgrades. There are two modes of how worker upgrades can be done: * (*Recommended*) Automatically by vCluster when the control-plane was updated * Manually using vCluster CLI or kubeadm #### Use automatic upgrades[​](#use-automatic-upgrades "Direct link to Use automatic upgrades") Automatic worker node upgrades are enabled by default. vCluster upgrades each node when it detects a version mismatch between the control plane and the worker node. By default, one node is upgraded at a time. Upgrade starts 2 minutes after the control plane is upgraded. There is also a grace period for newly added worker nodes (upgrade is not going to start if node was created less than 2 minutes ago). tip You can exclude a node from automatic upgrade by labeling it with `vcluster.loft.sh/skip-auto-upgrade: true` The upgrade Pod runs directly on the node and completes the following steps: 1. Downloads the Kubernetes binary bundle from the vCluster control plane. 2. Replaces the `kubeadm` binary. 3. Runs `kubeadm upgrade node`. 4. Cordons the node. 5. Replaces other binaries such as `containerd`, `kubelet`, and so on. 6. Restarts `containerd` and `kubelet` if necessary. 7. Uncordons the node. warning Node upgrades typically do not restart pods. However, depending on the Kubernetes version change, restarts might occur in some cases. #### Perform manual upgrades[​](#perform-manual-upgrades "Direct link to Perform manual upgrades") You can either upgrade a node using the vCluster CLI by running the following command: Upgrade a specific node manually ``` vcluster node upgrade my-node ``` Alternatively, you can manually upgrade by following the [official Kubeadm Kubernetes guide](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/upgrading-linux-nodes/). ## Node profiles[​](#node-profiles "Direct link to Node profiles") A node that joined with a [node profile](https://vcluster.com/docs/platform/administer/node-profiles/overview.md) stays in sync with that profile for as long as it's part of the tenant cluster. Edits to the profile converge onto every node referencing it, and removed keys are pruned, except `startupTaints`, which apply only at join. See [Continuous reconciliation](https://vcluster.com/docs/platform/administer/node-profiles/overview.md#continuous-reconciliation) for the full behavior. Convergence is prompt rather than tied to a polling interval, so edits are typically reflected within moments in `kubectl get node -o yaml`. ## Reuse a node[​](#reuse-a-node "Direct link to Reuse a node") A node can be re-used for another tenant cluster by using the `--force-join` flag. Before joining the node to a new vCluster, it should be deleted from the original vCluster. Reuse a node ``` # Delete the node from vCluster vcluster node delete my-node # Create a new token from the vCluster you want to join vcluster token create # Then on the node itself run this to join it curl ... | sh -s -- --force-join ``` Rejoining the same cluster If you are rejoining the original vCluster, then the `kube-flannel` and `kube-proxy` pods are restarted. All workloads will be restarted. ## Remove vCluster[​](#remove-vcluster "Direct link to Remove vCluster") When deleting a vCluster with private nodes, it's recommended that each node is removed from the vCluster before deleting the vCluster itself. ### Remove worker node[​](#remove-worker-node "Direct link to Remove worker node") To remove a node from the cluster, follow standard procedures as with any other Kubernetes distro. First cordon a node, then drain it, which can be done using `kubectl` commands. We supply a vCluster CLI command to delete worker nodes from the cluster. Cordon and drain the node ``` vcluster node delete ``` Force Delete a Node If you shut down, delete or otherwise make the node instance unreachable before running `vcluster node delete`, it will prevent draining. To proceed, add `--drain=false` to the delete command. ## Clean up orphaned node[​](#clean-up-orphaned-node "Direct link to Clean up orphaned node") There may be cases where the cluster is deleted before all worker nodes were removed. In these scenarios, the `kubelet`, `kube-proxy` and other components running on the node are still trying to connect to the now deleted cluster control plane. The node can be cleaned up by using any join script with the `--reset-only` flag. Cleaning up control-plane node can be done by running `/var/lib/vcluster/node/join.sh --reset-only`. If the worker node joined through vCluster VPN, `--reset-only` leaves the `vcluster-vpn` service running with the deleted cluster's VPN state, and the next join to a new cluster hangs. Stop the service and remove its config as root: ``` systemctl disable --now vcluster-vpn rm -f /etc/vcluster-vpn/config.json ``` ## Load Docker images to a node[​](#load-docker-images-to-a-node "Direct link to Load Docker images to a node") There may be instances where you want to load an image directly to the private node. Before loading any images to the node, the node has to be in a `Ready` state and part of a tenant cluster. The command pulls the image locally onto the machine that is running the command and saves it as a `.tar` archive. It creates a pod on the private node and copies the image using `kubectl cp` to the pod. The pod imports the image onto the node. Load Docker image into specific node ``` vcluster node load-image my-node --image nginx:latest ``` --- # Node Requirements Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** ## Private node requirements[​](#private-node-requirements "Direct link to Private node requirements") * vCluster CLI installed on your local machine * Access to a control plane cluster * Worker nodes * kubeadm is used in the join script, so meeting the [kubeadm requirements](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin) is recommended ## Host requirements[​](#host-requirements "Direct link to Host requirements") * Linux distribution with systemd support (required for service management) * `iptables` binaries installed (automatically available on Ubuntu, may need manual installation on other distributions) * `curl` installed * Root user access info On RHEL 8 and 9, SELinux in enforcing mode is supported with the `vcluster-selinux` RPM. The Private Node join script installs it automatically; no extra steps are required. See [SELinux support](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/selinux.md). ## Supported operating systems[​](#supported-operating-systems "Direct link to Supported operating systems") | OS | Version | Additional Instructions | | ------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Ubuntu | 24.04 | | | Ubuntu | 22.04 | | | RHEL (Red Hat Enterprise Linux) | 9 | [Install `vcluster-selinux`](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/selinux.md) or disable SELinux | | RHEL (Red Hat Enterprise Linux) | 8 | [Install `vcluster-selinux`](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/selinux.md) and pin Kubernetes to 1.31, or disable SELinux | | RHEL (Red Hat Enterprise Linux) | 10 | Disable SELinux | | CentOS Stream | 9 | Disable SELinux and install `iptables` | | Rocky | 10 | Disable SELinux and install `iptables` | | Rocky | 9 | Disable SELinux and install `iptables` | ### CentOS Stream and Rocky support[​](#centos-stream-and-rocky-support "Direct link to CentOS Stream and Rocky support") Install iptables ``` yum install iptables-services systemctl enable iptables systemctl start iptables ``` --- Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. # Deploy Private Nodes in an air-gapped environment This document explains how to deploy Private Nodes in environments without internet access, known as air-gapped environments. important Enterprise features require the vCluster Platform. If you deploy a vCluster in an air-gapped environment and want to use enterprise features, you must also deploy the platform in air-gapped mode and connect the vCluster to it. ## Overview[​](#overview "Direct link to Overview") When deploying Private Nodes, there are artifacts that are typically accessed using an internet connection, but without access to the internet, these artifacts need to be available to the Kubernetes cluster through a private registry: * **Kubernetes Control-Plane image**—Pulled from `loft-sh/kubernetes` registry. After ensuring that Kubernetes Control-Plane image is present on the server which you want to use as Private Node, and uploading the required images to your private registry, create the `vcluster.yaml` configuration file and follow the installation steps. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Access to a node that satisfies the [node requirements](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/node-requirements.md) ### Registry requirements[​](#registry-requirements "Direct link to Registry requirements") * OCI-compliant private registry - A private registry accessible to the Private Nodes, vCluster Control-Plane and a separate, internet-connected machine. * Ability to push images to your private registry. ### Deployment requirements[​](#deployment-requirements "Direct link to Deployment requirements") Ensure you have the following: * Access to pull the images from the private registry * On the internet-connected machine for populating the registry: * `wget` installed * `docker` installed * You are logged into GitHub Container Registry ## Populate images to a private registry[​](#populate-images-private-registry "Direct link to Populate images to a private registry") Each vCluster release includes multiple assets to help you upload the images to your private registry. * `images-private-nodes.txt` - The required images to run Private Node, which assumes using the default Kubernetes version. * `images-private-nodes-optional.txt` - An optional set of images needed for worker node daemonsets and agents. * `download-images.sh` - A bash script that quickly iterates over all the images files to pull them and package them into a tarball to a machine that has internet access. * `push-images.sh` - A bash script that takes the tarball generated from the download script to push them to your private registry. ### Pull and push images[​](#pull-push-images "Direct link to Pull and push images") Modify the following with your specific values to replace on the whole page and generate copyable commands: VCLUSTER\_VERSION0.29.1 REGISTRYecr.io/myteam 1. Download the assets from the [vCluster GitHub release](https://github.com/loft-sh/vcluster/releases) and make the scripts executable. note The `images-private-nodes.txt` contains all required images for the default Kubernetes version. ``` wget https://github.com/loft-sh/vcluster/releases/download/v0.29.1/images-private-nodes.txt wget https://github.com/loft-sh/vcluster/releases/download/v0.29.1/download-images.sh wget https://github.com/loft-sh/vcluster/releases/download/v0.29.1/push-images.sh chmod +x ./download-images.sh chmod +x ./push-images.sh ``` 2. Run `download-images.sh` to pull all images and create a tarball of the images. Review the output to confirm all images were pulled successfully and packaged in the tarball. Download and package images ``` ./download-images.sh --image-list images-private-nodes.txt ``` Multi-platform support If you need to support multiple CPU architectures (e.g., AMD64 and ARM64), use the `--platforms` flag to specify which platforms to download. This is useful when your cluster has nodes with different architectures. Download images for multiple platforms ``` ./download-images.sh --platforms "linux/amd64,linux/arm64" --image-list images-private-nodes.txt ``` 3. Run `push-images.sh` to upload all required images to your private registry. When pushing images into your private registry, the public private registry is removed and only the repository and image name are pushed. This allows vCluster to set your private registry to use for all images used in deploying vCluster. Push images to private registry ``` ./push-images.sh --registry ecr.io/myteam --image-list images-private-nodes.txt ``` Additional push options **Multi-platform support**: If you downloaded images for multiple platforms, specify the same `--platforms` flag when pushing: ``` ./push-images.sh --registry ecr.io/myteam --image-list images-private-nodes.txt --platforms "linux/amd64,linux/arm64" ``` **Insecure registry**: If your private registry uses HTTP instead of HTTPS, add the `--insecure` flag: ``` ./push-images.sh --registry ecr.io/myteam --image-list images-private-nodes.txt --insecure ``` 4. **Optional**: By default, Private Node uses Flannel for CNI and Local Path Provisioner. To download and push these, download the `images-private-nodes-optional.txt` file. The `images-private-nodes-optional.txt` includes images for various optional configurations. You can edit the file and keep only the images you need, removing those for unused features or versions. note The `images-private-nodes-optional.txt` may include multiple Kubernetes distributions, versions, or other feature-dependent images. You can edit the file to retain only what's necessary for your deployment. Pull, package, and push all optional images ``` wget https://github.com/loft-sh/vcluster/releases/download/v0.29.1/images-private-nodes-optional.txt ./download-images.sh --image-list images-private-nodes-optional.txt --images vcluster-images-optional.tar.gz ./push-images.sh --registry ecr.io/myteam --image-list images-private-nodes-optional.txt ``` tip You can also use the `--platforms` and `--insecure` flags with these commands if needed, as described in the previous steps. ## Configure vCluster[​](#configure-vCluster "Direct link to Configure vCluster") The vcluster.yaml file contains all configuration settings for your vCluster deployment. Use a private registry without credentials Set the default private registry that doesn't have authentication. Authenticated private registries are not supported in this version. Setting the private registry ``` controlPlane: advanced: defaultImageRegistry: ecr.io/myteam ``` ### Redirect workload image pulls on joined nodes[​](#workload-image-mirrors "Direct link to Redirect workload image pulls on joined nodes") The `defaultImageRegistry` setting above only affects the images vCluster deploys for its own control plane components. Workloads scheduled onto joined Private Nodes pull their images through each node's containerd runtime, which reaches out to public registries such as `docker.io` and `ghcr.io` by default. In an air-gapped environment, those pulls fail. To redirect them to your private registry, configure containerd registry mirrors through `privateNodes.joinNode.containerd.registry`. As each node joins, vCluster renders these settings into a containerd `hosts.toml` file on the node, following the containerd [registry host model](https://github.com/containerd/containerd/blob/main/docs/hosts.md). Joined nodes only This mechanism applies to real Private Nodes joined through kubeadm, such as bare metal servers or cloud VMs. It does not apply to vind, whose Docker driver ignores this setting and serves images through a host-cache registry proxy instead. For vind, see [Deploy vind with restricted network access](https://vcluster.com/docs/vcluster/deploy/control-plane/docker-container/air-gapped.md). Add the following to your `vcluster.yaml`. Each key under `mirrors` is the registry you want to redirect, and each `hosts[].server` entry is the mirror endpoint that containerd contacts in its place. If you omit `capabilities`, containerd defaults to `pull` and `resolve`. Modify the following with your specific values to generate a copyable command: MIRROR\_ENDPOINThttps\://registry.internal Configure containerd registry mirrors ``` privateNodes: joinNode: containerd: registry: mirrors: # Redirect Docker Hub pulls to your internal registry "docker.io": hosts: - server: https://registry.internal capabilities: ["pull", "resolve"] # Redirect GitHub Container Registry pulls to your internal registry "ghcr.io": hosts: - server: https://registry.internal capabilities: ["pull", "resolve"] ``` If your private registry requires authentication, add an `auth` section keyed by the registry host. vCluster writes these credentials into the node's containerd configuration. Modify the following with your specific values to generate a copyable command: REGISTRY\_HOSTregistry.internal USERNAMEmy-user PASSWORDmy-password Authenticate to the mirror registry ``` privateNodes: joinNode: containerd: registry: auth: # Credentials for your internal registry "registry.internal": username: my-user password: my-password ``` After your private registry is populated with the required images, you can proceed with adding Private Nodes. ### Installation[​](#installation "Direct link to Installation") For further steps follow [the primary guide](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md) --- # Confidential compute Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** This page explains where vCluster fits in a confidential compute platform and shows the configuration patterns to connect it to a confidential runtime. vCluster has no confidential compute feature of its own. It provides tenant isolation and workload routing around the confidential runtime you deploy on the worker nodes. If your confidential compute stack runs on standard Kubernetes, it runs through vCluster. For provider-specific instance settings, node image contents, attestation verification, and support boundaries, follow your provider and runtime documentation. [Confidential compute](https://confidentialcomputing.io/) protects workload code and data in use by running it inside a hardware-backed trusted execution environment (TEE). Common examples are Intel TDX and AMD SEV-SNP. vCluster gives each tenant a dedicated Kubernetes control plane and routes workloads to confidential-capable nodes using standard Kubernetes mechanisms such as RuntimeClass, node selectors, and node labels. The TEE protects workload data on the worker nodes. Tenants get that hardware-level protection combined with Kubernetes-level isolation from other tenants through vCluster. Where the TEE sits in the stack determines how you configure the node image and what tenants need to do to use it: * **Cloud-provider Confidential VMs** (GCP Confidential VMs, Azure Confidential Computing, AWS EC2 instances with AMD SEV-SNP): the hypervisor applies hardware-backed protection to the entire VM. All workloads on the node run inside the TEE. Attestation usually runs at node bootstrap time, before the node joins the cluster. No RuntimeClass is required. * **Pod-level runtime** (Kata Containers, Confidential Containers): each pod runs in a lightweight VM backed by a hardware TEE. Requires the runtime on the node image, a RuntimeClass object available to the tenant cluster, and a `runtimeClassName` in the pod spec. This approach works on any compatible bare metal or cloud VM. The sections below apply to both paths unless noted. ## Reference architecture[​](#reference-architecture "Direct link to Reference architecture") Use vCluster as the tenant Kubernetes layer. Add vMetal if the confidential-capable capacity comes from physical servers you manage, and add vNode if tenant workloads need stronger runtime isolation. Then install and operate the confidential compute runtime on the private nodes selected for confidential workloads. Confidential compute architecture with vCluster, private nodes, optional vMetal and vNode, and separate Confidential VM and pod-level runtime paths In this model, vCluster decides which tenant gets access to which Kubernetes environment. Auto nodes selects the right capacity, vMetal prepares and joins the machine, vNode strengthens the workload boundary, and the confidential runtime executes the workload in the hardware-backed environment. For the strongest isolation guarantee, assign private nodes to a single tenant cluster rather than sharing nodes across tenants. The components in this architecture each cover a distinct layer: * **vCluster** provides each tenant with an isolated Kubernetes control plane and a scheduling boundary for workloads. * **Private nodes** give the tenant dedicated worker nodes for workloads that require dedicated compute. * **Auto nodes** connect workload demand to node provisioning through vCluster Platform. * **vMetal** (optional) provisions and recycles physical servers that have the required CPU, firmware, OS image, drivers, and runtime configuration. * **vNode** (optional) adds a runtime isolation layer for privileged or untrusted workloads inside the tenant environment. * **A confidential runtime** — either a cloud-provider Confidential VM or a pod-level runtime such as Kata Containers and Confidential Containers — provides the hardware-backed TEE and attestation. ## Implementation checklist[​](#implementation-checklist "Direct link to Implementation checklist") 1. Select hardware or cloud instance types that support the target confidential compute technology. 2. Configure provider-specific VM options, firmware, Shielded VM features (secure boot, vTPM, integrity monitoring), and host OS images. 3. Prepare the node image for your runtime path: * **Cloud-provider VMs**: enable Confidential VM features and configure bootstrap attestation. * **Pod-level runtime**: install Kata Containers, Confidential Containers, runtime classes, and attestation services. 4. Register the hardware in vMetal or another node provider. 5. Add node type properties that describe the confidential compute capability. 6. Configure `privateNodes.autoNodes` to select those node types. 7. Add vNode where tenant workloads need stronger runtime isolation. 8. Validate scheduling, attestation, and workload execution from inside the tenant cluster. ## Build the capacity layer[​](#build-the-capacity-layer "Direct link to Build the capacity layer") Start by creating node types that represent confidential-capable hardware. Node type properties become selector requirements, so you can expose hardware features such as `tdx`, `sev-snp`, GPU model, firmware track, or image family to vCluster auto nodes. NodeProvider — platform admin ``` apiVersion: management.loft.sh/v1 kind: NodeProvider metadata: name: bare-metal spec: # Use the Metal3 node provider when capacity comes from physical servers. metal3: clusterRef: cluster: datacenter namespace: metal3 nodeTypes: - name: tdx-large resources: cpu: "64" memory: 512Gi bareMetalHosts: # Select only BareMetalHost resources prepared for TDX workloads. selector: matchLabels: confidential-compute: tdx # Expose confidential compute traits as node type properties. properties: confidential-compute: tdx ``` This node provider creates a `tdx-large` node type. Auto nodes can select this node type when a tenant cluster requests TDX-capable capacity. If the confidential-capable capacity comes from physical servers, [vMetal](https://www.vmetal.ai/docs/) can register servers, provision the OS image, and join each server to the tenant cluster as a private node. Any node provider that can attach a prepared worker to the cluster works. vMetal is one option, not a requirement. Prepare the OS image and bootstrap flow for the confidential runtime before offering the node type to tenants. This typically includes firmware settings, Shielded VM features (secure boot, vTPM, and integrity monitoring), kernel support, and the confidential runtime configuration for your chosen path. Configure host maintenance behavior according to the provider and confidential compute technology. Some Confidential VM machine types support live migration, while others require stop-on-maintenance behavior such as `on_host_maintenance: TERMINATE` or the provider-specific equivalent. Confirm the supported maintenance policy for the selected instance type before publishing the node type. Cloud-provider confidential node types also require an explicit zone, not just a region. Cloud providers enforce zone placement for Confidential VM instances. Attestation **Cloud-provider VMs**: attestation usually runs as a bootstrap gate before the node joins the cluster. The gate should request and verify provider or hardware attestation evidence against your policy before allowing the kubeadm join. Checking hardware markers such as `/dev/sev-guest` for SEV-SNP or `/dev/tdx_guest` for TDX is only a local sanity check. It does not replace quote or report verification. Depending on the provider and trust model, verification can be local, provider-managed, or performed by an external verifier. **Pod-level runtime**: each pod generates an attestation quote at workload start time. The node image must include device plugins for quote generation, such as the Intel TDX attestation driver or AMD SEV-SNP device plugin, and the Data Center Attestation Primitives libraries your attestation service expects. The attestation service endpoint must be reachable from the private nodes. If your platform enforces network policies, add an egress rule that permits attestation traffic before publishing the node type to tenants. ## Route workloads to confidential nodes[​](#route-workloads-to-confidential-nodes "Direct link to Route workloads to confidential nodes") A platform admin configures the tenant cluster to use confidential nodes by requiring the confidential-capable node type property. vCluster Platform provisions matching nodes when the tenant cluster needs capacity. vcluster.yaml — platform admin ``` privateNodes: # Enable dedicated worker nodes for this tenant cluster. enabled: true autoNodes: - provider: bare-metal dynamic: - name: confidential-workers # Require a node type that advertises TDX support. nodeTypeSelector: - property: confidential-compute operator: In values: - tdx ``` This configuration lets vCluster Platform provision matching private nodes when the tenant cluster needs confidential-capable worker capacity. Private nodes join the tenant cluster as real nodes, so their OS-level labels are visible inside the cluster and the `nodeSelector` works without any additional configuration. For the pod-level runtime path, tenants add a `runtimeClassName` referencing the RuntimeClass installed in the control plane cluster and exposed to the tenant cluster. RuntimeClass syncing is disabled by default, so the platform admin must sync the allowed RuntimeClass objects into the tenant cluster before tenants can reference them. The next two examples apply only to the pod-level runtime path. If you use cloud-provider Confidential VMs, the private node selection above is the vCluster-specific configuration; finish the VM image, maintenance policy, and attestation setup in your provider tooling. vcluster.yaml — platform admin (pod-level runtime path) ``` sync: fromHost: runtimeClasses: enabled: true selector: matchLabels: confidential-compute: tdx privateNodes: enabled: true autoNodes: - provider: bare-metal dynamic: - name: confidential-workers nodeTypeSelector: - property: confidential-compute operator: In values: - tdx ``` Label the RuntimeClass in the control plane cluster so only approved runtime classes are synced: RuntimeClass — control plane cluster ``` apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: kata-qemu-tdx labels: confidential-compute: tdx handler: kata-qemu-tdx ``` For the full selector and patches reference, see [Runtime classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/runtime-classes.md). For cloud-provider Confidential VMs, omit `runtimeClassName`. Protection applies at the VM level and standard containers run inside the TEE without modification. Pod spec — tenant (pod-level runtime path) ``` apiVersion: v1 kind: Pod metadata: name: confidential-workload spec: # Use the confidential runtime configured on the private node image. runtimeClassName: kata-qemu-tdx # Schedule only onto nodes that advertise TDX support. nodeSelector: confidential-compute: tdx containers: - name: app image: ghcr.io/example/confidential-app:1.0.0 ``` This pod runs with the confidential RuntimeClass and schedules onto nodes that match the confidential compute property. ## Add vNode for tenant isolation[​](#add-vnode-for-tenant-isolation "Direct link to Add vNode for tenant isolation") vNode adds a runtime isolation layer for tenant workloads. It's useful when tenants need privileged workloads or host-like APIs without giving those workloads direct access to the underlying host. For confidential compute builds, use vNode as an isolation layer around tenant workloads and pair it with the confidential RuntimeClass exposed on prepared private nodes. This is a useful pattern for platform teams that need both tenant isolation and hardware-backed workload execution. Review the [vNode architecture](https://www.vnode.com/docs/architecture/) and [vNode limitations](https://www.vnode.com/docs/limitations/) while designing the runtime stack. vNode operates at the OS layer using Linux user namespaces, while Kata Containers operates at the VM layer. Their interaction depends on kernel version and node configuration. Validate vNode, Kata Containers, Confidential Containers, the node image, and the target hardware together before publishing the node type to tenant teams. ## Learn more[​](#learn-more "Direct link to Learn more") For cloud-provider Confidential VMs, start with your provider's Confidential VM and attestation documentation to understand instance requirements, firmware options, and the bootstrap attestation flow. For the pod-level runtime path, use these resources to plan the confidential runtime and attestation parts of the platform. | Resource | Use it for | | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | [Kata Containers](https://katacontainers.io/) | Pod-level virtualized container runtime | | [Kata Containers documentation](https://github.com/kata-containers/kata-containers/tree/main/docs) | Runtime architecture and configuration | | [Confidential Containers](https://confidentialcontainers.org/) | Kubernetes workloads that use hardware-backed trusted execution environments | | [Confidential Containers documentation](https://github.com/confidential-containers/documentation) | Operator, runtime, and attestation guidance | --- # FIPS 140-2 enablement Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. The [National Institute of Standards and Technology (NIST)](https://www.nist.gov/federal-information-processing-standards-fips) develops Federal Information Processing Standards (FIPS) to ensure the security and interoperability of computer systems used by the U.S. government. FIPS 140-2 is a U.S. Federal Government security standard used to approve cryptographic modules. This document explains how vCluster Standalone and all its components are built with FIPS-validated cryptographic libraries. ## Use of FIPS compatible Go toolchain[​](#use-of-fips-compatible-go-toolchain "Direct link to Use of FIPS compatible Go toolchain") vCluster is written in [Go](https://go.dev/), and the FIPS-compliant builds are compiled using the `GOFIPS140=v1.0.0` environment variable. The FIPS 140-3 validated Go Cryptographic Module now underlies Go’s built-in crypto libraries, starting with the Go Cryptographic Module v1.0.0 that is included in Go 1.24. ### FIPS support in vCluster components[​](#fips-support-in-vcluster-components "Direct link to FIPS support in vCluster components") Most components used in vCluster are statically compiled with the GOFIPS140=v1.0.0 enabled Go compiler. From a component perspective, vCluster includes multiple sub-components it depends on. Kubernetes components sourced from [Kubernetes image with -full-fips tag suffix](https://github.com/loft-sh/kubernetes/pkgs/container/kubernetes): * containerd and containerd-shim * cni network plugins * kubeadm * kubectl * kubelet * kube-apiserver * kube-controller-manager * kube-scheduler * runc
Other components: * [Kine](https://github.com/loft-sh/fips-builds/pkgs/container/kine-fips) * [ETCD](https://github.com/loft-sh/fips-builds/pkgs/container/etcd-fips) * [Konnectivity Server](https://github.com/loft-sh/fips-builds/pkgs/container/konnectivity-server-fips)
## Run a FIPS-compliant vCluster with private nodes enabled[​](#run-a-fips-compliant-vcluster-with-private-nodes-enabled "Direct link to Run a FIPS-compliant vCluster with private nodes enabled") To run vCluster in a FIPS environment, you must reconfigure the repositories used to reference the FIPS images. The following is an example of a `vcluster.yaml` file that one can use to create a FIPS-compliant vCluster instance: ``` controlPlane: statefulSet: image: repository: loft-sh/vcluster-pro-fips distro: k8s: image: registry: ghcr.io repository: loft-sh/kubernetes tag: v1.33.5-fips # specify Kubernetes version here with -fips suffix backingStore: etcd: embedded: # The use of embedded etcd is recommended, yet optional enabled: true privateNodes: enabled: true ``` To create the FIPS-compliant vCluster instance, run: ``` vcluster create my-fips-vcluster -f vcluster.yaml ``` ## Connect private node[​](#connect-private-node "Direct link to Connect private node") Connecting private nodes for FIPS-compliant vCluster is exactly the same process, as for non FIPS-compliant. vCluster will automatically recognize it is running in FIPS-compliant mode and use the FIPS-compliant bundle and images for Kubernetes components. Follow [these steps to connect a private node](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md). --- # SELinux support Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. `vcluster-selinux` is the SELinux policy module that vCluster Labs publishes for RHEL hosts running vCluster Standalone or a Private Node worker. It ships as a signed `.noarch` RPM for EL 8, EL 9, and EL 10 from the [vCluster SELinux repository on GitHub](https://github.com/loft-sh/vcluster-selinux). With the module loaded, vCluster runs on a host with SELinux in enforcing mode without disabling it or adding host-local allow rules. No extra steps required The vCluster installer and the Private Node join script detect SELinux on supported RHEL hosts and install the RPM before placing any vCluster binaries. Run the [standalone install](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) or the [Private Node join script](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md) the same way you would on any other host — the installer fetches, verifies, and loads the SELinux module automatically. ## Supported operating systems[​](#supported-operating-systems "Direct link to Supported operating systems") The SELinux module is required on hosts where the vCluster binary runs directly: **vCluster Standalone** control planes and **Private Node** workers. vCluster Platform, the Shared Nodes tenancy model, and tenant workloads running inside the cluster are unaffected. ### Product-supported[​](#supported-rhel "Direct link to Product-supported") | OS | SELinux mode | RPM | Notes | | ------- | ------------------------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | RHEL 10 | `Enforcing`, `Permissive`, `Disabled` | `el10` | Supported. Installer fetches and installs `el10` automatically when SELinux mode is `Enforcing` or `Permissive`. | | RHEL 9 | `Enforcing`, `Permissive`, `Disabled` | `el9` | Supported. Installer fetches and installs `el9` automatically when SELinux mode is `Enforcing` or `Permissive`. | | RHEL 8 | `Enforcing`, `Permissive`, `Disabled` | `el8` | Supported. Requires a [Kubernetes 1.31 pin](#rhel-8-k8s-pin) for the control plane to start regardless of SELinux mode. Installer fetches and installs `el8` automatically when SELinux mode is `Enforcing` or `Permissive`. | ### Tested derivatives[​](#supported-derivatives "Direct link to Tested derivatives") The `.noarch` RPM targets the EL family as a whole. CI validates the same RPM on AlmaLinux and Rocky Linux because RHEL subscriptions are not available to the public CI runners. These distributions are covered by the same install flow as the corresponding RHEL major version, but are not product-supported. | OS | RPM | CI coverage | | --------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | AlmaLinux 10 | `el10` | Same RPM as RHEL 10; community-tested. | | AlmaLinux 9 | `el9` | Yes — full standalone + Private Node e2e under enforcing. | | AlmaLinux 8 | `el8` | Same RPM as RHEL 8; community-tested. | | CentOS Stream 9 | `el9` | Same RPM as RHEL 9; community-tested. Requires `iptables` (see [node requirements](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/node-requirements.md)). | | Rocky Linux 10 | `el10` | Same RPM as RHEL 10; community-tested. | | Rocky Linux 9 | `el9` | Same RPM as RHEL 9; community-tested. Requires `iptables` (see [node requirements](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/node-requirements.md)). | | Rocky Linux 8 | `el8` | Yes — full standalone + Private Node e2e under enforcing. Requires `iptables` and the [Kubernetes 1.31 pin](#rhel-8-k8s-pin). | ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * vCluster `0.34` or newer. The installer and the Private Node join script install the RPM and run `restorecon` after binary placement starting with this version. * A RHEL 8, RHEL 9, or RHEL 10 host with `getenforce` reporting `Enforcing` or `Permissive`. RHEL 8 additionally requires a Kubernetes 1.31 pin in `vcluster.yaml`. See [Pin Kubernetes to 1.31 on RHEL 8](#rhel-8-k8s-pin). * Network access to `https://rpm.vcluster.com` from the host, or a pre-staged RPM. See [Install offline or with a custom RPM mirror](#install-offline). * `dnf` installed. The RPM declares `container-selinux`, `policycoreutils`, `policycoreutils-python-utils`, `libselinux-utils`, and `selinux-policy-base` as dependencies. ## Install[​](#install "Direct link to Install") On a RHEL 8, 9, or 10 host that can reach `rpm.vcluster.com`, no separate SELinux step is required. Use the [standalone install](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) or [Private Node join](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md) flow as documented. RHEL 8 also requires the [Kubernetes 1.31 pin](#rhel-8-k8s-pin). When `getenforce` returns `Enforcing` or `Permissive` and the RPM is not already installed, the installer: 1. Reads `${VERSION_ID%%.*}` from `/etc/os-release`. 2. Writes a yum repo file for `https://rpm.vcluster.com/stable/el${EL_VERSION}/noarch`. 3. Verifies the package against the GPG key at `https://rpm.vcluster.com/public.key`. 4. Runs `dnf install -y vcluster-selinux` before placing any vCluster binaries on the host. For convenience, the same commands as the install pages: Install vCluster Standalone (auto-installs vcluster-selinux on RHEL) ``` curl -fsSL https://github.com/loft-sh/vcluster/releases/download/v0.37.0/install-standalone.sh | sudo bash -s -- --config /etc/vcluster/vcluster.yaml ``` Join a Private Node worker (auto-installs vcluster-selinux on RHEL) ``` curl -sfLk "$JOIN_URL" | sudo bash ``` `$JOIN_URL` is the URL `vcluster token create` returns for the tenant cluster. See [Join Manually Provisioned Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md) for how to mint a join token and the full join flow. If SELinux is `Enforcing` and the RPM install fails, the installer exits non-zero before placing any vCluster binaries on the host. If SELinux is `Permissive`, the installer prints a warning and continues. The host runs vCluster without SELinux enforcement of the `vcluster-selinux` rules. ### Pin Kubernetes to 1.31 on RHEL 8[​](#rhel-8-k8s-pin "Direct link to Pin Kubernetes to 1.31 on RHEL 8") RHEL 8 ships glibc 2.28. The containerd binary in the default vCluster Kubernetes bundle (**v1.36.0**) links against glibc 2.32 or newer and will not load on an EL 8 host. The kubelet and any tenant pods stay stuck and the node never reaches `Ready`. The Kubernetes 1.31.x bundles are built against an older glibc and run on EL 8. Pin the Kubernetes version in `vcluster.yaml` before running the installer: /etc/vcluster/vcluster.yaml on a RHEL 8 host ``` controlPlane: standalone: enabled: true joinNode: enabled: true containerd: enabled: true distro: k8s: version: v1.31.11 ``` With this configuration, the standalone install on RHEL 8 is otherwise identical to RHEL 9. The `el8` RPM comes from `rpm.vcluster.com/stable/el8/noarch`, the SELinux policy loads, and `systemd` transitions `vcluster.service` into `container_runtime_t`. warning The pin is host-wide. Every tenant Kubernetes version the host serves is 1.31.x. To run a newer Kubernetes on the control plane, run the host on RHEL 9 instead. ### Enable SELinux enforcement for tenant pods[​](#containerd-selinux "Direct link to Enable SELinux enforcement for tenant pods") Containerd applies per-pod MCS labels only when its configuration has `enable_selinux = true`. The installer does not enable this flag by default because it changes runtime behavior for workloads that may already be running. Pass `--containerd-selinux` to enable it: Standalone with containerd SELinux enforcement ``` curl -fsSL https://github.com/loft-sh/vcluster/releases/download/v0.37.0/install-standalone.sh | sudo bash -s -- --config /etc/vcluster/vcluster.yaml --containerd-selinux ``` Private Node worker with containerd SELinux enforcement ``` curl -sfLk "$JOIN_URL" | sudo bash -s -- --containerd-selinux ``` With `enable_selinux = true` in `/etc/containerd/config.toml`, each tenant pod on the worker runs under `container_t` with its own MCS category. Combined with the module's `container_t` → `vcluster_data_t` deny rules, a compromised tenant pod cannot read host PKI or the backing-store database through the filesystem. ### Air-gapped install[​](#install-offline "Direct link to Air-gapped install") Hosts without egress to `rpm.vcluster.com` need either a reachable mirror or a pre-staged RPM. Pair this with the rest of the air-gapped guidance in [Deploy Private Nodes in an air-gapped environment](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/air-gapped.md). #### Point at a custom RPM URL[​](#point-at-a-custom-rpm-url "Direct link to Point at a custom RPM URL") Pass `--selinux-rpm-url` or set the `VCLUSTER_SELINUX_RPM_URL` environment variable to a URL the host can reach: Standalone with a custom RPM URL ``` curl -fsSL https://github.com/loft-sh/vcluster/releases/download/v0.37.0/install-standalone.sh | sudo bash -s -- --config /etc/vcluster/vcluster.yaml --selinux-rpm-url https://internal.example.com/vcluster-selinux--.el9.noarch.rpm ``` `--selinux-rpm-url` accepts either a direct `.rpm` URL or a yum-repo URL. `dnf install` accepts both. #### Pre-stage the RPM at image-build time[​](#pre-stage-the-rpm-at-image-build-time "Direct link to Pre-stage the RPM at image-build time") Bake `vcluster-selinux` into the host image and tell the installer to skip its own fetch with `--skip-selinux-rpm`: Install the RPM once, during image build ``` sudo dnf install -y vcluster-selinux ``` Run the installer later without re-fetching the RPM ``` curl -fsSL https://github.com/loft-sh/vcluster/releases/download/v0.37.0/install-standalone.sh | sudo bash -s -- --config /etc/vcluster/vcluster.yaml --skip-selinux-rpm ``` warning Pass `--skip-selinux-rpm` only when `vcluster-selinux` is already installed on the host or SELinux is disabled. With SELinux `Enforcing` and no module loaded, `vcluster.service` fails to transition into `container_runtime_t` and the control plane does not start. ### Install manually[​](#install-manually "Direct link to Install manually") To pin a specific release, download the `.noarch.rpm` for your RHEL major version from the [vcluster-selinux releases page](https://github.com/loft-sh/vcluster-selinux/releases) and install it directly: ``` sudo dnf install https://github.com/loft-sh/vcluster-selinux/releases/download//vcluster-selinux--.el9.noarch.rpm ``` Or install from the same yum repo the vCluster installer would have configured: /etc/yum.repos.d/vcluster-selinux.repo ``` sudo tee /etc/yum.repos.d/vcluster-selinux.repo <<'EOF' [vcluster-selinux-stable] name=vCluster SELinux (stable) baseurl=https://rpm.vcluster.com/stable/el9/noarch enabled=1 gpgcheck=1 repo_gpgcheck=0 gpgkey=https://rpm.vcluster.com/public.key EOF sudo dnf install -y vcluster-selinux ``` When installing the RPM manually, run `install-standalone.sh` or the join script with `--skip-selinux-rpm` so the installer does not re-fetch the RPM. ## SELinux labels[​](#what-the-module-labels "Direct link to SELinux labels") | Path | SELinux type | | -------------------------------------------------------------------------------- | -------------------------- | | `/var/lib/vcluster/bin/vcluster` (the entrypoint `install-standalone.sh` places) | `container_runtime_exec_t` | | `/var/lib/vcluster(/.*)?` (PKI, kine/etcd backing store, sockets, pid files) | `vcluster_data_t` | | `/etc/vcluster(/.*)?`, `/etc/vcluster-vpn(/.*)?`, `/etc/crictl.yaml` | `container_config_t` | | `/opt/cni(/.*)?`, `/etc/cni(/.*)?` | `container_file_t` | | `/usr/local/bin/vcluster-vpn` | `container_runtime_exec_t` | | `/etc/systemd/system/vcluster*` | `container_unit_file_t` | The module also registers a `semanage fcontext` override for `/var/run/flannel(/.*)?` → `container_file_t` so the flannel pod can write its runtime state. It pre-creates `/var/lib/vcluster`, `/etc/vcluster`, `/etc/vcluster-vpn`, `/opt/cni`, `/etc/cni`, `/opt/local-path-provisioner`, `/run/flannel`, and `/run/kubernetes` so the labels are correct regardless of whether the RPM or the installer runs first. `kubelet`, `containerd`, `runc`, `/etc/containerd`, `/var/lib/containerd`, and `/var/lib/kubelet` are covered by `container-selinux`. This module does not change their labels. Control-plane binaries are labeled vcluster\_data\_t on disk The module's `.fc` file declares `container_runtime_exec_t` for the Kubernetes control-plane binaries (`kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, `etcd`, `etcdctl`, `kine`, `konnectivity-server`, `helm`, `kubectl`, `vcluster-cli`). vCluster extracts these binaries from its bundle on first start, after the RPM's `%post` has already run, so they inherit `vcluster_data_t` from their parent directory. This is functional: `container_runtime_t` has managed access to `vcluster_data_t` and generates no AVCs. To apply the declared label on disk, run `sudo restorecon -R /var/lib/vcluster/bin` after `vcluster.service` has started once. ## Verify[​](#verify "Direct link to Verify") After `install-standalone.sh` returns, the installer loads the module, the vCluster service runs under `container_runtime_t`, and the audit log contains no denials for the install window: ``` sudo semodule -l | grep '^vcluster' # vcluster ls -Z /var/lib/vcluster/bin/vcluster # system_u:object_r:container_runtime_exec_t:s0 /var/lib/vcluster/bin/vcluster sudo cat /proc/$(systemctl show -p MainPID --value vcluster.service)/attr/current # system_u:system_r:container_runtime_t:s0 ls -Z /var/lib/vcluster/pki/ca.key # system_u:object_r:vcluster_data_t:s0 /var/lib/vcluster/pki/ca.key sudo ausearch -m avc --start recent \ | grep -E 'vcluster_|container_runtime_t|container_t' || echo "no denials" ``` ## Upgrade[​](#upgrade "Direct link to Upgrade") `sudo dnf update vcluster-selinux` unloads the previous policy module, installs the new one, and reruns `restorecon` over the paths the RPM owns. No manual steps are required. If a release adds a new control-plane binary path, the release notes call out a one-off `sudo restorecon -R /var/lib/vcluster` to apply the new label. ## Uninstall[​](#uninstall "Direct link to Uninstall") ``` sudo dnf remove vcluster-selinux ``` The RPM's `%postun` unloads the policy module, removes the flannel `semanage fcontext` override, and `restorecon`s the covered paths back to their pre-install defaults. ## Troubleshoot[​](#troubleshoot "Direct link to Troubleshoot") ### vCluster service fails to start under enforcing[​](#service-fails-to-start "Direct link to vCluster service fails to start under enforcing") A denial like `avc: denied { execute }` on `container_runtime_exec_t` in `journalctl -u vcluster.service` indicates that `systemd` could not exec the vCluster binary with the expected label. Either the module is not loaded, or the binary was placed before the module relabeled the parent directory. Confirm the module and the RPM are in place: ``` rpm -q vcluster-selinux sudo semodule -l | grep '^vcluster' getenforce ``` If `rpm -q` does not return a version, install the RPM (see [Install](#install)). If the RPM and module are both present but the binary was placed before the RPM's `%post` ran, relabel and restart: ``` sudo restorecon -R /var/lib/vcluster /etc/vcluster sudo systemctl restart vcluster.service ``` ### Installer fails to fetch the RPM[​](#rpm-install-fails "Direct link to Installer fails to fetch the RPM") If the installer exits with a line like `failed to install vcluster-selinux RPM`, it could not reach `rpm.vcluster.com` and no `--selinux-rpm-url` was passed. Choose one of the following: * Allow egress to `rpm.vcluster.com` and rerun the installer. * Pre-install the RPM on the host and rerun with `--skip-selinux-rpm`. See [Install manually](#install-manually). * Rerun with `--selinux-rpm-url ` pointing to a reachable mirror. See [Install offline or with a custom RPM mirror](#install-offline). ### A control-plane binary fails to execute[​](#binary-mislabeled "Direct link to A control-plane binary fails to execute") A denial on a binary under `/var/lib/vcluster/bin/` with `tcontext=...vcluster_data_t` in `ausearch` indicates that the RPM's `.fc` file is missing an entry for that binary. Open an issue at [loft-sh/vcluster-selinux](https://github.com/loft-sh/vcluster-selinux/issues) with the binary name and the denial. As a per-host workaround: ``` sudo semanage fcontext -a -t container_runtime_exec_t '/var/lib/vcluster/bin/' sudo restorecon -v /var/lib/vcluster/bin/ sudo systemctl restart vcluster.service ``` ### Flannel pod can't write /var/run/flannel[​](#flannel-denials "Direct link to Flannel pod can't write /var/run/flannel") Confirm the RPM's `semanage fcontext` override for flannel is still in place: ``` sudo semanage fcontext -l | grep flannel # /var/run/flannel(/.*)? all files system_u:object_r:container_file_t:s0 ``` If the line is missing, reinstall the RPM. The RPM's `%post` registers the override. Remove it manually with `semanage fcontext -d`. --- # vCluster VPN Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** vCluster VPN allows you to connect the private worker nodes to the vCluster control plane through vCluster Platform. This is useful in scenarios where the nodes cannot reach the control plane directly, for example if you cannot use `LoadBalancer` or `NodePort` type services, but nodes can reach the vCluster Platform URL. vCluster VPN builds a network across the control plane and the worker nodes powered by the same technology used by [tailscale](https://tailscale.com/). You don't need a tailscale account for VPN to work as vCluster reuses tailscale as a library under the hood and will never reach out to any tailscale services directly. The only requirement for the VPN to work is that the vCluster control plane and any worker node(s) can reach the platform URL (e.g. `https://platform.my-domain.com`). vCluster VPN figures out a direct connection between worker nodes and the control plane through [wireguard](https://tailscale.com/blog/how-tailscale-works), which can traverse even complex NAT setups and make direct connections. If wireguard fails to create a direct connection, traffic is routed through the vCluster Platform.
Enabling VPN ``` privateNodes: enabled: true vpn: enabled: true # This enables node to control plane vpn nodeToNode: enabled: true # This enables node to node vpn ``` ## When to use vCluster VPN[​](#when-to-use-vcluster-vpn "Direct link to When to use vCluster VPN") ### VPN is useful in the following scenarios:[​](#vpn-is-useful-in-the-following-scenarios "Direct link to VPN is useful in the following scenarios:") * Worker nodes cannot reach the control plane directly, because the control plane service cannot be exposed with a `NodePort` or a `LoadBalancer`. * Worker nodes cannot reach each other directly, because the nodes are in different VPCs or spread across cloud and bare-metal. * Traffic between nodes is not encrypted through the CNI and might be sniffed by other parties. ### When to *not* use VPN:[​](#when-to-not-use-vpn "Direct link to when-to-not-use-vpn") * Only the control plane cannot reach the worker nodes directly. This is already solved by using [Konnectivity](https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/) and does not require vCluster VPN or any extra configuration * Nodes cannot reach the vCluster platform URL, in this case the VPN cannot be used * You need the maximum possible network throughput. While tailscale will only add minimal overhead it still cannot be tolerated in some cases * You cannot change the networking setup on the node. vCluster VPN will create a new network interface (tailscale0) and needs permissions to create it * When the goal is to isolate node networking. While vCluster VPN will encrypt all traffic, it cannot prevent nodes reaching each other or block traffic between them. If that is the goal, use a proper VLAN or underlay setup for this. Only use vCluster VPN if you have to If there is a direct way for node-to-node or node-to-control-plane communication it's usually preferred over using vCluster VPN. ## How does it work?[​](#how-does-it-work "Direct link to How does it work?") When enabled, the join script will setup a systemd service called `vcluster-vpn` that connects the node to the platform. The service runs a modified version of [tailscaled](https://tailscale.com/kb/1278/tailscaled). As soon as the node has joined the vpn it will join the node itself into the vCluster. On each vCluster control plane an integrated tailnet service is running to connect the control plane to the vpn as well. Within the Kubernetes cluster the Kubernetes endpoint will then be the tailscale ip of the control plane. If node to node communication is wanted, the default cni flannel is then configured to use the network interface `tailscale0` on the node as well as the internal node ip will be exchanged with the tailscale one. ## Node to control plane VPN[​](#node-to-control-plane-vpn "Direct link to Node to control plane VPN") Node to control plane VPN allows your nodes to communicate to the control plane over the tailscale vpn. This is useful if you cannot expose the vCluster control plane via a `LoadBalancer` service or a public ip. To configure node to control plane tunneling, you need to set the following in your `vcluster.yaml`: ``` privateNodes: enabled: true vpn: enabled: true # This enables node to control plane vpn ``` After you have started the vCluster, you should see that the endpoints of the `kubernetes` service is a `100.64.x.x` tailscale ip address: ``` $ kubectl get endpoints NAME ENDPOINTS AGE kubernetes 100.64.0.4:443 5d19h ``` ## Node to node VPN[​](#node-to-node-vpn "Direct link to Node to node VPN") Node to node VPN allows your nodes to communicate with each other over the tailscale vpn. This is useful when the nodes have no direct connection to each other, for example if you have nodes in different clouds or you use a mix of bare-metal and cloud nodes. To configure node to node VPN, you just need to set the following in your `vcluster.yaml`: ``` privateNodes: enabled: true vpn: enabled: true # This enables node to control plane vpn nodeToNode: enabled: true # This enables node to node vpn ``` Using node to node VPN requires node to control plane VPN, so you cannot opt-out of that. After you have started the vCluster and joined a node, you should see that the node ip address is a `100.64.x.x` tailscale ip address: ``` $ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME my-node Ready 5d19h v1.36.0 100.64.0.5 Ubuntu 6.14.10 containerd://2.0.5 ``` ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") If you experience any problems with the connection, you can check the vCluster VPN logs directly or via the [tailscale CLI](https://tailscale.com/kb/1080/cli) ### Make sure cni is configured to use interface tailscale0[​](#make-sure-cni-is-configured-to-use-interface-tailscale0 "Direct link to Make sure cni is configured to use interface tailscale0") If you want to use your own custom cni (not flannel) and node-to-node communication over the vpn, make sure to configure it to use interface `tailscale0` instead of the default one. ### Check vCluster VPN logs[​](#check-vcluster-vpn-logs "Direct link to Check vCluster VPN logs") vCluster VPN is installed as a systemd service on the node that should join, you can check the logs via the following command on the node: ``` journalctl -u vcluster-vpn --no-pager ``` ### Join hangs on a previously used node[​](#join-hangs-on-a-previously-used-node "Direct link to Join hangs on a previously used node") If the node was joined to a cluster that was later deleted, `vcluster-vpn` keeps that cluster's VPN state. The join script rewrites `/etc/vcluster-vpn/config.json` but doesn't restart the service, so the join waits for a control plane it can't reach. The logs show `platform access key not found. Please login again`. Restart the service on the node: ``` systemctl restart vcluster-vpn ``` The join continues once the service reconnects. ### Check with tailscale CLI[​](#check-with-tailscale-cli "Direct link to Check with tailscale CLI") Do not install tailscaled daemon Make sure to only install the CLI and NOT tailscaled as vCluster VPN is replacing that and if you install tailscale the regular way it will override and conflict with vCluster VPN. If the logs look good, you can also install tailscale CLI on the node via: ``` # Arch either amd64 or arm64 ARCH=amd64 # Download and install tailscale curl -sSLf https://pkgs.tailscale.com/stable/tailscale_1.86.0_$ARCH.tgz > tailscale.tgz && tar -zxf tailscale.tgz && mv tailscale_1.86.0_$ARCH/tailscale /usr/local/bin ``` Then you can use tailscale CLI commands to debug: ``` # Try to ping control plane or other nodes (check 'kubectl get endpoints' for the control plane ip or 'kubectl get nodes -o wide' for node ips) tailscale ping 100.64.X.X # Check if platform url shows up tailscale debug derp-map # Check if peers correctly show up tailscale debug netmap ``` ### Changing settings after node join[​](#changing-settings-after-node-join "Direct link to Changing settings after node join") When changing the way the virtual cluster control plane is exposed, e.g. switching from LoadBalancer to VPN, you will need to re-join the nodes to pick up the new settings. ### Configure firewall to enable Tailscale to establish peer-to-peer connections[​](#configure-firewall-to-enable-tailscale-to-establish-peer-to-peer-connections "Direct link to Configure firewall to enable Tailscale to establish peer-to-peer connections") While it is possible to utilize the vCluster Platform as a DERP relay server, there are cases where the additional latency is not acceptable. You may check whether your nodes are using a relay using the `tailscale status` command: ``` 1 2 3 4 5 100.1.2.3 device-a alice@ linux active; direct , tx 1116 rx 1124 100.4.5.6 device-b bob@ macOS active; relay , tx 1351 rx 4262 100.7.8.9 device-c charlie@ windows idle; tx 1214 rx 50 100.0.1.2 device-d diane@ iOS — ``` If you notice that two devices are connected over a relay server, you can try opening firewall ports to enable them to connect peer-to-peer (direct): * UDP connections from `:41641` to `*:*`. * UDP connections to `*:3478` * TCP connections to `*:80` * TCP connections to `*:443` In some cases it may be enough to enable connections to UDP port 41641. --- # cert-manager Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. You can integrate cert-manager with your tenant cluster to manage TLS certificates across control plane and tenant environments. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Run the command `kubectl auth can-i create clusterrole -A` to verify that your current kube-context has administrative privileges. info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm`: Helm v3.10 is required for deploying the platform. Refer to the [Helm Installation Guide](https://Helm.sh/docs/intro/install/) if you need to install it. * `kubectl`: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. - cert-manager operator installed on your control plane cluster. For setup instructions, see the [cert-manager installation guide](https://cert-manager.io/docs/installation/). ## Enable the integration[​](#enable-the-integration "Direct link to Enable the integration") Enable cert-manager integration in your tenant cluster configuration: Enable cert-manager integration ``` integrations: certManager: enabled: true ``` This configuration enables the integration and configures automatic resource sync: * **Imports**: Cluster-scoped `ClusterIssuers` from your control plane cluster into the tenant cluster * **Exports**: Namespaced `Issuers` and `Certificates` from the tenant cluster to the control plane cluster * **Syncs**: Generated certificate secrets back to the tenant cluster automatically ## Set up cluster contexts[​](#set-up-cluster-contexts "Direct link to Set up cluster contexts") Create or update a tenant cluster following the [vCluster quick start guide](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md). Configure cluster contexts to simplify switching between control plane and tenant clusters: ``` export HOST_CTX="your-host-context" export VCLUSTER_CTX="vcluster-ctx" ``` Find your contexts Run `kubectl config get-contexts` to list available contexts. ## Certificate issuer configurations[​](#certificate-issuer-configurations "Direct link to Certificate issuer configurations") Choose the appropriate certificate issuer based on your security and operational requirements. Self-signed certificates for development Self-signed certificates work well for development environments and internal services that don't require public trust. 1. Tenant Cluster Create a self-signed issuer self-signed-issuer.yaml ``` apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: selfsigned-issuer namespace: default spec: selfSigned: {} ``` Apply the issuer: Apply self-signed issuer ``` kubectl --context=$VCLUSTER_CTX apply -f self-signed-issuer.yaml ``` 2. Tenant Cluster Create a certificate self-signed-certificate.yaml ``` apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: app-tls-cert namespace: default spec: secretName: app-tls-secret duration: 2160h # 90 days renewBefore: 360h # 15 days commonName: myapp.local dnsNames: - myapp.local - api.myapp.local privateKey: algorithm: RSA size: 2048 issuerRef: name: selfsigned-issuer kind: Issuer ``` Apply the certificate: Apply certificate ``` kubectl --context=$VCLUSTER_CTX apply -f self-signed-certificate.yaml ``` warning Self-signed certificates generate browser warnings and should not be used in production environments accessible to external users. Private PKI for enterprise environments Private PKI configurations provide enterprise-grade certificate management with custom certificate authorities. 1. Control Plane Cluster Create a root CA certificate Create a self-signed root CA in the control plane cluster (ClusterIssuers are imported from control plane to tenant cluster): root-ca.yaml ``` apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-issuer spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: root-ca-cert namespace: cert-manager spec: isCA: true commonName: "Enterprise Root CA" subject: organizations: - "Enterprise Corp" organizationalUnits: - "IT Security" countries: - "US" secretName: root-ca-secret duration: 87600h # 10 years renewBefore: 78840h # 9 years privateKey: algorithm: ECDSA size: 256 issuerRef: name: selfsigned-issuer kind: ClusterIssuer ``` Apply in the control plane cluster: Apply root CA in control plane cluster ``` kubectl --context=$HOST_CTX apply -f root-ca.yaml ``` 2. Control Plane Cluster Create a CA issuer ca-issuer.yaml ``` apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: enterprise-ca-issuer spec: ca: secretName: root-ca-secret ``` Apply in the control plane cluster: Apply CA issuer in control plane cluster ``` kubectl --context=$HOST_CTX apply -f ca-issuer.yaml ``` 3. Tenant Cluster Issue certificates from your CA ca-signed-certificate.yaml ``` apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: enterprise-app-cert namespace: default spec: secretName: enterprise-app-tls duration: 8760h # 1 year renewBefore: 720h # 30 days commonName: myapp.enterprise.local dnsNames: - myapp.enterprise.local - api.myapp.enterprise.local subject: organizations: - "Enterprise Corp" organizationalUnits: - "Application Services" usages: - digital signature - key encipherment - server auth issuerRef: name: enterprise-ca-issuer kind: ClusterIssuer ``` Apply in the tenant cluster: Apply certificate in tenant cluster ``` kubectl --context=$VCLUSTER_CTX apply -f ca-signed-certificate.yaml ``` **Security considerations**: * Store root CA private keys securely and offline when possible * Implement proper certificate lifecycle management * Monitor certificate expiration dates * Establish clear revocation procedures ACME for public-facing services [ACME](https://cert-manager.io/docs/configuration/acme/) (Automatic Certificate Management Environment) issuers like [Let's Encrypt](https://letsencrypt.org) provide free, automatically-renewed certificates for public-facing services. 1. Tenant Cluster Create an ACME issuer acme-issuer.yaml ``` apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: letsencrypt-prod namespace: default spec: acme: email: admin@yourdomain.com server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt-prod-account-key solvers: - http01: ingress: ingressClassName: nginx # the ingress-nginx project has been deprecated, we recommend using a different ingress class - dns01: cloudflare: email: admin@yourdomain.com apiKeySecretRef: name: cloudflare-api-key-secret key: api-key ``` 2. Tenant Cluster Create an ACME certificate acme-certificate.yaml ``` apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: public-app-cert namespace: default spec: secretName: public-app-secret duration: 2160h # 90 days renewBefore: 720h # 30 days dnsNames: - myapp.example.com - api.myapp.example.com issuerRef: name: letsencrypt-prod kind: Issuer ``` **Rate limits**: Let's Encrypt enforces rate limits. Use staging environment (`https://acme-staging-v02.api.letsencrypt.org/directory`) for testing. PKI for compliance requirements PKI configurations require specific certificate formats and validation procedures for compliance with standards. gov-compliant-certificate.yaml ``` apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: gov-service-cert namespace: default spec: secretName: gov-service-secret duration: 8760h # 1 year maximum renewBefore: 2160h # 90 days commonName: service.application.1234567890 subject: organizations: - "Organization" organizationalUnits: - "PKI" - "CONTRACTOR" countries: - "US" usages: - digital signature - key encipherment - server auth privateKey: algorithm: RSA size: 2048 # Federal minimum requirement issuerRef: name: gov-pki-issuer kind: ClusterIssuer ``` **Compliance requirements**: * Use hierarchical 3-tier CA structure * Include specific Distinguished Name fields * Maintain Certificate Revocation Lists (CRLs) * Follow Federal PKI Certificate Policy guidelines ## Verify certificate creation[​](#verify-certificate-creation "Direct link to Verify certificate creation") Check that your certificates generate correctly: 1. Tenant Cluster Check certificate status Verify certificate creation ``` kubectl --context=$VCLUSTER_CTX describe certificate -n ``` Look for `Ready: True` in the status conditions. 2. Tenant Cluster Verify secret creation Check certificate secret ``` kubectl --context=$VCLUSTER_CTX get secret -n -o yaml ``` Confirm the secret contains `tls.crt` and `tls.key` data. 3. Control Plane Cluster Confirm synchronization Check control plane cluster resources ``` kubectl --context=$HOST_CTX get certificate,issuer -A ``` Verify resources appear in the control plane cluster namespace. ## Use certificates in applications[​](#use-certificates-in-applications "Direct link to Use certificates in applications") Reference your certificate secrets in Ingress resources: secure-ingress.yaml ``` apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: secure-app-ingress annotations: kubernetes.io/ingress.class: nginx # ingress-nginx has been deprecated nginx.ingress.kubernetes.io/ssl-redirect: "true" spec: tls: - hosts: - myapp.example.com secretName: your-tls-secret # Use the secretName from your Certificate rules: - host: myapp.example.com http: paths: - path: / pathType: Prefix backend: service: name: app-service port: number: 80 ``` tip Replace `your-tls-secret` with the actual `secretName` specified in your Certificate resource (e.g., `app-tls-secret`, `enterprise-app-tls`, or `public-app-secret` from the examples above). ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") Certificate creation issues **Check certificate status**: ``` kubectl --context=$VCLUSTER_CTX describe certificate ``` **Common problems**: * **Pending status**: Check CertificateRequest events * **Failed validation**: Verify DNS/HTTP challenge accessibility * **Rate limiting**: Switch to staging ACME server for testing **Examine certificate requests**: ``` kubectl --context=$VCLUSTER_CTX get certificaterequests kubectl --context=$VCLUSTER_CTX describe certificaterequest ``` Integration synchronization problems **Verify integration status**: Control Plane Cluster Check cert-manager installation ``` kubectl --context=$HOST_CTX -n cert-manager get pods kubectl --context=$HOST_CTX -n cert-manager logs deployment/cert-manager ``` Tenant Cluster Verify integration configuration ``` # Check if integration is enabled kubectl --context=$VCLUSTER_CTX get configmap vcluster-config -o yaml # Check resource synchronization kubectl --context=$VCLUSTER_CTX get events --field-selector reason=Sync ``` **Common synchronization issues**: * RBAC permissions missing on control plane cluster * Network policies blocking communication * Resource naming conflicts between clusters ACME challenge failures For detailed troubleshooting of ACME challenges: **Check challenge status**: ``` kubectl --context=$VCLUSTER_CTX describe order kubectl --context=$VCLUSTER_CTX describe challenge ``` For comprehensive ACME troubleshooting, including HTTP-01 and DNS-01 challenge issues, see the [cert-manager ACME troubleshooting guide](https://cert-manager.io/docs/troubleshooting/acme/). For troubleshooting guidance, see the [cert-manager troubleshooting documentation](https://cert-manager.io/docs/troubleshooting/). ## Advanced configuration[​](#advanced-configuration "Direct link to Advanced configuration") Custom sync configurations Fine-tune resource synchronization behavior: Advanced cert-manager integration ``` integrations: certManager: enabled: true sync: toHost: certificates: enabled: true issuers: enabled: true fromHost: clusterIssuers: enabled: true selector: labels: vcluster.loft.sh/managed: "true" ``` **Configuration options**: * `selector`: Filter which ClusterIssuers to import * Individual resource type controls for granular sync management ## Configuration reference[​](#configuration-reference "Direct link to Configuration reference") ## `certManager` object [​](#certManager "Direct link to certManager")CertManager reuses a host cert-manager and makes its CRDs from it available inside the vCluster.- Certificates and Issuers will be synced from the virtual cluster to the host cluster. - ClusterIssuers will be synced from the host cluster to the virtual cluster. ### `enabled` boolean false [​](#certManager-enabled "Direct link to certManager-enabled")Enabled defines if this option should be enabled. ### `sync` object [​](#certManager-sync "Direct link to certManager-sync")Sync contains advanced configuration for syncing cert-manager resources. #### `toHost` object [​](#certManager-sync-toHost "Direct link to certManager-sync-toHost") ##### `certificates` object [​](#certManager-sync-toHost-certificates "Direct link to certManager-sync-toHost-certificates")Certificates defines if certificates should get synced from the virtual cluster to the host cluster. ##### `enabled` boolean true [​](#certManager-sync-toHost-certificates-enabled "Direct link to certManager-sync-toHost-certificates-enabled")Enabled defines if this option should be enabled. ##### `issuers` object [​](#certManager-sync-toHost-issuers "Direct link to certManager-sync-toHost-issuers")Issuers defines if issuers should get synced from the virtual cluster to the host cluster. ##### `enabled` boolean true [​](#certManager-sync-toHost-issuers-enabled "Direct link to certManager-sync-toHost-issuers-enabled")Enabled defines if this option should be enabled. #### `fromHost` object [​](#certManager-sync-fromHost "Direct link to certManager-sync-fromHost") ##### `clusterIssuers` object [​](#certManager-sync-fromHost-clusterIssuers "Direct link to certManager-sync-fromHost-clusterIssuers")ClusterIssuers defines if (and which) cluster issuers should get synced from the host cluster to the virtual cluster. ##### `enabled` boolean true [​](#certManager-sync-fromHost-clusterIssuers-enabled "Direct link to certManager-sync-fromHost-clusterIssuers-enabled")Enabled defines if this option should be enabled. ##### `selector` object [​](#certManager-sync-fromHost-clusterIssuers-selector "Direct link to certManager-sync-fromHost-clusterIssuers-selector")Selector defines what cluster issuers should be imported. ##### `labels` object {} [​](#certManager-sync-fromHost-clusterIssuers-selector-labels "Direct link to certManager-sync-fromHost-clusterIssuers-selector-labels")Labels defines what labels should be looked for --- # Custom Plugins ### Overview[​](#overview "Direct link to Overview") vCluster plugins are Kubernetes operators that can access both the tenant cluster and the control plane cluster, enabling advanced syncing, resource management, and custom automation. They run inside the vCluster syncer and can define syncers, hooks, or controllers to align resources, modify behavior, or enforce policies across clusters. This dual-cluster access allows plugins to extend vCluster capability beyond what standard operators can do within a single cluster. You can configure plugins [here](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/plugins.md#config-reference) in `vcluster.yaml`. ### Custom plugins[​](#custom-plugins "Direct link to Custom plugins") Besides the existing vCluster plugins, you can also develop your own vCluster plugins. Recommended Reads If you want to start developing your own vCluster plugins, read about [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) and [kube builder](https://book.kubebuilder.io/introduction.html), which uses the controller runtime internally. vCluster provides an [SDK](https://github.com/loft-sh/vcluster-sdk) for writing plugin controllers that abstracts a lot of the syncer complexity away from the user, but still gives you access to the underlying structures if you need it. Internally, the vCluster SDK uses the [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) project, which is used by vCluster itself to create the controllers. The vCluster SDK lets you write custom plugin controllers with just a few lines of code. Since the plugin SDK interfaces are mostly compatible with the vCluster syncers, you can also take a look at how those are implemented in [the vCluster itself](https://github.com/loft-sh/vcluster/tree/main/pkg/controllers/resources), which work in most cases the same way as if those would be implemented in a plugin. It would be even possible to reimplement all vCluster syncers in a separate plugin. --- # External Secrets Operator Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * `kubectl` installed * `external-secrets` operator installed on your control plane cluster. See instructions at External secrets version By default, vCluster uses the same CRD version as the External Secrets Operator installed on your control plane cluster. The `version` field allows you to explicitly set which CRD version to use (for example, `v1beta1` or `v1`). Ensure your chosen version is supported by the External Secrets Operator on your control plane cluster. # External secrets integration To enable the external secret integration, set the following fields: ``` integrations: externalSecrets: enabled: true version: v1 # Optional. If not specified, uses the same CRD version as your host cluster's External Secrets Operator sync: toHost: stores: enabled: true fromHost: clusterStores: enabled: true ``` This enables the integration and the sync for all custom resources: * `ExternalSecret`: namespaced, synced from tenant cluster into control plane cluster and then bi-directionally * `SecretStore`: namespaced, synced from tenant cluster into control plane cluster * `ClusterSecretStore`: cluster scoped, synced from control plane cluster into tenant cluster Once the tenant cluster is up and running, you can create a `SecretStore` inside the tenant cluster. For this guide, you use the `fake` store type, which prefills data instead of connecting to a distant secret store. API Version Compatibility External Secrets Operator `v0.16.2` is the last supporting `v1beta1` API version. For `v0.17.0+` versions, it provides exclusively `v1`. ``` apiVersion: external-secrets.io/v1 kind: SecretStore metadata: name: fake spec: provider: fake: data: - key: "/foo/bar" value: "HELLO1" version: "v1" - key: "/foo/bar" value: "HELLO2" version: "v2" - key: "/foo/baz" value: '{"john": "doe"}' version: "v1" ``` Inside the tenant cluster, create the store with `kubectl apply -f fake.yaml`. This creates a corresponding store in the control plane cluster. You can then create an `ExternalSecret` in the tenant cluster, which references the `SecretStore`. ``` apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: example spec: refreshInterval: 1h secretStoreRef: name: fake kind: SecretStore target: name: secret-to-be-created data: - secretKey: foo_bar remoteRef: key: /foo/bar version: v1 dataFrom: - extract: key: /foo/baz version: v1 ``` After the `ExternalSecret` is created in the tenant cluster, the integration creates a corresponding resource inside the control plane cluster. The external secret operator running in the control plane cluster creates the corresponding Kubernetes secret which the integration imports into the tenant cluster. Running `kubectl get secrets` in the tenant cluster includes the `secret-to-be-created` in its output. ## Config reference[​](#config-reference "Direct link to Config reference") ## `externalSecrets` object [​](#externalSecrets "Direct link to externalSecrets")ExternalSecrets reuses a host external secret operator and makes certain CRDs from it available inside the vCluster.- ExternalSecrets will be synced from the virtual cluster to the host cluster. - SecretStores will be synced from the virtual cluster to the host cluster and then bi-directionally. - ClusterSecretStores will be synced from the host cluster to the virtual cluster. ### `enabled` boolean false [​](#externalSecrets-enabled "Direct link to externalSecrets-enabled")Enabled defines whether the external secret integration is enabled or not ### `version` string [​](#externalSecrets-version "Direct link to externalSecrets-version")Version defines the version of the external secrets operator to use. If empty, the storage version will be used. ### `webhook` object [​](#externalSecrets-webhook "Direct link to externalSecrets-webhook")Webhook defines whether the host webhooks are reused or not #### `enabled` boolean false [​](#externalSecrets-webhook-enabled "Direct link to externalSecrets-webhook-enabled")Enabled defines if this option should be enabled. ### `sync` object [​](#externalSecrets-sync "Direct link to externalSecrets-sync")Sync defines the syncing behavior for the integration #### `toHost` object [​](#externalSecrets-sync-toHost "Direct link to externalSecrets-sync-toHost")ToHost defines what resources are synced from the virtual cluster to the host ##### `externalSecrets` object [​](#externalSecrets-sync-toHost-externalSecrets "Direct link to externalSecrets-sync-toHost-externalSecrets")ExternalSecrets allows to configure if only a subset of ExternalSecrets matching a label selector should get synced from the virtual cluster to the host cluster. ##### `selector` object [​](#externalSecrets-sync-toHost-externalSecrets-selector "Direct link to externalSecrets-sync-toHost-externalSecrets-selector") ##### `matchLabels` object {} [​](#externalSecrets-sync-toHost-externalSecrets-selector-matchLabels "Direct link to externalSecrets-sync-toHost-externalSecrets-selector-matchLabels") ##### `matchExpressions` object\[] [​](#externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions "Direct link to externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions") ##### `key` string [​](#externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-key "Direct link to externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-key") ##### `operator` string [​](#externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-operator "Direct link to externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-operator") ##### `values` string\[] [​](#externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-values "Direct link to externalSecrets-sync-toHost-externalSecrets-selector-matchExpressions-values") ##### `stores` object [​](#externalSecrets-sync-toHost-stores "Direct link to externalSecrets-sync-toHost-stores")Stores defines if secret stores should get synced from the virtual cluster to the host cluster and then bi-directionally. ##### `selector` object [​](#externalSecrets-sync-toHost-stores-selector "Direct link to externalSecrets-sync-toHost-stores-selector") ##### `matchLabels` object {} [​](#externalSecrets-sync-toHost-stores-selector-matchLabels "Direct link to externalSecrets-sync-toHost-stores-selector-matchLabels") ##### `matchExpressions` object\[] [​](#externalSecrets-sync-toHost-stores-selector-matchExpressions "Direct link to externalSecrets-sync-toHost-stores-selector-matchExpressions") ##### `key` string [​](#externalSecrets-sync-toHost-stores-selector-matchExpressions-key "Direct link to externalSecrets-sync-toHost-stores-selector-matchExpressions-key") ##### `operator` string [​](#externalSecrets-sync-toHost-stores-selector-matchExpressions-operator "Direct link to externalSecrets-sync-toHost-stores-selector-matchExpressions-operator") ##### `values` string\[] [​](#externalSecrets-sync-toHost-stores-selector-matchExpressions-values "Direct link to externalSecrets-sync-toHost-stores-selector-matchExpressions-values") ##### `enabled` boolean false [​](#externalSecrets-sync-toHost-stores-enabled "Direct link to externalSecrets-sync-toHost-stores-enabled")Enabled defines if this option should be enabled. #### `fromHost` object [​](#externalSecrets-sync-fromHost "Direct link to externalSecrets-sync-fromHost")FromHost defines what resources are synced from the host cluster to the virtual cluster ##### `clusterStores` object [​](#externalSecrets-sync-fromHost-clusterStores "Direct link to externalSecrets-sync-fromHost-clusterStores")ClusterStores defines if cluster secrets stores should get synced from the host cluster to the virtual cluster. ##### `selector` object [​](#externalSecrets-sync-fromHost-clusterStores-selector "Direct link to externalSecrets-sync-fromHost-clusterStores-selector") ##### `matchLabels` object {} [​](#externalSecrets-sync-fromHost-clusterStores-selector-matchLabels "Direct link to externalSecrets-sync-fromHost-clusterStores-selector-matchLabels") ##### `matchExpressions` object\[] [​](#externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions "Direct link to externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions") ##### `key` string [​](#externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-key "Direct link to externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-key") ##### `operator` string [​](#externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-operator "Direct link to externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-operator") ##### `values` string\[] [​](#externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-values "Direct link to externalSecrets-sync-fromHost-clusterStores-selector-matchExpressions-values") ##### `enabled` boolean false [​](#externalSecrets-sync-fromHost-clusterStores-enabled "Direct link to externalSecrets-sync-fromHost-clusterStores-enabled")Enabled defines if this option should be enabled. --- # GPU and inference autoscaling Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** This guide shows how to configure [Horizontal Pod Autoscaler (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) and KEDA scaling inside a vCluster with [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md). The metrics pipeline uses NVIDIA DCGM Exporter, Prometheus, and Prometheus Adapter to expose real GPU hardware metrics through the Kubernetes custom metrics API. The same Prometheus pipeline can also expose model-serving metrics such as queue depth, concurrency, and latency. ## Use cases[​](#use-cases "Direct link to Use cases") GPU and inference autoscaling help when your workloads have variable GPU utilization or request load: * **Inference serving**: Scale model-serving Deployments up during traffic spikes and down during quiet periods to reduce idle GPU cost. * **Batch processing**: Run multiple GPU jobs with a shared pool of replicas that grows and shrinks based on actual GPU load. * **Development clusters**: Give data-science teams a vCluster with autoscaling GPU workloads while maintaining hard limits through `maxReplicas`. ## How it works[​](#how-it-works "Direct link to How it works") With Private Nodes, vCluster workloads run directly on dedicated physical nodes rather than inside control plane cluster pods. This means: * DCGM Exporter DaemonSet pods are scheduled directly on the private GPU nodes and have access to real GPU hardware through the NVIDIA Management Library (NVML). * GPU metrics reflect actual hardware utilization, not virtualized or approximated values. * HPA scaling decisions are based on real hardware load, making autoscaling reliable for GPU-intensive workloads. For hardware metrics, the pipeline flows through four components: ``` GPU Node (Private) └── DCGM Exporter DaemonSet ← scrapes GPU hardware metrics └── Prometheus ← collects via ServiceMonitor └── Prometheus Adapter ← exposes as custom metrics API └── HPA ← scales pods on GPU utilization ``` For model-serving metrics, the runtime exposes Prometheus metrics from the inference pod. Prometheus scrapes those metrics and either: * Prometheus Adapter exposes them to HPA as custom metrics. * KEDA queries Prometheus directly and scales the Deployment from the query result. Use hardware metrics for capacity signals such as GPU utilization and memory pressure. Use serving metrics for demand signals such as request concurrency, queue depth, time to first token, tokens per second, and p95 or p99 latency. Inference providers usually need both. If you are building an inference provider platform, see [Inference Provider: Managed Model Serving](https://vcluster.com/docs/vcluster/production-guide/inference-provider.md) for the full production path including tenancy model, endpoint routing, and Day 2 operations. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * A vCluster with Private Nodes enabled and at least one GPU node attached * NVIDIA GPU drivers and the NVIDIA device plugin installed on the private GPU nodes, usually through the [NVIDIA GPU Operator](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md#install-the-nvidia-gpu-operator-in-a-tenant-cluster) * `helm` and `kubectl` connected to the vCluster ## Step 1: Install DCGM exporter[​](#step-1-install-dcgm-exporter "Direct link to Step 1: Install DCGM exporter") Already running the NVIDIA GPU Operator? The GPU Operator deploys DCGM Exporter by default. Installing the standalone chart below alongside it creates two competing DaemonSets on the same GPU nodes. If the Operator manages your GPU stack, skip this step. To add Kubernetes pod labels such as endpoint, customer, or tier to the exported metrics, enable the optional label enrichment shown in [GPU and accelerator support](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md#install-the-nvidia-gpu-operator-in-a-tenant-cluster). Then continue with Step 2, and follow the GPU Operator path in Step 3 for the ServiceMonitor. Only follow the steps below if you manage GPU drivers and device plugins outside the GPU Operator. Install the [NVIDIA DCGM Exporter](https://github.com/NVIDIA/dcgm-exporter) inside the vCluster. It runs as a DaemonSet on the private GPU nodes and exposes per-GPU metrics in Prometheus format at port `9400/metrics`. ``` helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts helm repo update helm install dcgm-exporter gpu-helm-charts/dcgm-exporter \ --namespace monitoring \ --create-namespace ``` What enablePodLabels adds DCGM Exporter's Kubernetes mapping adds the core `pod`, `namespace`, and `container` attribution through the kubelet pod-resources API. The HPA example below only requires this core attribution. To copy labels from the Kubernetes Pod object onto GPU metrics, add `--set kubernetes.enablePodLabels=true` to the Helm command. These extra dimensions let you query by endpoint, customer, model, or tier. Verify the DaemonSet is running on your GPU node: ``` kubectl get daemonset dcgm-exporter -n monitoring ``` Confirm the metrics endpoint is reachable and returning GPU data: ``` DCGM_POD=$(kubectl get pods -l app.kubernetes.io/name=dcgm-exporter -n monitoring -o jsonpath='{.items[0].metadata.name}') kubectl port-forward $DCGM_POD 9400:9400 -n monitoring & curl -s http://localhost:9400/metrics | grep DCGM_FI_DEV_GPU_UTIL # Expected: DCGM_FI_DEV_GPU_UTIL{gpu="0",...,pod="",namespace="",...} 42 ``` ## Step 2: Install Prometheus[​](#step-2-install-prometheus "Direct link to Step 2: Install Prometheus") Install the `kube-prometheus-stack` Helm chart inside the vCluster. This deploys Prometheus, the Prometheus Operator, and Alertmanager. ``` helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install kube-prometheus prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --create-namespace \ --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false ``` serviceMonitorSelectorNilUsesHelmValues Setting this to `false` tells Prometheus to discover all ServiceMonitor resources in the cluster, not only those created by the Helm release. This is required for Prometheus to find the DCGM Exporter ServiceMonitor created in the next step. Verify that Prometheus pods are running: ``` kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus ``` ## Step 3: Configure a ServiceMonitor for DCGM exporter[​](#step-3-configure-a-servicemonitor-for-dcgm-exporter "Direct link to Step 3: Configure a ServiceMonitor for DCGM exporter") Prometheus needs a ServiceMonitor that matches the DCGM Exporter Service you're running. The standalone chart and the GPU Operator use different Service labels, namespaces, and port names, so follow the path that matches your Step 1 choice. ### Use the GPU Operator path[​](#use-the-gpu-operator-path "Direct link to Use the GPU Operator path") The Operator can create and manage the ServiceMonitor for you. Now that Step 2 installed the Prometheus Operator CRDs, enable it on the Operator, reusing the values you already set: ``` helm upgrade gpu-operator nvidia/gpu-operator \ --namespace gpu-operator \ --reuse-values \ --set dcgmExporter.serviceMonitor.enabled=true ``` Because Step 2 set `serviceMonitorSelectorNilUsesHelmValues=false`, Prometheus discovers this ServiceMonitor automatically. It selects the Operator's `nvidia-dcgm-exporter` Service in the `gpu-operator` namespace on the `gpu-metrics` port. If you'd rather define the ServiceMonitor yourself, apply this instead of enabling the Operator flag: servicemonitor-dcgm-operator.yaml ``` apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: nvidia-dcgm-exporter namespace: monitoring spec: namespaceSelector: matchNames: - gpu-operator selector: matchLabels: app: nvidia-dcgm-exporter endpoints: - port: gpu-metrics interval: 15s path: /metrics ``` ### Standalone chart path[​](#standalone-chart-path "Direct link to Standalone chart path") If you installed the standalone DCGM Exporter in Step 1, create this ServiceMonitor instead: servicemonitor-dcgm.yaml ``` apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: dcgm-exporter namespace: monitoring spec: selector: matchLabels: app.kubernetes.io/name: dcgm-exporter endpoints: - port: metrics interval: 15s path: /metrics ``` Apply it: ``` kubectl apply -f servicemonitor-dcgm.yaml ``` ### Confirm the target[​](#confirm-the-target "Direct link to Confirm the target") Confirm Prometheus is scraping DCGM by checking the Prometheus targets UI: ``` kubectl port-forward svc/kube-prometheus-kube-prome-prometheus 9090:9090 -n monitoring # Open http://localhost:9090/targets and verify the dcgm-exporter target shows State: UP ``` Service name The Prometheus service name `kube-prometheus-kube-prome-prometheus` is generated from the Helm release name `kube-prometheus`. If you used a different release name, run `kubectl get svc -n monitoring` to find the correct service name. ## Step 4: Install Prometheus adapter[​](#step-4-install-prometheus-adapter "Direct link to Step 4: Install Prometheus adapter") The Prometheus Adapter translates Prometheus metrics into the Kubernetes [custom metrics API](https://github.com/kubernetes/metrics), which HPA can query. Create a values file for the adapter that maps `DCGM_FI_DEV_GPU_UTIL` to a custom metric named `gpu_utilization`: prometheus-adapter-values.yaml ``` prometheus: url: http://kube-prometheus-kube-prome-prometheus.monitoring.svc port: 9090 rules: custom: - seriesQuery: 'DCGM_FI_DEV_GPU_UTIL{namespace!="",pod!=""}' resources: overrides: namespace: resource: namespace pod: resource: pod name: matches: "DCGM_FI_DEV_GPU_UTIL" as: "gpu_utilization" metricsQuery: 'avg(DCGM_FI_DEV_GPU_UTIL{<<.LabelMatchers>>}) by (<<.GroupBy>>)' ``` Install the adapter: ``` helm install prometheus-adapter prometheus-community/prometheus-adapter \ --namespace monitoring \ -f prometheus-adapter-values.yaml ``` Verify the custom metric is available: ``` kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq '.resources[].name' | grep gpu # Expected: "pods/gpu_utilization" ``` ## Available GPU metrics[​](#available-gpu-metrics "Direct link to Available GPU metrics") The default DCGM Exporter profile exposes several metrics. You can use any of these as HPA targets by adding entries under `rules.custom` in the Prometheus Adapter config: | Metric | Description | | ------------------------- | ------------------------------------ | | `DCGM_FI_DEV_GPU_UTIL` | GPU core utilization (%) | | `DCGM_FI_DEV_FB_USED` | GPU framebuffer memory used (MiB) | | `DCGM_FI_DEV_POWER_USAGE` | GPU power draw (W) | | `DCGM_FI_DEV_SM_CLOCK` | Streaming multiprocessor clock (MHz) | | `DCGM_FI_DEV_MEM_CLOCK` | Memory clock (MHz) | For memory-bound workloads such as large language model inference, `DCGM_FI_DEV_FB_USED` may be a better scaling signal than GPU core utilization. ## Scale on serving metrics[​](#scale-on-serving-metrics "Direct link to Scale on serving metrics") GPU utilization is useful, but it is often a lagging signal for inference. A model server can have a growing request queue while GPU utilization is still low, or a saturated key-value cache while core utilization looks healthy. Add serving metrics from your runtime and use them alongside DCGM metrics. Choose one owner for the workload's autoscaler. The following HPA and KEDA examples target a direct Deployment. If KServe manages the workload, configure its autoscaler through the `InferenceService` instead of creating a competing HPA or `ScaledObject`. Common serving metrics include: | Signal | Use when | | ---------------------------- | ------------------------------------------------------------------------------------ | | Request concurrency | You want to keep active requests per replica below a service-level target. | | Queue depth | You want to scale before queued requests increase latency. | | Time to first token | You serve streaming responses and need to maintain low interactive latency. | | Tokens per second | You need throughput-based scaling for generated-token workloads. | | p95 or p99 latency | You scale based on customer-visible latency, often with smoothing to avoid flapping. | | GPU memory or cache pressure | You need to protect model cache, KV cache, or batch admission behavior. | ### Expose runtime metrics to Prometheus[​](#expose-runtime-metrics-to-prometheus "Direct link to Expose runtime metrics to Prometheus") Most serving runtimes expose Prometheus metrics on an HTTP endpoint such as `/metrics`. Add a port and a ServiceMonitor for the runtime. The exact port and metric names depend on the runtime. servicemonitor-vllm.yaml ``` apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: vllm-inference namespace: inference spec: selector: matchLabels: app: vllm-inference endpoints: - port: http path: /metrics interval: 15s ``` Apply it: ``` kubectl apply -f servicemonitor-vllm.yaml ``` Confirm Prometheus contains the runtime metric you want to scale on before configuring HPA or KEDA: ``` kubectl port-forward svc/kube-prometheus-kube-prome-prometheus 9090:9090 -n monitoring # Query your runtime metric in Prometheus, for example vllm:num_requests_waiting ``` ### Scale a direct Deployment with HPA[​](#scale-a-direct-deployment-with-hpa "Direct link to Scale a direct Deployment with HPA") Add Prometheus Adapter rules for the runtime metric. Merge the serving rules into the same adapter values file you use for GPU metrics. This example maps vLLM waiting requests to a custom pod metric named `inference_queue_depth`. prometheus-adapter-serving-values.yaml ``` prometheus: url: http://kube-prometheus-kube-prome-prometheus.monitoring.svc port: 9090 rules: custom: - seriesQuery: 'vllm:num_requests_waiting{namespace!="",pod!=""}' resources: overrides: namespace: resource: namespace pod: resource: pod name: matches: "vllm:num_requests_waiting" as: "inference_queue_depth" metricsQuery: 'avg(vllm:num_requests_waiting{<<.LabelMatchers>>}) by (<<.GroupBy>>)' ``` Upgrade the adapter with the merged GPU and serving metric rules: ``` helm upgrade --install prometheus-adapter prometheus-community/prometheus-adapter \ --namespace monitoring \ -f prometheus-adapter-serving-values.yaml ``` Create an HPA that scales when average queue depth exceeds five waiting requests per pod: inference-queue-hpa.yaml ``` apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: vllm-inference-hpa namespace: inference spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: vllm-inference minReplicas: 1 maxReplicas: 8 behavior: scaleDown: stabilizationWindowSeconds: 300 metrics: - type: Pods pods: metric: name: inference_queue_depth target: type: AverageValue averageValue: "5" ``` Apply it: ``` kubectl apply -f inference-queue-hpa.yaml kubectl get hpa vllm-inference-hpa -n inference -w ``` ### Scale a direct Deployment with KEDA[​](#scale-a-direct-deployment-with-keda "Direct link to Scale a direct Deployment with KEDA") KEDA is often a good fit when your serving signal is naturally expressed as a Prometheus query. [Install KEDA](https://keda.sh/docs/latest/deploy/) in the tenant cluster, then create a `ScaledObject` that queries Prometheus. vllm-keda-scaledobject.yaml ``` apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: vllm-inference namespace: inference spec: scaleTargetRef: name: vllm-inference minReplicaCount: 1 maxReplicaCount: 8 cooldownPeriod: 300 triggers: - type: prometheus metadata: serverAddress: http://kube-prometheus-kube-prome-prometheus.monitoring.svc:9090 metricName: inference_queue_depth threshold: "5" query: sum(vllm:num_requests_waiting{namespace="inference",pod=~"vllm-inference-.*"}) ``` Use KEDA for query-driven scaling or scale-to-zero patterns. The example keeps one warm replica because `minReplicaCount` is `1`. Set it to `0` only after you design and test the endpoint's cold-start path. Use HPA with Prometheus Adapter when you want Kubernetes-native custom metrics and HPA behavior. In both cases, set conservative scale-down windows. Loading large models is slow, and repeated cold starts degrade response latency. ## Step 5: Create an HPA targeting GPU utilization[​](#step-5-create-an-hpa-targeting-gpu-utilization "Direct link to Step 5: Create an HPA targeting GPU utilization") Create an HPA that scales your GPU workload when average GPU utilization exceeds 50%. Replace `gpu-workload` with the name of your Deployment: gpu-hpa.yaml ``` apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: gpu-workload-hpa namespace: default spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: gpu-workload minReplicas: 1 maxReplicas: 4 metrics: - type: Pods pods: metric: name: gpu_utilization target: type: AverageValue averageValue: "50" # DCGM_FI_DEV_GPU_UTIL is 0–100; this targets 50% utilization ``` Apply the HPA: ``` kubectl apply -f gpu-hpa.yaml ``` Monitor scaling behavior: ``` kubectl get hpa gpu-workload-hpa -w # NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS # gpu-workload-hpa Deployment/gpu-workload 42/50 (avg) 1 4 1 ``` ## Troubleshoot common issues[​](#troubleshoot-common-issues "Direct link to Troubleshoot common issues") ### Custom metric not found[​](#custom-metric-not-found "Direct link to Custom metric not found") If `kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"` returns an error or doesn't list `gpu_utilization`: 1. Verify the Prometheus Adapter pod is running: `kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus-adapter` 2. Check Prometheus Adapter logs for configuration errors: `kubectl logs -n monitoring -l app.kubernetes.io/name=prometheus-adapter` 3. Confirm Prometheus contains DCGM data by port-forwarding to Prometheus and querying `DCGM_FI_DEV_GPU_UTIL` directly. ### HPA shows unknown targets[​](#hpa-shows-unknown-targets "Direct link to HPA shows unknown targets") This usually means the metric is registered but no data is available for the target pods: 1. Verify your GPU workload pods are running and consuming GPU resources. 2. Check that DCGM Exporter metrics include `pod` and `namespace` labels. If these labels are missing, verify that Kubernetes mapping is enabled and that the exporter can access the kubelet pod-resources socket. If labels from the Kubernetes Pod object are missing, confirm label enrichment is enabled with `kubernetes.enablePodLabels=true` for the standalone chart or `dcgmExporter.enablePodLabels=true` for the GPU Operator. 3. Wait 1–2 minutes for the metrics pipeline to propagate data from DCGM through Prometheus to the adapter. --- # Istio | | | Enterprise | | | | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Istio Integration](https://vcluster.com/docs/vcluster/integrations/istio.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. # Istio integration This guide shows how to set up Istio integration with your tenant cluster. This enables you to use one Istio installation from the control plane cluster instead of installing Istio in each tenant cluster. You can include your tenant workloads in the mesh by setting `istio.io/dataplane-mode=ambient` label on the tenant Namespaces or Pods. You can exclude your tenant workloads from the mesh by setting `istio.io/dataplane-mode=none` label either on the Namespace or on the Pod. ## Istio supported versions[​](#istio-supported-versions "Direct link to Istio supported versions") Integration works with following Istio / Gateway API Versions: | Istio Version | Gateway API Version | | ------------- | ------------------- | | v1.29 | 1.4 | | v1.30 | 1.5 | | v1.31 | 1.6 | ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Run the command `kubectl auth can-i create clusterrole -A` to verify that your current kube-context has administrative privileges. info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm`: Helm v3.10 is required for deploying the platform. Refer to the [Helm Installation Guide](https://Helm.sh/docs/intro/install/) if you need to install it. * `kubectl`: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. - `istio` operator installed on your control plane cluster in ambient mode with DNS Capture disabled warning To disable DNS capture, set `values.cni.ambient.dnsCapture: false` in your Istio configuration. This integration works only with Istio in Ambient mode. Sidecar mode is not supported. ## Enable the integration[​](#enable-the-integration "Direct link to Enable the integration") Enable the Istio integration in your tenant cluster configuration: Enable istio integration ``` integrations: istio: enabled: true ``` This configuration: * Enables the integration. * Installs Resource Definitions for `DestinationRules`, `Gateways` and `VirtualServices` into the tenant cluster. * Exports `DestinationRules`, `Gateways` and `VirtualServices` from the tenant cluster to the control plane cluster (and re-writes) service references to the services translated names in the control plane cluster. * Adds `istio.io/dataplane-mode` label to the synced Pods based on the value of this label set in the virtual namespace. warning Only `DestinationRules`, `Gateways`, and `VirtualServices` from `networking.istio.io/v1` API Version are synced to the control plane clusters. Other kinds are not yet supported. Prefer the Kubernetes Gateway API for new gateways Istio [intends to make the Kubernetes Gateway API the default API for traffic management](https://istio.io/latest/docs/tasks/traffic-management/ingress/gateway-api/). For new setups that expose tenant workloads outside the cluster, consider Kubernetes `Gateway` resources instead of Istio `Gateway` (`networking.istio.io`) resources: * To let tenants create their own `Gateway` and `HTTPRoute` resources, enable [Gateway API sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md). * To let tenants attach `HTTPRoute` resources to a shared Gateway in the control plane cluster, see [Imported Gateways and GatewayClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/gateways.md). The integration continues to sync Istio `Gateway` resources, so existing configurations keep working. `VirtualService` and `DestinationRule` resources are still required for the mesh-internal routing shown in this guide. ### Network policies[​](#network-policies "Direct link to Network policies") If you are using [network policies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md), consult the [Ambient and Kubernetes NetworkPolicy](https://istio.io/latest/docs/ambient/usage/networkpolicy/) documentation. Depending on the underlying configuration, the following policies may be required: vcluster.yaml ``` policies: networkPolicy: workload: ingress: - ports: # Allow HBONE traffic per https://istio.io/latest/docs/ambient/usage/networkpolicy/ - port: 15008 protocol: TCP - from: # Allow kubelet health probe per https://istio.io/latest/docs/ambient/usage/networkpolicy/ - ipBlock: cidr: 169.254.7.127/32 ``` ## Route request based on the version label of the app[​](#route-request-based-on-the-version-label-of-the-app "Direct link to Route request based on the version label of the app") 1. ### Set up cluster contexts[​](#set-up-cluster-contexts "Direct link to Set up cluster contexts") Setting up the control plane and tenant cluster contexts makes it easier to switch between them. You can adjust the values below according to your setup. Here, the `HOST_CONTEXT` is the context of the control plane cluster, while the `VCLUSTER_CONTEXT` is the context to access the created vCluster. The `VCLUSTER_HOST_NAMESPACE` is the namespace where the vCluster is created on the control plane cluster. The `ISTIO_NAMESPACE` is the namespace to deploy the Istio integration. The commands in the following steps will be automatically updated to follow your configuration. For the GatewayAPI version, you should set it according to your Istio version / GatewayAPI version matrix above. Modify the following with your specific values to replace on the whole page and generate copyable commands: HOST\_CONTEXTyour-host-context VCLUSTER\_CONTEXTvcluster-ctx VCLUSTER\_HOST\_NAMESPACEvcluster GATEWAY\_API\_VERSIONv1.2.1 ISTIO\_NAMESPACEistio tip You can find your contexts by running `kubectl config get-contexts` 2. ### Create waypoint proxy in the host[​](#create-waypoint-proxy-in-the-host "Direct link to Create waypoint proxy in the host") In this tutorial, you set Kubernetes service name as a host in the VirtualService `spec.hosts`. To make it work, you need a Waypoint proxy in the tenant cluster's control plane namespace. In many cases it is optional however. Refer to Istio documentation for more information on Waypoint proxies. Install Gateway CRD first in the host: Install Gateway CRD ``` kubectl --context="your-host-context" get crd gateways.gateway.networking.k8s.io &> /dev/null || \ kubectl --context="your-host-context" apply -f "https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml" ``` this is a Gateway for Waypoint you need: waypoint-gateway.yaml ``` apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: waypoint labels: istio.io/waypoint-for: service spec: gatewayClassName: istio-waypoint listeners: - name: mesh port: 15008 protocol: HBONE ``` create it in the control plane cluster: Create Waypoint Gateway ``` kubectl --context="your-host-context" create -f waypoint-gateway.yaml --namespace="vcluster" ``` 3. ### Create virtual namespace with ambient mode enabled[​](#create-virtual-namespace-with-ambient-mode-enabled "Direct link to Create virtual namespace with ambient mode enabled") First, you need to create a test namespace: Create a test namespace ``` kubectl --context="vcluster-ctx" create namespace istio ``` and label it with `istio.io/dataplane-mode: ambient`: Label the test namespace ``` kubectl --context="vcluster-ctx" label namespace istio istio.io/dataplane-mode=ambient ``` 4. ### Create two versions of your app[​](#create-two-versions-of-your-app "Direct link to Create two versions of your app") Next, you create 3 deployments: two of them are nginx server and the third one is to curl the other two. Create NGINX deployments that respond with different response bodies based on the contents of their respective ConfigMaps: configmap1.yaml ``` apiVersion: v1 kind: ConfigMap metadata: name: nginx-configmap-v1 namespace: istio data: index.html: | Welcome to nginx v1!

Hello from Nginx Version 1!

``` deployment1.yaml ``` apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment-v1 namespace: istio labels: app: nginx version: v1 spec: replicas: 1 selector: matchLabels: app: nginx version: v1 template: metadata: labels: app: nginx version: v1 spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 volumeMounts: - name: nginx-index-v1 mountPath: /usr/share/nginx/html/index.html subPath: index.html volumes: - name: nginx-index-v1 configMap: name: nginx-configmap-v1 ``` Create v1 config map ``` kubectl --context="vcluster-ctx" create -f configmap1.yaml --namespace istio ``` Create v1 deployment ``` kubectl --context="vcluster-ctx" create -f deployment1.yaml --namespace istio ``` make sure that this nginx app is up and running: Wait for v1 pods ``` kubectl --context="vcluster-ctx" wait --for=condition=ready pod -l app=nginx --namespace istio --timeout=300s ``` Create an additional NGINX deployment configured to serve a different response body, using a separate ConfigMap: configmap2.yaml ``` apiVersion: v1 kind: ConfigMap metadata: name: nginx-configmap-v2 namespace: istio data: index.html: | Welcome to nginx v2!

Hello from Nginx Version 2!

``` deployment2.yaml ``` apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment-v2 namespace: istio labels: app: nginx version: v2 spec: replicas: 1 selector: matchLabels: app: nginx version: v2 template: metadata: labels: app: nginx version: v2 spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 volumeMounts: - name: nginx-index-v2 mountPath: /usr/share/nginx/html/index.html subPath: index.html volumes: - name: nginx-index-v2 configMap: name: nginx-configmap-v2 ``` Create v2 config map ``` kubectl --context="vcluster-ctx" create -f configmap2.yaml --namespace istio ``` Create v1 deployment ``` kubectl --context="vcluster-ctx" create -f deployment2.yaml --namespace istio ``` To ensure your NGINX application is up and running in your Kubernetes cluster, use the following command: Wait for v2 pods ``` kubectl --context="vcluster-ctx" wait --for=condition=ready pod -l app=nginx --namespace istio --timeout=300s ``` Create a Service that targets Pods from both Deployments by using a shared label: service.yaml ``` apiVersion: v1 kind: Service metadata: name: nginx-service namespace: istio labels: app: nginx istio.io/use-waypoint: "waypoint" spec: ports: - port: 80 targetPort: 80 selector: app: nginx ``` The istio.io/use-waypoint: waypoint label directs Istio to route traffic for the labeled resource through the waypoint proxy within the same namespace. This configuration enables [Layer 7 (L7) policy](https://istio.io/latest/docs/ambient/usage/l7-features/) enforcement and observability features provided by the waypoint proxy. Applying this label to a namespace ensures that all Pods and Services within that namespace use the specified waypoint proxy. To deploy a Service defined in the `service.yaml` file within the test namespace of the Kubernetes cluster specified by the `VCLUSTER_CONTEXT` context, use the following command: Create service ``` kubectl --context="vcluster-ctx" create -f service.yaml --namespace istio ``` To test connectivity between the two NGINX deployments, deploy a temporary Pod equipped with `curl`: client\_deployment.yaml ``` apiVersion: apps/v1 kind: Deployment metadata: name: client namespace: istio labels: app: client spec: replicas: 1 selector: matchLabels: app: client template: metadata: labels: app: client spec: containers: - name: nginx image: nginx:latest ports: - containerPort: 80 ``` Create client deployment ``` kubectl --context="vcluster-ctx" create -f client_deployment.yaml --namespace istio ``` 5. ### Configure your desired traffic routing using DestinationRule and VirtualService[​](#configure-your-desired-traffic-routing-using-destinationrule-and-virtualservice "Direct link to Configure your desired traffic routing using DestinationRule and VirtualService") You can create `DestinationRules` and `VirtualService` in the tenant cluster. Create a pair that routes our request based on the request path: 1. Requesting `/v2` endpoint should route our request to pods with `version=v2` label 2. All other requests are routed to `version=v1` pods. Save this `DestinationRule` and `VirtualService` definition, and apply it in the tenant cluster: destination\_rule.yaml ``` apiVersion: networking.istio.io/v1 kind: DestinationRule metadata: name: nginx-destination namespace: istio spec: host: nginx-service.istio.svc.cluster.local # vCluster translates it to the host service automatically subsets: - name: v1 labels: version: v1 - name: v2 labels: version: v2 ``` virtual\_service.yaml ``` apiVersion: networking.istio.io/v1 kind: VirtualService metadata: name: nginx-service namespace: istio spec: hosts: - nginx-service.istio.svc.cluster.local # vCluster translates it to the host service automatically http: - name: "nginx-v2" match: - uri: prefix: "/v2" rewrite: uri: "/" route: - destination: host: nginx-service.istio.svc.cluster.local # vCluster translates it to the host service automatically subset: v2 - name: "nginx-v1" route: - destination: host: nginx-service.istio.svc.cluster.local # vCluster translates it to the host service automatically subset: v1 ``` To apply a `DestinationRule` configuration to the tenant cluster specified by the `VCLUSTER_CONTEXT` context, use the following command: Create destination rule ``` kubectl --context="vcluster-ctx" create -f destination_rule.yaml ``` Create virtual service ``` kubectl --context="vcluster-ctx" create -f virtual_service.yaml ``` 6. ### Verify that DestinationRule and VirtualService is synced to the control plane cluster[​](#verify-that-destinationrule-and-virtualservice-is-synced-to-the-control-plane-cluster "Direct link to Verify that DestinationRule and VirtualService is synced to the control plane cluster") Check destination rule in the control plane cluster ``` kubectl --context="your-host-context" get destinationrules --namespace "vcluster" ``` Check virtual service in the host cluster ``` kubectl --context="your-host-context" get virtualservices --namespace "vcluster" ``` You should see an output mapping from a hash value prefixed with `v` to `HOST`: ``` $ kubectl --context="your-host-context" get destinationrules --namespace "vcluster" NAME HOST AGE v7f8a9b2cafds nginx-destination-x-istio-x-vcluster 1m $ kubectl --context="your-host-context" get virtualservices --namespace "vcluster" NAME GATEWAYS HOSTS AGE v3d4e5f6g ... ["nginx-service-x-istio-x-vcluster.vcluster.svc.cluster.local"] 2m ``` 7. ### Test traffic routing[​](#test-traffic-routing "Direct link to Test traffic routing") Execute a `curl` command from within the client Pod to verify responses from the two NGINX deployments. Depending on the request path, you should receive either "Hello from Nginx Version 1!" or "Hello from Nginx Version 2!" in the response: Query version 2 ``` kubectl --context="vcluster-ctx" exec -it -n istio deploy/client -- curl nginx-service/v2 Welcome to nginx v2!

Hello from Nginx Version 2!

``` Query version 1 ``` kubectl --context="vcluster-ctx" exec -it -n istio deploy/client -- curl nginx-service Welcome to nginx v1!

Hello from Nginx Version 1!

``` Seeing the same output means that request was intercepted by Istio and routed as we specified in the `DestinationRule` and `VirtualService`. 8. ### Summary[​](#summary "Direct link to Summary") Istio integration enables you to re-use one Istio instance from the control plane cluster for multiple tenant clusters. Tenant cluster users can define their own `Gateway`, `DestinationRule` and `VirtualService` without interfering with each other. ## Fields translated during the sync to control plane cluster[​](#fields-translated-during-the-sync-to-control-plane-cluster "Direct link to Fields translated during the sync to control plane cluster") Following fields of `Gateway` are modified by vCluster during the sync to the control plane cluster: * reference to the TLS Secret is re-written `spec.servers[*].tls.credentialName`. Secret is automatically synced to the control plane cluster. * namespace, `.` and `*` prefix, followed by `/` is stripped from `spec.servers[*].hosts[*]`, so e.g. `foo-namespace/loft.sh` becomes `loft.sh` in the control plane cluster object. * additional labels `vcluster.loft.sh/managed-by: [YOUR VIRTUAL CLUSTER NAME]` and `vcluster.loft.sh/namespace: [VIRTUAL NAMESPACE]` are automatically added to the `spec.subsets[*].labels` For additional information how Secret and Service references are translated, read [How does syncing work?](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync#how-does-syncing-work) Following fields of `DestinationRule` are modified by vCluster during the sync to the control plane cluster: * reference to the virtual Kubernetes Service is re-written for `spec.host` * reference to the TLS Secret in `spec.trafficPolicy.portLevelSettings[*].tls.credentialName` & `spec.trafficPolicy.tls.credentialName` is re-written. Secrets are automatically synced to the control plane cluster. * additional labels Following fields of `VirtualService` are modified by vCluster during the sync to the control plane cluster: * reference to the virtual Kubernetes Service is re-written for: * `spec.hosts[*]` * `spec.http[*].route[*].destination.host` * `spec.http[*].mirrors[*].destination.host` * `spec.tcp[*].route[*].destination.host` * `spec.tls[*].route[*].destination.host` * reference to the `networking.istio.io/v1` kind: `Gateway` is re-written for: * `spec.gateways[*]` * `spec.http[*].match[*].gateways[*]` * `spec.tls[*].match[*].gateways[*]` * `spec.tcp[*].match[*].gateways[*]` * reference to the `networking.istio.io/v1` kind: `VirtualService` is re-written for: * `spec.http[*].delegate` Fields not supported in `VirtualService`: * `spec.exportTo` * `spec.http[*].match[*].sourceLabels` * `spec.http[*].match[*].sourceNamespace` * `spec.tcp[*].match[*].sourceLabels` * `spec.tcp[*].match[*].sourceNamespace` * `spec.tls[*].match[*].sourceLabels` * `spec.tls[*].match[*].sourceNamespace` ## Config reference[​](#config-reference "Direct link to Config reference") ## `istio` object [​](#istio "Direct link to istio")Istio syncs DestinationRules, Gateways and VirtualServices from virtual cluster to the host. ### `enabled` boolean false [​](#istio-enabled "Direct link to istio-enabled")Enabled defines if this option should be enabled. ### `sync` object [​](#istio-sync "Direct link to istio-sync") #### `toHost` object [​](#istio-sync-toHost "Direct link to istio-sync-toHost") ##### `destinationRules` object [​](#istio-sync-toHost-destinationRules "Direct link to istio-sync-toHost-destinationRules") ##### `enabled` boolean true [​](#istio-sync-toHost-destinationRules-enabled "Direct link to istio-sync-toHost-destinationRules-enabled")Enabled defines if this option should be enabled. ##### `gateways` object [​](#istio-sync-toHost-gateways "Direct link to istio-sync-toHost-gateways") ##### `enabled` boolean true [​](#istio-sync-toHost-gateways-enabled "Direct link to istio-sync-toHost-gateways-enabled")Enabled defines if this option should be enabled. ##### `virtualServices` object [​](#istio-sync-toHost-virtualServices "Direct link to istio-sync-toHost-virtualServices") ##### `enabled` boolean true [​](#istio-sync-toHost-virtualServices-enabled "Direct link to istio-sync-toHost-virtualServices-enabled")Enabled defines if this option should be enabled. --- # KubeVirt Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. vCluster offers the possibility to reuse a [KubeVirt](https://kubevirt.io/) installation of the control plane cluster within the vCluster. vCluster is able to sync [KubeVirt](https://kubevirt.io/) resources from the vCluster to the control plane cluster and allows you to use `virtctl` within the vCluster. Looking for KubeVirt as storage instead? This page covers creating KubeVirt VM resources in the tenant cluster and syncing them to the control plane cluster. If you want to provision PersistentVolumes on KubeVirt-backed private nodes instead, see [KubeVirt CSI](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/deploy.md#kubevirt-csi), a separate, unrelated feature. Tenant Pod Security doesn't see VM workloads Host-side KubeVirt creates the `virt-launcher` pod that actually runs the VM. Tenant-cluster Pod Security Standards never see the VM's tenant-side custom resource at all, since Pod Security Standards only evaluate Pods. On shared nodes, this needs two separate host-side admission policies. One evaluates the VirtualMachine or VirtualMachineInstance object for permitted networks and devices. A separate policy, or Pod Security Standards, covers the generated `virt-launcher` Pod for ordinary Pod-level security context. See [KubeVirt](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#kubevirt) in the shared-node hardening guide. ## Enable KubeVirt Integration[​](#enable-kubevirt-integration "Direct link to Enable KubeVirt Integration") To enable the integration, use the following configuration inside your `vcluster.yaml`: ``` integrations: kubeVirt: enabled: true # If you want vCluster to sync data volumes as well. #sync: # dataVolumes: # enabled: true ``` Next you can deploy and use KubeVirt resources inside the vCluster which will be synced to the control plane cluster. ## Sync DataVolumes[​](#sync-datavolumes "Direct link to Sync DataVolumes") To sync KubeVirt [DataVolumes](https://kubevirt.io/user-guide/storage/disks_and_volumes/), install the [KubeVirt Containerized Data Importer](https://kubevirt.io/user-guide/storage/containerized_data_importer/). Then enable the following setting in the `vcluster.yaml`: ``` integrations: kubeVirt: enabled: true sync: dataVolumes: enabled: true ``` After vCluster restarts, the `DataVolumes` CRD should be visible within the vCluster. ## Limitations[​](#limitations "Direct link to Limitations") vCluster currently only supports syncing and using the following resources inside the vCluster: * `VirtualMachine` * `VirtualMachineInstance` * `VirtualMachinePool` * `VirtualMachineClone` * `DataVolume` (needs to be enabled using `integrations.kubeVirt.sync.dataVolumes.enabled`) The following [virtctl](https://kubevirt.io/user-guide/user_workloads/virtctl_client_tool/) commands are supported: * `virtctl start` * `virtctl stop` * `virtctl migrate` * `virtctl ssh` * `virtctl vnc` * `virtctl pause` * `virtctl unpause` * `virtctl scp` * `virtctl restart` * `virtctl port-forward` * `virtctl console` ## How this differs from KubeVirt CSI[​](#how-this-differs-from-kubevirt-csi "Direct link to How this differs from KubeVirt CSI") This integration syncs KubeVirt VM resources (`VirtualMachine`, `VirtualMachineInstance`, and so on) from the tenant cluster to the control plane cluster. This lets you manage virtual machines from inside the tenant cluster. It doesn't provision storage. [KubeVirt CSI](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/deploy.md#kubevirt-csi) is a separate, unrelated feature. It's a CSI storage driver that provisions PersistentVolumes as CDI DataVolumes and hot-plugs them into the KubeVirt virtual machines backing the tenant cluster's private nodes. Enable it with `deploy.csi.kubeVirt.enabled`. The two features can be used together or independently. ## Config reference[​](#config-reference "Direct link to Config reference") ## `kubeVirt` object [​](#kubeVirt "Direct link to kubeVirt")KubeVirt reuses a host kubevirt and makes certain CRDs from it available inside the vCluster ### `enabled` boolean false [​](#kubeVirt-enabled "Direct link to kubeVirt-enabled")Enabled signals if the integration should be enabled ### `apiService` object [​](#kubeVirt-apiService "Direct link to kubeVirt-apiService")APIService holds information about where to find the virt-api service. Defaults to virt-api/kubevirt. #### `service` object [​](#kubeVirt-apiService-service "Direct link to kubeVirt-apiService-service")Service is a reference to the service for the API server. ##### `name` string [​](#kubeVirt-apiService-service-name "Direct link to kubeVirt-apiService-service-name")Name is the name of the host service of the apiservice. ##### `namespace` string [​](#kubeVirt-apiService-service-namespace "Direct link to kubeVirt-apiService-service-namespace")Namespace is the name of the host service of the apiservice. ##### `port` integer [​](#kubeVirt-apiService-service-port "Direct link to kubeVirt-apiService-service-port")Port is the target port on the host service to connect to. ### `webhook` object [​](#kubeVirt-webhook "Direct link to kubeVirt-webhook")Webhook holds configuration for enabling the webhook within the vCluster #### `enabled` boolean true [​](#kubeVirt-webhook-enabled "Direct link to kubeVirt-webhook-enabled")Enabled defines if this option should be enabled. ### `sync` object [​](#kubeVirt-sync "Direct link to kubeVirt-sync")Sync holds configuration on what resources to sync #### `dataVolumes` object [​](#kubeVirt-sync-dataVolumes "Direct link to kubeVirt-sync-dataVolumes")If DataVolumes should get synced ##### `enabled` boolean false [​](#kubeVirt-sync-dataVolumes-enabled "Direct link to kubeVirt-sync-dataVolumes-enabled")Enabled defines if this option should be enabled. #### `virtualMachineInstanceMigrations` object [​](#kubeVirt-sync-virtualMachineInstanceMigrations "Direct link to kubeVirt-sync-virtualMachineInstanceMigrations")If VirtualMachineInstanceMigrations should get synced ##### `enabled` boolean true [​](#kubeVirt-sync-virtualMachineInstanceMigrations-enabled "Direct link to kubeVirt-sync-virtualMachineInstanceMigrations-enabled")Enabled defines if this option should be enabled. #### `virtualMachineInstances` object [​](#kubeVirt-sync-virtualMachineInstances "Direct link to kubeVirt-sync-virtualMachineInstances")If VirtualMachineInstances should get synced ##### `enabled` boolean true [​](#kubeVirt-sync-virtualMachineInstances-enabled "Direct link to kubeVirt-sync-virtualMachineInstances-enabled")Enabled defines if this option should be enabled. #### `virtualMachines` object [​](#kubeVirt-sync-virtualMachines "Direct link to kubeVirt-sync-virtualMachines")If VirtualMachines should get synced ##### `enabled` boolean true [​](#kubeVirt-sync-virtualMachines-enabled "Direct link to kubeVirt-sync-virtualMachines-enabled")Enabled defines if this option should be enabled. #### `virtualMachineClones` object [​](#kubeVirt-sync-virtualMachineClones "Direct link to kubeVirt-sync-virtualMachineClones")If VirtualMachineClones should get synced ##### `enabled` boolean true [​](#kubeVirt-sync-virtualMachineClones-enabled "Direct link to kubeVirt-sync-virtualMachineClones-enabled")Enabled defines if this option should be enabled. #### `virtualMachinePools` object [​](#kubeVirt-sync-virtualMachinePools "Direct link to kubeVirt-sync-virtualMachinePools")If VirtualMachinePools should get synced ##### `enabled` boolean true [​](#kubeVirt-sync-virtualMachinePools-enabled "Direct link to kubeVirt-sync-virtualMachinePools-enabled")Enabled defines if this option should be enabled. --- # Metrics server Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** ### Install metrics server (inside vCluster)[​](#install-metrics-server-inside-vcluster "Direct link to Install metrics server (inside vCluster)") In case the above recommended method of getting metrics in vCluster using the metrics server proxy does not fulfill your requirements and you need a dedicated metrics server installation in the vCluster you can follow this section. Make sure the vCluster has access to the control plane cluster's nodes. Install the [metrics server](https://github.com/kubernetes-sigs/metrics-server#installation) using the official method into the vCluster. Wait until the metrics server has started. You should be now able to use `kubectl top pods` and `kubectl top nodes` within the vCluster: ``` kubectl top pods --all-namespaces NAMESPACE NAME CPU(cores) MEMORY(bytes) kube-system coredns-854c77959c-q5878 3m 17Mi kube-system metrics-server-5fbdc54f8c-fgrqk 0m 6Mi ``` Metrics server errors If you see the below error after installing metrics-server (check [this upstream issue](https://github.com/k3s-io/k3s/issues/5344) for more information): ``` loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503 ``` Create a file named `metrics_patch.yaml` with the following contents: ``` spec: template: spec: containers: - name: metrics-server command: - /metrics-server - --metric-resolution=30s - --kubelet-insecure-tls=true - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP ``` and apply the patch with kubectl: ``` kubectl patch deployment metrics-server --patch-file metrics_patch.yaml -n kube-system ``` ### How does it work?[​](#how-does-it-work "Direct link to How does it work?") By default, vCluster will create a service for each node that redirects incoming traffic from within the vCluster to the node kubelet to vCluster itself. This means that if workloads within the vCluster try to scrape node metrics the traffic reaches vCluster first. vCluster will redirect the incoming request to the control plane cluster, rewrite the response (pod names, pod namespaces etc) and return it to the requester. ### Reuse host metrics server[​](#reuse-host-metrics-server "Direct link to Reuse host metrics server") Create a `vcluster.yaml` with: ``` integrations: metricsServer: enabled: true nodes: true pods: true ``` Then create a vCluster with: ``` vcluster create host-metrics-vcluster -f vcluster.yaml ``` Then the metrics api should be available within the vCluster. This will obviously require a metrics server to be installed on the control plane cluster. If your metrics server is not using the default `kube-system/metrics-server` service, you can also configure a custom one via: ``` integrations: metricsServer: enabled: true apiService: service: name: my-custom-metrics-server namespace: my-custom-namespace port: 443 ``` If you are using [network policies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md), you need to add the metrics server pod to the allowed rules that the control plane can communicate with: ``` policies: networkPolicy: controlPlane: egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: 'kube-system' podSelector: matchLabels: k8s-app: metrics-server ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `metricsServer` object [​](#metricsServer "Direct link to metricsServer")MetricsServer reuses the metrics server from the host cluster within the vCluster. ### `enabled` boolean false [​](#metricsServer-enabled "Direct link to metricsServer-enabled")Enabled signals the metrics server integration should be enabled. ### `apiService` object [​](#metricsServer-apiService "Direct link to metricsServer-apiService")APIService holds information about where to find the metrics-server service. Defaults to metrics-server/kube-system. #### `service` object [​](#metricsServer-apiService-service "Direct link to metricsServer-apiService-service")Service is a reference to the service for the API server. ##### `name` string [​](#metricsServer-apiService-service-name "Direct link to metricsServer-apiService-service-name")Name is the name of the host service of the apiservice. ##### `namespace` string [​](#metricsServer-apiService-service-namespace "Direct link to metricsServer-apiService-service-namespace")Namespace is the name of the host service of the apiservice. ##### `port` integer [​](#metricsServer-apiService-service-port "Direct link to metricsServer-apiService-service-port")Port is the target port on the host service to connect to. ### `nodes` boolean true [​](#metricsServer-nodes "Direct link to metricsServer-nodes")Nodes defines if metrics-server nodes api should get proxied from host to virtual cluster. ### `pods` boolean true [​](#metricsServer-pods "Direct link to metricsServer-pods")Pods defines if metrics-server pods api should get proxied from host to virtual cluster. --- # Inference serving stacks Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** An inference serving stack deploys inside the tenant cluster. vCluster provides the tenant Kubernetes API, GPU access, resource sync, and templates for installing the stack. The stack can contain an execution engine, a distributed serving system, and an inference control plane. These components work together instead of acting as interchangeable alternatives. This page uses vLLM for the direct engine pattern. It also shows how KServe manages a vLLM-backed endpoint through an `InferenceService` resource. Deliver the serving stack declaratively with the [Argo CD integration](https://vcluster.com/docs/platform/integrations/argocd.md). Platform creates and manages the `ArgoCDApplication` object. Argo CD delivers the manifests from your Git repository into the tenant cluster. See [Build a direct engine template](#build-a-direct-engine-template) for the direct pattern. note This page focuses on the private-node endpoint pattern, where the tenant cluster owns its worker-node software stack. Provider-owned shared serving pools can also run behind a product API, but they need a different tenancy and routing design. ## Integration map[​](#integration-map "Direct link to Integration map") | Layer | Component | | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tenant Kubernetes API, object lifecycle, and sync | vCluster | | GPU node provisioning and reclaim | Private Nodes, Auto Nodes, and optionally vMetal | | GPU driver, device plugin, GPU Operator, MIG, vGPU, or Dynamic Resource Allocation | Node image and vendor components | | Model execution and serving | vLLM, SGLang, NVIDIA Triton, or a provider-owned model server | | Distributed serving and request orchestration | [Ray Serve](https://docs.ray.io/en/latest/serve/llm/architecture/overview.html) or [NVIDIA Dynamo](https://docs.nvidia.com/dynamo/design-docs/overall-architecture) | | Kubernetes inference control plane | [KServe](https://kserve.github.io/website/docs/concepts/architecture/control-plane) or another controller that manages serving resources | | External endpoint routing | Gateway API, Ingress, Knative networking, a service mesh, or a provider traffic layer | | Product API and customer lifecycle | Your inference provider platform | The layer names describe each project's primary role. Some projects span more than one layer. For example, Ray Serve provides ingress, routing, and autoscaling around engines such as vLLM or SGLang. KServe manages the lifecycle of `InferenceService` and `LLMInferenceService` resources. Its data plane can use vLLM, NVIDIA Triton, or another supported model server. ## Choose a deployment pattern[​](#choose-a-deployment-pattern "Direct link to Choose a deployment pattern") Choose which component owns the generated workloads, routes, and autoscalers before you build the tenant cluster template. | Pattern | Resources you declare | Resources the serving system manages | | -------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | Direct engine | Deployment, Service, route, and autoscaler | The engine manages model execution inside its pods. | | Distributed serving | Resources required by Ray Serve, NVIDIA Dynamo, or another serving system | The serving system manages its workers, request routing, and distributed execution. | | KServe Standard mode | `InferenceService` or `LLMInferenceService` | KServe creates workloads, Services, routing resources, and configured autoscalers. | | KServe Knative mode | `InferenceService` | KServe and Knative manage revisions, request routing, activation, and request-driven autoscaling. | Production stacks often combine these patterns. Ray Serve can use vLLM or SGLang as its engine. NVIDIA Dynamo supports multiple execution backends. KServe can deploy a vLLM-backed `InferenceService`. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * A tenant cluster with [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) enabled. * At least one GPU node attached to the tenant cluster. * A vendor GPU Operator, device plugin, or Dynamic Resource Allocation driver installed where your GPU stack requires it. * An endpoint networking implementation. Direct deployments need a Gateway API controller, Ingress controller, or provider traffic layer. KServe needs the networking layer for its selected deployment mode. * A model source, registry credential, or object storage path your runtime can reach. * Enough local disk on each GPU node for the runtime image and model weights. Runtime images alone commonly run tens of GB once extracted, and model weights, engine caches, and any additional models you load later share that same disk. * For production delivery, an [Argo CD connector](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md) enabled on the tenant cluster and a Git repository holding your runtime manifests. For GPU setup details, see [GPU and accelerator support](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md). For bare metal GPU node provisioning, see the [vMetal GPU Quickstart](https://www.vmetal.ai/docs/deploy/gpu-quickstart). ## Preflight the tenant cluster[​](#preflight-the-tenant-cluster "Direct link to Preflight the tenant cluster") Before installing the serving stack, confirm the tenant cluster can see GPU capacity and any routing resources your selected pattern expects. Check the GPU resource advertised by the private node: ``` kubectl get nodes -o 'custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu' kubectl describe node | grep -A5 Allocatable ``` For AMD or another accelerator vendor, replace `nvidia.com/gpu` with the resource name exposed by that vendor's device plugin or DRA driver. If you installed the NVIDIA GPU Operator, confirm its validator pod reached `Completed`. That means the driver stack reconciled: ``` kubectl get pods -n gpu-operator -l app=nvidia-cuda-validator ``` The smoke test below proves a real workload can actually use it. Surface-level driver checks can mask gaps that only show up under real inference traffic. For a direct deployment that uses an imported Gateway, confirm the tenant cluster can see it before creating routes: ``` kubectl get gateway -A kubectl describe gateway shared-inference -n shared-gateways ``` If the Gateway is missing, fix the tenant cluster template before deploying the runtime. For imported Gateways, the template must enable `sync.fromHost.gateways` and `sync.toHost.gatewayApi.httpRoutes`. For tenant-owned Gateways, the template must enable the Gateway API sync resources your controller requires. ## Deploy a direct vLLM smoke test[​](#deploy-a-direct-vllm-smoke-test "Direct link to Deploy a direct vLLM smoke test") The following example shows the direct engine pattern. It uses a small vLLM Deployment and Service to validate GPU scheduling, model loading, and in-cluster serving. Treat it as a smoke test, not as a production endpoint template. For production delivery, see [Build a direct engine template](#build-a-direct-engine-template). vllm-inference.yaml ``` apiVersion: apps/v1 kind: Deployment metadata: name: vllm-inference namespace: inference spec: replicas: 1 selector: matchLabels: app: vllm-inference template: metadata: labels: app: vllm-inference spec: terminationGracePeriodSeconds: 60 containers: - name: vllm image: vllm/vllm-openai:v0.8.5 imagePullPolicy: IfNotPresent args: - --model - facebook/opt-125m - --host - 0.0.0.0 - --port - "8000" env: - name: HF_HOME value: /models/cache ports: - name: http containerPort: 8000 readinessProbe: httpGet: path: /health port: http initialDelaySeconds: 30 periodSeconds: 10 startupProbe: httpGet: path: /health port: http failureThreshold: 60 periodSeconds: 10 resources: requests: cpu: "4" memory: 16Gi limits: cpu: "8" memory: 32Gi nvidia.com/gpu: "1" volumeMounts: - name: model-cache mountPath: /models/cache volumes: - name: model-cache emptyDir: {} --- apiVersion: v1 kind: Service metadata: name: vllm-inference namespace: inference labels: app: vllm-inference spec: selector: app: vllm-inference ports: - name: http port: 80 targetPort: http ``` Note the `--host 0.0.0.0` argument. The runtime's HTTP server must bind to all interfaces, not a loopback address, so the Service can reach it. vLLM defaults to this in its CLI. Adapting [Ray Serve](https://www.vcluster.com/blog/run-ray-serve-nvcf-function-vcluster-tenant-cluster) or another runtime to this pattern may require setting it explicitly. Apply the manifest inside the tenant cluster: ``` kubectl create namespace inference kubectl apply -f vllm-inference.yaml kubectl rollout status deployment/vllm-inference -n inference ``` Validate the Service from inside the tenant cluster: ``` kubectl port-forward svc/vllm-inference 8000:80 -n inference curl http://localhost:8000/v1/models ``` If something doesn't come up cleanly: * **Pod stays `Pending`:** check node capacity, GPU resource names, project quotas, allowed node types, node selectors, and taints. * **Pod stays in `ContainerCreating` for several minutes:** that's likely the first pull of a multi-gigabyte runtime image on a fresh node. Later pods on that same node start fast once the image is cached. * **Pod starts but the model does not load:** check model registry credentials, outbound network policy, model cache storage, and runtime logs. ## Plan model storage[​](#plan-model-storage "Direct link to Plan model storage") Large models make storage and startup behavior part of the endpoint architecture. Decide where model weights live, how access credentials are managed, how weights are cached, and how endpoint readiness behaves while the runtime loads them. Common patterns include: | Pattern | Use when | Tradeoff | | ------------------------------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------ | | Object storage, such as S3, GCS, Azure Blob, or MinIO | Models are shared across regions, endpoints, or customers and downloaded at startup. | Simple source of truth, but large models can make cold starts slow unless you add cache warming. | | PersistentVolume model cache | A tenant or endpoint repeatedly loads the same model. | Reduces repeated downloads, but requires capacity planning, cleanup, and access controls. | | Local NVMe or node image cache | You sell low-latency tiers for a small set of popular models. | Fastest startup after scheduling, but ties models to node images or node-local cache lifecycle. | | Shared filesystem, such as NFS or a parallel filesystem | Multiple replicas need a shared model store inside the same environment. | Avoids repeated downloads, but can become a throughput bottleneck for large model loads. | | Runtime image with model baked in | Small, fixed models change rarely. | Simple deployment, but large images slow pulls and make model rollout the same as image rollout. | For a production engine Application, make the model source, model version, credentials secret, cache volume, and cache size explicit `ArgoCDApplicationTemplate` parameters. The engine should mount registry or object-storage credentials. Prepare the cache with an init container or sidecar when needed. Report readiness only after the model is loaded. For large models, plan warmup separately from pod scheduling. A pod can be `Running` while the endpoint is still downloading weights, building a cache, compiling kernels, or loading the model into GPU memory. Surface that state in your product API. ## Build a direct engine template[​](#build-a-direct-engine-template "Direct link to Build a direct engine template") Package the engine as an Argo CD Application so a template update or Git commit rolls out consistently across each applicable tenant cluster. Store the Deployment, Service, and route in a Helm chart. Then declare an `ArgoCDApplicationTemplate` that points at it. Keep model identity and customer-specific values as parameters so the same template serves every endpoint tier. The `source.helm.values` block below references each parameter with Go template syntax (`{{ .Values.parameterName }}`). This is the same parameter mechanism [templates use for tenant clusters](https://vcluster.com/docs/platform/administer/templates/parameters.md), applied here to the Helm values that render the chart. ``` apiVersion: management.loft.sh/v1 kind: ArgoCDApplicationTemplate metadata: name: vllm-inference spec: template: spec: source: repoURL: "https://github.com/acme/inference-runtimes" targetRevision: main path: "runtimes/vllm-inference" # Helm chart helm: values: | model: "{{ .Values.model }}" cacheSize: "{{ .Values.cacheSize }}" destination: namespace: "{{ .Values.namespace }}" project: default syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ``` Set `CreateNamespace=true` so Argo CD creates the target namespace itself. That option only takes effect when the Application also sets `spec.destination.namespace`, so the template sets it from the `namespace` parameter. This is easy to miss: `target: vcluster` resolves the destination cluster, but it doesn't set the namespace on its own. Without `spec.destination.namespace`, the Application fails to sync into a namespace that doesn't already exist, unlike the smoke test above, where `kubectl create namespace` handled that step manually. Reference the template and pass its parameter values from the tenant cluster's `vcluster.yaml`, alongside the [Argo CD connector](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md): vcluster.yaml ``` integrations: argoCD: enabled: true connector: argocd-main deploy: argoCD: applications: - name: vllm-inference target: vcluster template: name: vllm-inference parameters: model: facebook/opt-125m cacheSize: 20Gi namespace: inference ``` Argo CD syncs the Application into the tenant cluster once the connector registers it. `selfHeal: true` reverts manual drift, and a Git commit to the template path rolls out on the next reconcile. A production direct engine Application should package: * pinned runtime images and a rollout policy * model registry, object storage, or artifact repository credentials * persistent or pre-warmed model cache storage * model source, model version, cache size, and credential parameters * resource requests and limits for CPU, memory, GPU, and ephemeral storage * node selectors, affinities, tolerations, or runtime classes that match the endpoint tier * readiness, startup, and liveness probes tuned to model load time * PodDisruptionBudget and graceful shutdown for draining traffic * NetworkPolicies and service account permissions * metrics scraping, logs, traces, and runtime-specific dashboards * authentication and authorization at the provider traffic layer * route, DNS, and certificate policy Use a CNI that enforces NetworkPolicy, such as Calico or Cilium. Otherwise a NetworkPolicy that looks correct can silently do nothing. Keep driver configuration, GPU presentation mode, and node image details in the GPU stack and node type. Don't embed them into the engine Application. ## Deploy through an inference control plane[​](#deploy-through-an-inference-control-plane "Direct link to Deploy through an inference control plane") Use KServe when you want a Kubernetes inference control plane to own endpoint lifecycle. KServe reconciles an `InferenceService` into the model-serving workload and its supporting resources. Install the KServe controllers, custom resource definitions, and serving runtimes before applying an `InferenceService`. For Standard mode, configure KServe with Gateway API or an Ingress controller. The network controller remains a platform prerequisite even when KServe creates each endpoint's routing resource. See the [KServe Standard mode requirements](https://kserve.github.io/website/docs/admin-guide/kubernetes-deployment). This example follows KServe's [text generation pattern](https://kserve.github.io/website/docs/model-serving/generative-inference/tasks/text-generation). It selects Standard mode and uses the Hugging Face serving runtime. That runtime uses vLLM as its default backend for supported generative models: kserve-vllm.yaml ``` apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: name: qwen-vllm namespace: inference annotations: # Use Kubernetes Deployments instead of Knative revisions. serving.kserve.io/deploymentMode: "Standard" spec: predictor: model: # KServe's Hugging Face runtime uses vLLM for supported models. modelFormat: name: huggingface storageUri: hf://Qwen/Qwen2.5-0.5B-Instruct args: - --model_name=qwen resources: requests: cpu: "2" memory: 6Gi nvidia.com/gpu: "1" limits: cpu: "4" memory: 12Gi nvidia.com/gpu: "1" minReplicas: 1 maxReplicas: 4 ``` Apply the resource and wait for KServe to report it ready: ``` kubectl create namespace inference --dry-run=client -o yaml | kubectl apply -f - kubectl apply -f kserve-vllm.yaml kubectl wait --for=condition=Ready inferenceservice/qwen-vllm \ --namespace inference \ --timeout=15m kubectl get inferenceservice qwen-vllm -n inference ``` KServe creates the Deployment, Service, routing resource, and Horizontal Pod Autoscaler for this Standard-mode endpoint. Don't add a second copy of those resources to the engine chart. When multiple endpoints share one KServe control plane, install KServe once. Gate endpoint delivery until its custom resource definitions are established and its controller is ready. For a self-contained Application, place the KServe installation and `InferenceService` in the same Application. Use sync waves to order the controller before the endpoint resource. For advanced large language model deployments, KServe also provides `LLMInferenceService`. It supports capabilities such as prefix-aware routing and disaggregated prefill and decode. ## Sequence the GPU stack and the serving stack[​](#sequence-the-gpu-stack-and-the-serving-stack "Direct link to Sequence the GPU stack and the serving stack") Argo CD has no native way to make one Application wait for another. If the GPU stack is separate from the serving stack, Argo CD syncs both without ordering them. For this specific pair, that is usually safe to leave unordered. A runtime pod requesting `nvidia.com/gpu` before the device plugin advertises it just stays `Pending`. Argo CD's `selfHeal` and the Kubernetes scheduler's own retry logic both converge once the GPU stack is ready, so the pod waits rather than fails. Treat a custom resource and its controller as a hard dependency. For example, an `InferenceService` needs the KServe custom resource definitions and controller before Argo CD applies it. With vCluster Platform 4.12, use a [Stack](https://vcluster.com/docs/platform/understand/what-are-stacks.md) when separate Apps or Argo CD Applications need explicit ordering. Stack tasks can wait for healthy dependencies and pass outputs to later tasks. See [Create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md). When you use Argo CD without a Stack, put hard dependencies in one Application. Use `argocd.argoproj.io/sync-wave` annotations to order resources inside it. Sync waves don't order separate Applications. Plan for minutes, not seconds, for the sync itself. Argo CD's reconcile interval defaults to a few minutes and is configurable. If you don't trigger syncs explicitly, a template change reaches each tenant cluster on its next scheduled sync rather than instantly. A freshly created tenant cluster can take a while to reach fully `Healthy` across its Applications, depending on connector registration and image pull time. For template guidance, see [Deploy applications](https://vcluster.com/docs/platform/integrations/argocd/deploy-applications.md), [Templates](https://vcluster.com/docs/platform/administer/templates/overview.md), [Quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md), and [Allowed node types](https://vcluster.com/docs/platform/administer/projects/allowed/node-types.md). ## Expose an inference endpoint[​](#expose-an-inference-endpoint "Direct link to Expose an inference endpoint") Routing ownership depends on the deployment pattern. Don't create a standalone route when a serving control plane already creates one. ### Route a direct engine deployment[​](#route-a-direct-engine-deployment "Direct link to Route a direct engine deployment") For new direct HTTP endpoint deployments, prefer Gateway API. The most common provider model is: 1. The platform team owns Gateway API CRDs, the Gateway controller, DNS, certificates, and shared `Gateway` resources in the control plane cluster. 2. The tenant cluster imports approved Gateways with `sync.fromHost.gateways`. 3. The runtime's Argo CD Application creates `HTTPRoute` resources in the tenant cluster alongside the Deployment and Service. 4. vCluster syncs the tenant route to the control plane cluster and enforces the Gateway attachment policy. The tenant cluster template needs Gateway sync enabled. The following example maps a platform-owned Gateway to a tenant-facing Gateway and allows routes from selected tenant namespaces: vcluster.yaml ``` sync: fromHost: gatewayClasses: enabled: true selector: matchLabels: inference.example.com/sync: "yes" gateways: enabled: true selector: matchLabels: inference.example.com/sync: "yes" mappings: byName: "platform-gateways/public-inference": "shared-gateways/shared-inference" allowedRoutes: overrides: - hostNamespace: platform-gateways name: public-inference allowedHostnames: - "*.inference.example.com" virtualNamespacePolicy: from: Selector selector: matchLabels: inference.example.com/route-access: "allowed" toHost: gatewayApi: httpRoutes: enabled: true ``` Label the namespace that may attach routes: ``` kubectl label namespace inference inference.example.com/route-access=allowed ``` For the full Gateway API model, see [Gateway API](https://vcluster.com/docs/vcluster/key-features/gateway-api.md), [Gateway API sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md), and [imported Gateways and GatewayClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/gateways.md). Add an `HTTPRoute` for the endpoint: vllm-route.yaml ``` apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: vllm-inference namespace: inference spec: parentRefs: - name: shared-inference namespace: shared-gateways hostnames: - customer-a.inference.example.com rules: - backendRefs: - name: vllm-inference port: 80 ``` Add the route to the same Git path as the Deployment and Service. The direct engine Application then delivers all three together. This approach avoids relying on Argo CD to sequence a separate route Application after the backend exists. Once Argo CD syncs the Application, confirm the route from inside the tenant cluster: ``` kubectl describe httproute vllm-inference -n inference ``` Look for `Accepted=True` and `ResolvedRefs=True` conditions. If the route fails to become ready, check the imported Gateway, the allowed hostname list, listener policy, and Gateway API sync configuration. After DNS points to the Gateway address, validate the endpoint externally: ``` curl -H "Host: customer-a.inference.example.com" http:///v1/models ``` In production, handle TLS and enforce customer authentication in your provider traffic layer, API gateway, service mesh, or runtime sidecar. Do not expose unauthenticated model endpoints directly to the public internet. ### Route a control-plane-managed endpoint[​](#route-a-control-plane-managed-endpoint "Direct link to Route a control-plane-managed endpoint") In Standard mode, KServe creates an HTTPRoute or Ingress for each `InferenceService`. The platform team still provides the underlying network controller and configures the Gateway or Ingress class that KServe uses. If the network controller runs in the control plane cluster, configure vCluster to sync the generated route. KServe owns the route lifecycle, while vCluster transports the resource between clusters. See [Gateway API sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md). KServe can create a default Gateway during installation. A provider can instead configure KServe to use a shared Gateway. In both cases, keep shared listener, DNS, certificate, and authentication policy in the platform traffic layer. Don't add the direct engine `HTTPRoute` from the previous section to a KServe-managed endpoint. Check the URL and generated networking resource through the `InferenceService` status: ``` kubectl get inferenceservice qwen-vllm -n inference \ -o jsonpath='{.status.url}{"\n"}' kubectl get httproute,ingress -n inference ``` Standard mode doesn't provide request-triggered scale-from-zero for HTTP endpoints. KServe Knative mode follows a different path. Knative manages revisions, request routing, activation, and scale-to-zero. This mode requires Knative and a supported networking layer. KServe recommends Standard mode for generative inference. It positions [Knative mode](https://kserve.github.io/website/docs/admin-guide/serverless) primarily for predictive inference. Alternative: serverless routing For a serverless alternative that hands routing to NVIDIA's control plane instead of Gateway API, see [Run Ray Serve as an NVCF Function](https://www.vcluster.com/blog/run-ray-serve-nvcf-function-vcluster-tenant-cluster). ## Add autoscaling and observability[​](#add-autoscaling-and-observability "Direct link to Add autoscaling and observability") Start with runtime metrics and GPU hardware metrics. [GPU and inference autoscaling](https://vcluster.com/docs/vcluster/integrations/gpu-hpa-dcgm.md) shows how to expose NVIDIA DCGM metrics and model-serving metrics to HPA or KEDA. Choose one owner for each workload's autoscaler. For a direct Deployment, create the HPA or KEDA `ScaledObject` yourself. For KServe, configure autoscaling through the `InferenceService` instead of attaching a competing autoscaler. For inference workloads, GPU utilization alone is rarely enough. Add runtime metrics such as: * request concurrency * queue depth * time to first token * tokens per second * p95 and p99 latency * GPU memory usage * cache pressure Use these signals with Horizontal Pod Autoscaler, KEDA, a runtime-specific autoscaler, or your provider control plane. For the first production endpoint, verify that each signal is visible before enabling autoscaling: ``` kubectl top pods -n inference kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 kubectl logs deployment/vllm-inference -n inference ``` ## Validate provider readiness[​](#validate-provider-readiness "Direct link to Validate provider readiness") Before exposing the endpoint to customers, validate the full provider path: * Product automation creates or selects the project, template, quota, and allowed node type. * The tenant cluster reaches `Ready` and the private GPU node joins. * The serving stack's `ArgoCDApplication` reports `Synced` and `Healthy`. * The GPU device plugin, GPU Operator, or DRA driver exposes the expected resource. * The serving pods or workers schedule on the intended GPU node type. * The model loads from the approved source and reaches readiness. * The direct route or serving control plane reports its endpoint ready. * The endpoint accepts authorized traffic and rejects unauthorized traffic. * Logs, metrics, and alerts include endpoint, tenant, model, and node tier labels. * Delete and configured scale-down workflows drain traffic and reclaim GPU capacity. --- # CI testing with setup-vind The [`setup-vind`](https://github.com/loft-sh/setup-vind) GitHub Action provisions Kubernetes clusters via [vCluster's Docker driver (vind)](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/docker.md) for CI/CD testing. It replaces `setup-kind` with a full tenant cluster running in Docker, usable for end-to-end (e2e) tests, multi-cluster setups, and more. Compared to KinD, vind provides: * **No image loading** - vind uses the host Docker daemon directly, so there's no need to run `kind load docker-image` before tests. * **LoadBalancer support** - Kubernetes Services of type `LoadBalancer` work out of the box and are reachable within the CI runner. * **Full vCluster configuration** - the cluster supports any [vCluster configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml), including multi-node setups and VPN-based private nodes. The action handles the full lifecycle automatically: * **Main step**: installs the vCluster CLI, sets the Docker driver, creates the cluster, and waits for nodes to become ready. * **Post step**: exports container logs as a GitHub artifact and deletes the cluster. ## Basic usage[​](#basic-usage "Direct link to Basic usage") ``` # .github/workflows/e2e.yaml name: E2E Tests on: pull_request: branches: [main] jobs: e2e: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: loft-sh/setup-vind@v1 - run: kubectl get nodes - run: make e2e ``` tip After the `setup-vind` step completes, `kubectl` is configured and the cluster is ready to use. The action waits up to 5 minutes for nodes to register before proceeding. ## Inputs[​](#inputs "Direct link to Inputs") | Input | Default | Description | | ----------------------- | -------- | --------------------------------------------------- | | `version` | `latest` | vCluster CLI version to install | | `name` | `vind` | Cluster name | | `config` | | Path to `vcluster.yaml` values file (repo-relative) | | `kubernetes-version` | | Kubernetes version for the cluster | | `skipClusterDeletion` | `false` | Skip cluster deletion in post step | | `skipClusterLogsExport` | `false` | Skip log export in post step | ## Examples[​](#examples "Direct link to Examples") ### Specify a Kubernetes version[​](#specify-a-kubernetes-version "Direct link to Specify a Kubernetes version") Modify the following with your specific values to generate a copyable command: KUBERNETES\_VERSION1.36.0 ``` - uses: loft-sh/setup-vind@v1 with: kubernetes-version: "1.36.0" ``` ### Use a config file[​](#use-a-config-file "Direct link to Use a config file") Modify the following with your specific values to generate a copyable command: CONFIG\_PATHhack/vcluster.yaml ``` - uses: loft-sh/setup-vind@v1 with: config: hack/vcluster.yaml ``` ### Multi-cluster setup[​](#multi-cluster-setup "Direct link to Multi-cluster setup") Create multiple clusters in the same job by giving each a unique name: Modify the following with your specific values to generate a copyable command: CLUSTER\_NAME\_1platform CLUSTER\_NAME\_2agent ``` - uses: loft-sh/setup-vind@v1 with: name: platform - uses: loft-sh/setup-vind@v1 with: name: agent ``` ## Migrate from setup-kind[​](#migrate-from-setup-kind "Direct link to Migrate from setup-kind") The following table shows how `setup-kind` inputs map to `setup-vind`. Version numbers are illustrative. | `setup-kind` | `setup-vind` | Notes | | ----------------------------- | ------------------------------ | ----------------------------------------- | | `version: v0.30.0` | `version: v0.32.0` | vCluster CLI version, not KinD | | `image: kindest/node:v1.36.0` | `kubernetes-version: "1.36.0"` | No node image needed | | `config: kind.yaml` | `config: vcluster.yaml` | Different config format (vCluster values) | | `kind load docker-image` | *(not needed)* | vind runs in Docker directly | ## Requirements and limitations[​](#requirements-and-limitations "Direct link to Requirements and limitations") Linux runners only The action supports `linux/amd64` and `linux/arm64` runners. It does not run on Windows or macOS GitHub-hosted runners. * **Docker required** - the runner must have Docker installed (standard on `ubuntu-latest`). * **Kernel modules** - the action pre-loads `br_netfilter`, `overlay`, and `bridge` modules automatically. No user action needed. --- # Architecture vCluster provisions tenant clusters on your infrastructure or directly on bare metal. Two choices shape your deployment architecture: where the control plane runs, and how worker nodes are deployed. Together, these determine the isolation level each tenant receives. Every tenant cluster is built on a certified Kubernetes distribution. This is true whether it's delivered to tenants as managed Kubernetes directly or as the foundation for a Slurm, Ray, or inference cluster running on top. This page covers that shared control plane and worker-node architecture. ## Control plane[​](#control-plane "Direct link to Control plane") Every tenant cluster runs a dedicated control plane. The control plane manages all operations within the tenant cluster and is completely invisible to tenants. It contains: * A **Kubernetes API server**, the management interface for all API requests within the tenant cluster. * A **controller manager**, which maintains the state of Kubernetes resources like pods, ensuring they match the desired configuration. * A **data store**, which stores all API resources. By default, an embedded SQLite database is used, but you can [choose other data stores](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store.md) like [etcd](https://etcd.io/), MySQL, and PostgreSQL. * A **syncer**, which synchronizes resources between the tenant cluster and the underlying infrastructure. * A **scheduler**, an optional component for scheduling workloads. By default, vCluster reuses the Control Plane Cluster's scheduler to reduce resource usage. You can [enable the virtual scheduler](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/virtual-scheduler.md) when you need node labels, taints, drain operations, or custom scheduling behavior. All of these components run together in a single container within a StatefulSet pod. The API server, controller manager, data store, and syncer are one unified process. CoreDNS deploys as a separate pod in the same namespace. If you opt into external etcd, it runs as its own StatefulSet. On the underlying cluster, `kubectl get pods -n ` shows something like: ``` NAME READY STATUS AGE vcluster-0 1/1 Running 5m vcluster-coredns-6b9f8d6f6b-xk2p9 1/1 Running 5m ``` Where that control plane runs is your first architectural choice. ### On an existing Kubernetes cluster[​](#on-an-existing-kubernetes-cluster "Direct link to On an existing Kubernetes cluster") Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** The default control plane placement. This choice sets where the control plane runs, not how tenant workloads run. Worker nodes are a separate choice, covered in [Worker nodes](#worker-nodes). The tenant cluster control plane runs as a pod on a Control Plane Cluster you operate. vCluster deploys the control plane as a StatefulSet (default) or Deployment inside a dedicated namespace on that cluster. Tenant cluster control planes as pods on a Control Plane Cluster Each tenant cluster runs inside its own namespace on the Control Plane Cluster. A namespace contains exactly one tenant cluster. Nesting tenant clusters inside one another is also supported. All resources synchronized to the Control Plane Cluster namespace carry owner references back to the tenant cluster. Deleting the tenant cluster — or its namespace — removes all associated resources automatically. No orphaned resources remain. ### vCluster Standalone[​](#vcluster-standalone "Direct link to vCluster Standalone") vCluster Standalone is a complete, zero-dependency Kubernetes distribution. It runs as a self-contained binary on bare metal or VMs, with no dependency on any other Kubernetes distribution. Once deployed, it behaves like any other Kubernetes cluster. You can install vCluster Platform on top of it, deploy tenant clusters, and join additional nodes, exactly as you would with any other cluster. For AI cloud providers standing up infrastructure from scratch, Standalone solves the "cluster one" problem. You can bootstrap your entire platform using only vCluster tooling, with no external dependencies. vCluster Standalone control planes as binaries on dedicated VMs ### vind (vCluster in Docker)[​](#vind-vcluster-in-docker "Direct link to vind (vCluster in Docker)") vind runs a complete tenant cluster entirely in Docker containers, with no Kubernetes dependency. The control plane and worker nodes run as containers on a single Docker host. vind always uses private nodes — worker nodes are Docker containers provisioned automatically. vind is suited for local development and CI/CD pipelines. It also supports hybrid deployments: external cloud nodes can join a vind cluster through a VPN, combining a local control plane with remote compute. Key capabilities that distinguish vind from KinD: * **Sleep and wake** — pause a cluster to free resources and resume it instantly, without deleting and recreating it. * **Load balancers out of the box** — Services of type `LoadBalancer` work automatically with no extra setup. * **Pull-through image cache** — vind uses the host Docker daemon's image cache directly, so there is no need to run `kind load docker-image` before tests. * **External nodes** — cloud instances can join a vind cluster through a VPN for hybrid development scenarios. | Feature | vind | KinD | | -------------- | -------------------------- | ----------------------------- | | Sleep and wake | Yes | No — must delete and recreate | | Load balancers | Automatic | Manual setup required | | Image caching | Pull-through Docker daemon | Direct registry pulls | | External nodes | Yes, using VPN | Local only | | CNI/CSI choice | Yes | Limited | ## Worker nodes[​](#worker-nodes "Direct link to Worker nodes") Your second architectural choice is how worker nodes are deployed. This determines compute isolation between tenants. For a customer-facing or otherwise production offering, work through [Choose a worker node model](https://vcluster.com/docs/vcluster/production-guide/choose-worker-node-model.md) before deciding. ### Private nodes[​](#private-nodes "Direct link to Private nodes") Each tenant cluster runs on dedicated nodes that join exclusively through a token-based process. Each tenant gets its own CNI, CSI, and compute. No cross-tenant visibility exists at the infrastructure level. From the tenant's perspective, the environment is indistinguishable from a single-tenant Kubernetes cluster. ![Architecture for Private Nodes](/docs/assets/images/private-nodes-571195ae75686af494ef4ce2fd0167e9.png) Architecture for Private Nodes Nodes can come from any Linux infrastructure: bare metal servers, cloud VMs, or any manually joined Linux machine. With [Auto Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/auto-nodes.md), vCluster Platform can provision and deprovision nodes automatically. This model suits GPU tenants, regulated workloads, and any environment where full infrastructure separation is required. ### Shared nodes[​](#shared-nodes "Direct link to Shared nodes") Tenant workloads run on the same cluster that hosts the control planes, sharing the existing node pool. Shared nodes are the vCluster OSS default and the only worker node model OSS provides. Multiple tenant clusters run side by side on the same physical nodes. A [syncer](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync) component translates workload resources from each tenant cluster into a dedicated namespace on the underlying cluster. Each tenant sees only their own namespaces, pods, and services. The translated copies are invisible to them. ![Architecture for Shared Nodes](/docs/assets/images/shared-nodes-639cc240ce049a169be9a607a7cf52d8.png) Architecture for Shared Nodes This model suits trusted tenants such as development environments, CI/CD pipelines, testing, and internal platforms. Tenants share the same kernel and worker nodes. So shared nodes aren't a security boundary for untrusted tenants with cluster or scheduler access or arbitrary workload execution. Use private nodes for external, resale, regulated, or otherwise untrusted tenants who get that access. A provider-operated offering where you own all workloads, such as a SaaS product or inference API, can still use shared nodes. If you choose shared nodes for a trusted-tenant or provider-operated use case, follow the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) to configure the baseline. See [Choose a worker node model](https://vcluster.com/docs/vcluster/production-guide/choose-worker-node-model.md) if you haven't made this decision yet. For tenants that need predictable per-tenant compute without full Private Nodes, you can scope workloads to a labeled node pool using [Dedicated Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes/isolated-workloads.md). Node selectors enforce placement on a reserved subset of the underlying cluster's nodes without provisioning separate infrastructure. A dedicated node pool is still the shared-node architecture. Tenants on that pool still share a kernel and worker nodes with each other. It scopes placement, not isolation, and doesn't provide private-node isolation guarantees. ### Worker node comparison[​](#worker-node-comparison "Direct link to Worker node comparison") | Feature | Private Nodes | Shared Nodes | vind (Docker) | | ------------------------------------------------- | ------------- | ------------ | ------------- | | Isolated control plane | Yes | Yes | Yes | | Suitable for untrusted or external tenants | Yes | No | No | | Custom CNI | Yes | No | Yes | | Custom CSI drivers | Yes | No | Limited | | Isolated network | Yes | No | Yes | | Reuse Control Plane Cluster controllers | No | Yes | No | | Tenant workloads visible on Control Plane Cluster | No | Yes | No | | Automatic node provisioning | Yes | No | Yes | | Requires Control Plane Cluster | Yes | Yes | No | | Suitable for local development | No | Limited | Yes | | Suitable for CI/CD | Limited | Yes | Yes | | Infrastructure overhead | High | Low | Low | See [Choose a worker node model](https://vcluster.com/docs/vcluster/production-guide/choose-worker-node-model.md) for the customer-access questions that decide which column applies to your offering. ## Virtual Nodes[​](#virtual-nodes "Direct link to Virtual Nodes") Virtual Nodes add a strong isolation boundary between tenant workloads at the node level. Using [vNode](https://www.vnode.com/docs/) to virtualize node boundaries, each tenant cluster gets its own view of the node environment. vNode enforces taints, tolerations, and scheduling boundaries per tenant. Virtual Nodes are not a separate deployment choice. They are an isolation layer that applies to any deployment model — shared nodes, private nodes, or Standalone. This makes them particularly valuable for GPU workloads where strict per-tenant separation is required regardless of how the underlying nodes are deployed. ## Syncer[​](#syncer "Direct link to Syncer") ![Syncer Architecture](/docs/assets/images/vcluster-architecture-ca6de4c1823051353b4ee33c25341b28.png) vCluster Architecture for the syncer The syncer links the tenant cluster to the Control Plane Cluster. It keeps the two environments consistent so tenants see a complete, functional Kubernetes cluster without direct access to the Control Plane Cluster. Its role depends on how worker nodes are deployed. **With private nodes**, only control plane state is synchronized. The syncer coordinates the tenant cluster's API server with the underlying infrastructure. From an operator's perspective this is nearly transparent. Workloads run directly on dedicated nodes with no sync overhead. **With shared nodes**, the syncer also translates workload resources between the tenant cluster and the underlying cluster. This includes Pods, ConfigMaps, Secrets, Services, Ingress, and Gateway API objects. Each resource type has specific sync behavior. Networking resources in particular require careful configuration. If you are deploying with shared nodes, read the [full sync documentation](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync) to understand what is synced and how to configure it for your workloads. ## Networking[​](#networking "Direct link to Networking") Networking within tenant clusters covers communication inside the tenant cluster, between tenant clusters, and between tenant clusters and the underlying cluster. note The networking features below apply to **shared nodes** deployments, where the syncer bridges tenant cluster resources to the underlying cluster. With private nodes, tenant workloads run on dedicated infrastructure and networking is fully isolated per tenant. #### Ingress traffic[​](#ingress-traffic "Direct link to Ingress traffic") Instead of running a separate Ingress controller in each tenant cluster, vCluster can [synchronize Ingress resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/ingresses.md) to use the underlying cluster's Ingress controller. This reduces per-tenant overhead and simplifies DNS management across tenant clusters. #### DNS[​](#dns "Direct link to DNS") By default, each tenant cluster deploys its own [CoreDNS](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/coredns.md) instance. CoreDNS lets pods within the tenant cluster resolve other services in the same environment. The syncer maps service DNS names in the tenant cluster to their corresponding IP addresses on the underlying cluster, following Kubernetes DNS naming conventions. As an Enterprise feature, [CoreDNS can be embedded](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/coredns.md) directly into the control plane pod, reducing the per-tenant footprint. #### Communication within a tenant cluster[​](#communication-within-a-tenant-cluster "Direct link to Communication within a tenant cluster") * **Pod to pod** — Pods within a tenant cluster communicate using the underlying cluster's network infrastructure. No additional configuration is required. The syncer ensures Kubernetes-standard networking is maintained. * **Pod to service** — Services within a tenant cluster are synchronized to allow pod communication, using [CoreDNS](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/coredns.md) for domain name resolution. * **Pod to underlying cluster service** — Services from the underlying cluster can be [replicated](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking/replicate-services.md) into a tenant cluster, allowing tenant pods to access platform services by name. * **Pod to another tenant cluster's service** — Services can be mapped [between tenant clusters](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking/resolve-dns.md) using DNS configuration that routes requests from one tenant cluster to another. #### Communication from the underlying cluster[​](#communication-from-the-underlying-cluster "Direct link to Communication from the underlying cluster") * **Underlying cluster pod to tenant cluster service** — Pods on the underlying cluster can access services within a tenant cluster by [replicating](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking/replicate-services.md#tenant-cluster-to-control-plane-cluster) tenant cluster services to any namespace on the underlying cluster. --- # GPU and accelerator support | | | Enterprise | | | | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [DRA Sync](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md#dynamic-resource-allocation-and-device-classes) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). vCluster supports GPU and accelerator workloads when the node exposes those devices through standard Kubernetes mechanisms. vCluster does not configure the physical GPU, install the vendor driver, or choose the device presentation mode. The node image, operating system, and vendor [device plugin](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) or [Dynamic Resource Allocation](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/) driver own that layer. From the tenant cluster's perspective, GPU workloads use the same Kubernetes APIs they would use on a regular cluster: * Extended resources such as `nvidia.com/gpu` or `amd.com/gpu`. * Vendor device plugins, such as the NVIDIA device plugin, AMD GPU device plugin, or an accelerator vendor's equivalent plugin. * GPU Operators, when the vendor provides one. * Dynamic Resource Allocation (DRA) objects such as `DeviceClass`, `ResourceClaim`, and `ResourceClaimTemplate`. * Optional higher-level schedulers or platforms, such as NVIDIA KAI Scheduler, NVIDIA Run:ai, or Slurm integrations. ## vCluster role[​](#vcluster-role "Direct link to vCluster role") vCluster provides the tenant Kubernetes control plane and syncs the Kubernetes objects that workloads need. It also lets tenants run isolated clusters on shared or private worker nodes. It does not sit in the device path between a pod and the GPU. This means: * If a node advertises `nvidia.com/gpu`, a tenant workload can request `nvidia.com/gpu`. * If a node advertises `amd.com/gpu`, a tenant workload can request `amd.com/gpu`. * If an accelerator vendor exposes a Kubernetes device plugin or DRA driver, vCluster can work with that driver's resources and DRA objects. * If the required driver, runtime configuration, device plugin, or DRA driver is missing from the node or tenant cluster, vCluster cannot make the device appear by itself. For private nodes, each tenant cluster can run its own GPU Operator, device plugin, DRA driver, scheduler, and accelerator CRDs. This is the common model for GPU cloud platforms because the tenant owns the full worker-node software stack. For shared host nodes, the device plugin and drivers usually run on the control plane cluster nodes. Tenant workloads can use the resources that the shared nodes advertise, subject to the sync and scheduling configuration. Review [GPU and device isolation](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#gpu-and-device-isolation) for the tenant-isolation gaps specific to shared GPU nodes. ## Install the NVIDIA GPU Operator in a tenant cluster[​](#install-the-nvidia-gpu-operator-in-a-tenant-cluster "Direct link to Install the NVIDIA GPU Operator in a tenant cluster") With [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md), the tenant cluster can install the NVIDIA GPU Operator directly because its workloads run on dedicated worker nodes. This is the common pattern for AI cloud and inference provider platforms: the tenant cluster owns the GPU Operator, device plugin, DCGM Exporter, MIG Manager, and related CRDs for its private node pool. note Install the GPU Operator inside the tenant cluster only when that tenant owns the GPU node software stack. On shared host nodes, the platform team usually installs the GPU driver stack and device plugin on the control plane cluster nodes instead. ### Before you install[​](#before-you-install "Direct link to Before you install") Prepare the private GPU nodes before installing the Operator: 1. Provision or join the GPU nodes to the tenant cluster. 2. Confirm the nodes run a supported Linux distribution, kernel, and container runtime. 3. Decide whether the Operator should install the NVIDIA driver or use a driver that is already installed in the node image. 4. Decide whether the tenant cluster needs MIG, NVIDIA vGPU, CDI, GPUDirect, or DCGM metrics. 5. Confirm `kubectl` and `helm` point at the tenant cluster, not the control plane cluster. For the full vendor matrix and chart options, see the [NVIDIA GPU Operator installation guide](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html). ### Install with Helm[​](#install-with-helm "Direct link to Install with Helm") Create the Operator namespace and label it for privileged workloads if your cluster uses Pod Security Admission: ``` kubectl create namespace gpu-operator kubectl label --overwrite namespace gpu-operator pod-security.kubernetes.io/enforce=privileged ``` Add the NVIDIA Helm repository: ``` helm repo add nvidia https://helm.ngc.nvidia.com/nvidia helm repo update ``` Install the Operator in the tenant cluster: ``` helm install gpu-operator nvidia/gpu-operator \ --namespace gpu-operator \ --wait ``` If your private node image already includes the NVIDIA driver, disable driver installation: ``` helm install gpu-operator nvidia/gpu-operator \ --namespace gpu-operator \ --wait \ --set driver.enabled=false ``` For inference endpoints, you can add Kubernetes pod labels such as endpoint, customer, or tier to DCGM metrics. Preserve the Operator values you selected during installation when you enable this optional enrichment: ``` helm upgrade gpu-operator nvidia/gpu-operator \ --namespace gpu-operator \ --wait \ --reuse-values \ --set dcgmExporter.enablePodLabels=true ``` DCGM Exporter's Kubernetes mapping provides the core `pod`, `namespace`, and `container` attribution through the kubelet pod-resources API. The `enablePodLabels` setting adds labels from the Kubernetes Pod object as extra Prometheus dimensions. ### Verify GPU availability[​](#verify-gpu-availability "Direct link to Verify GPU availability") Check the Operator pods: ``` kubectl get pods -n gpu-operator kubectl get clusterpolicy ``` Confirm the tenant cluster sees `nvidia.com/gpu` on the private GPU nodes: ``` kubectl get nodes -o 'custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu' ``` Run a CUDA smoke test: cuda-vectoradd.yaml ``` apiVersion: v1 kind: Pod metadata: name: cuda-vectoradd spec: restartPolicy: OnFailure containers: - name: cuda-vectoradd image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04 resources: limits: nvidia.com/gpu: 1 ``` Apply it and check the logs: ``` kubectl apply -f cuda-vectoradd.yaml kubectl logs pod/cuda-vectoradd ``` Expect output similar to this, ending in `Test PASSED`: ``` [Vector addition of 50000 elements] Copy input data from the host memory to the CUDA device CUDA kernel launch with 196 blocks of 256 threads Copy output data from the CUDA device to the host memory Test PASSED Done ``` Clean up the pod: ``` kubectl delete -f cuda-vectoradd.yaml ``` If the pod stays pending, check node readiness, taints, GPU resource names, project quotas, allowed node types, and whether the private node has joined the tenant cluster. If Operator pods fail, check the node OS, kernel, container runtime, driver installation mode, and the NVIDIA Operator logs. ## Supported vendors and accelerators[​](#supported-vendors-and-accelerators "Direct link to Supported vendors and accelerators") ### NVIDIA[​](#nvidia "Direct link to NVIDIA") NVIDIA GPUs commonly use the [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html) or the NVIDIA device plugin. The node advertises resources such as `nvidia.com/gpu`. Workloads request that resource in `resources.limits`. The node's driver and GPU Operator configuration control NVIDIA-specific modes such as [MIG](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html) or [NVIDIA vGPU](https://docs.nvidia.com/vgpu/latest/). vCluster consumes the resulting Kubernetes resources. It does not create MIG partitions or configure vGPU profiles. ### AMD[​](#amd "Direct link to AMD") AMD GPUs use the same Kubernetes mechanism. Install and configure the AMD driver stack and [AMD GPU device plugin](https://instinct.docs.amd.com/projects/k8s-device-plugin/en/latest/index.html), [AMD GPU Operator](https://instinct.docs.amd.com/projects/gpu-operator/en/main/device_plugin/device-plugin.html), or DRA driver. The node then advertises the AMD resource, commonly `amd.com/gpu`. Tenant workloads request that resource like any other Kubernetes extended resource. For DRA configuration, see [Dynamic resource allocation and device classes](#dynamic-resource-allocation-and-device-classes). ### Other accelerators[​](#other-accelerators "Direct link to Other accelerators") Other accelerators, such as SambaNova devices, FPGAs, DPUs, or custom AI accelerators, follow the same rule. If the vendor exposes the device to Kubernetes, vCluster can work with that Kubernetes-facing interface. Check the vendor documentation for the exact resource name, driver installation steps, and CRDs. Also confirm where the operator or controller should run. ## Dynamic resource allocation and device classes[​](#dynamic-resource-allocation-and-device-classes "Direct link to Dynamic resource allocation and device classes") Dynamic Resource Allocation is useful when workloads need more detail than a simple resource count. For example, workloads might need device attributes, capacity slices, or administrator-controlled device classes. DRA sync is disabled by default. To use DRA with shared host nodes, enable the settings your workload needs: * [`deviceClasses`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/device-classes.md) syncs allowed `DeviceClass` resources from the control plane cluster to the tenant cluster. * [`resourceClaims`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaims.md) syncs tenant-created `ResourceClaim` resources to the control plane cluster. * [`resourceClaimTemplates`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaimtemplates.md) syncs tenant-created `ResourceClaimTemplate` resources to the control plane cluster. Once `deviceClasses` sync is enabled, platform administrators create [`DeviceClass`](https://kubernetes.io/docs/reference/kubernetes-api/resource/device-class-v1/) resources on the control plane cluster and choose which classes are visible in each tenant cluster. For private nodes, tenants can also run the DRA driver and related controllers inside their tenant cluster when they own the worker-node software stack. ## Hardware presentation modes[​](#hardware-presentation-modes "Direct link to Hardware presentation modes") GPU presentation mode is determined before vCluster schedules a workload: | Mode | Where it is configured | vCluster role | | --------------------------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | Bare metal PCIe passthrough | Physical server, OS image, driver, and device plugin | Workloads request the advertised Kubernetes resource | | NVIDIA vGPU | NVIDIA vGPU host and guest driver stack, OS image, and operator or plugin configuration | Workloads request the resource exposed by that stack | | NVIDIA MIG | NVIDIA GPU Operator or device plugin configuration | Workloads request the MIG resources advertised by the plugin | | DRA device allocation | Vendor DRA driver and `DeviceClass` resources | Syncs allowed DRA objects between the control plane cluster and tenant cluster | If you provision physical GPU servers with vMetal, vMetal controls the bare metal lifecycle and node OS image. The OS image and post-provision configuration determine which GPU drivers, vGPU stack, MIG strategy, or vendor plugins are available. For that layer, see [GPU presentation modes in vMetal](https://www.vmetal.ai/docs/plan/gpu-presentation). ## Summary checklist[​](#summary-checklist "Direct link to Summary checklist") To make GPU or accelerator workloads work in a tenant cluster: 1. Prepare the worker node with the required firmware, OS image, kernel modules, and vendor driver stack. 2. Install the vendor device plugin, GPU Operator, or DRA driver in the right cluster. 3. Confirm the node advertises the expected resource or DRA devices. 4. Configure vCluster sync for any required CRDs, scheduler objects, or DRA objects. 5. Run a workload that requests the advertised resource name or references the synced `DeviceClass`. When several applications make up the GPU environment, use a [Stack](https://vcluster.com/docs/platform/understand/what-are-stacks.md) to deploy them repeatedly with dependency ordering and health gates. See [Create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) to package GPU operators, schedulers, monitoring, and dependent applications. For GPU bare metal provisioning and OS image guidance, see [vMetal GPU Quickstart](https://www.vmetal.ai/docs/deploy/gpu-quickstart). ## External references[​](#external-references "Direct link to External references") * [Kubernetes device plugins](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) * [Kubernetes Dynamic Resource Allocation](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/) * [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html) * [NVIDIA GPU Operator with MIG](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html) * [NVIDIA vGPU documentation](https://docs.nvidia.com/vgpu/latest/) * [AMD GPU device plugin for Kubernetes](https://instinct.docs.amd.com/projects/k8s-device-plugin/en/latest/index.html) --- # Building a GPU cloud platform AI cloud providers face a specific problem: enterprise AI teams expect dedicated Kubernetes clusters with full isolation, GPU Operator support, and the same operational behavior they get from hyperscaler Kubernetes services. Building and operating a real cluster per customer is expensive and operationally unsustainable. Sharing a single cluster across tenants introduces noisy neighbors and hard-to-enforce boundaries. vCluster solves this by virtualizing the Kubernetes control plane. Every tenant gets a dedicated API server, their own CRDs, RBAC, and a complete cluster experience. From the tenant's perspective it's indistinguishable from a real cluster. For the provider, it provisions in seconds and vCluster Platform manages it centrally, without the overhead of a physical cluster per customer. If your customers consume Kubernetes clusters directly, use the [AI Cloud production path](https://vcluster.com/docs/vcluster/production-guide/ai-cloud.md). If they consume managed model endpoints through your product API, use the [Inference Provider production path](https://vcluster.com/docs/vcluster/production-guide/inference-provider.md). ## How it fits into your platform[​](#how-it-fits-into-your-platform "Direct link to How it fits into your platform") A vCluster deployment has two layers: 1. A **Control Plane Cluster** that hosts the virtualized control planes for your tenant clusters. This can be an existing Kubernetes cluster, a purpose-built cluster, or a [vCluster Standalone](#bootstrap-with-vcluster-standalone) instance on your own infrastructure. 2. **Tenant clusters** that run on top of that infrastructure. Each tenant cluster has its own API server, controller manager, data store, and lifecycle. [vCluster Platform](https://vcluster.com/docs/platform/.md) sits above both layers. It handles provisioning, access control, lifecycle management, and node automation across all your tenant clusters and control plane clusters. ## Worker nodes for GPU workloads[​](#worker-nodes-for-gpu-workloads "Direct link to Worker nodes for GPU workloads") The worker node model determines how isolated each tenant's compute is. For customer-facing AI cloud workloads, private nodes are the model that provides full per-tenant infrastructure isolation. Shared infrastructure with per-tenant node pools is appropriate only for internal or trusted-tenant GPU platforms, not for externally facing or resold offerings. See [Choose a worker node model](https://vcluster.com/docs/vcluster/production-guide/choose-worker-node-model.md) for the underlying decision criteria. For details on NVIDIA, AMD, DRA, and other accelerator support, see [GPU and accelerator support](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md). ### Private nodes[​](#private-nodes "Direct link to Private nodes") Each tenant cluster runs on its own dedicated physical nodes, with a separate CNI, CSI, and control plane. No compute, network, or storage is shared. Tenants get full NCCL bandwidth and hardware-level separation. Private nodes are the right choice for: * Multi-node distributed training and large model inference * Customers requiring compliance or regulatory isolation * Workloads where performance predictability is non-negotiable [Deploy with private nodes →](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) ### Shared infrastructure with dedicated node pools[​](#shared-infrastructure-with-dedicated-node-pools "Direct link to Shared infrastructure with dedicated node pools") Each tenant cluster gets exclusive access to a labeled node pool on the control plane cluster. Compute is scoped per tenant without provisioning separate infrastructure. CNI, CSI, and platform services are shared. Compute and platform services are shared at the infrastructure level. This model is appropriate only for internal or trusted-tenant GPU platforms. An example is sharing a data center across your own organization's teams. Don't use it for externally facing or resold GPU offerings. For customer-facing GPU clouds, use private nodes. This model can fit: * Internal per-team GPU node pools with elastic scaling * Trusted internal teams that need compute separation but not full infrastructure isolation [Dedicated Nodes in the architecture overview →](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes) ### Add Virtual Nodes for stronger isolation[​](#add-virtual-nodes-for-stronger-isolation "Direct link to Add Virtual Nodes for stronger isolation") Virtual Nodes add a strong isolation boundary at the node level using [vNode](https://www.vnode.com/docs/). vNode enforces scheduling boundaries per tenant, giving each tenant its own view of the node environment. Virtual Nodes are not a deployment model. They are an optional isolation layer that applies to either worker node choice above (private or shared). This makes them valuable when strict per-tenant workload separation is required regardless of how the underlying nodes are deployed. [Learn about Virtual Nodes →](https://vcluster.com/docs/vcluster/introduction/architecture.md#virtual-nodes) ## Bootstrap with vCluster Standalone[​](#bootstrap-with-vcluster-standalone "Direct link to Bootstrap with vCluster Standalone") Every vCluster deployment needs a control plane cluster. For AI cloud providers standing up infrastructure from scratch, [vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md) is a natural choice. vCluster Standalone is a complete, zero-dependency Kubernetes distribution. It runs on bare metal or VMs with no dependency on any other Kubernetes distribution. Once deployed, it behaves like any Kubernetes cluster. You install vCluster Platform on top of it, deploy tenant clusters, and join bare metal worker nodes, all using vCluster tooling with no external dependencies. Combine Standalone with [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) to add accelerators from any supported infrastructure provider, whether that's your own bare metal or a cloud GPU instance joined into a tenant cluster. This solves the "cluster one" problem: you can bootstrap your entire platform stack from bare metal without adopting another Kubernetes distribution. [Deploy vCluster Standalone →](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md) ## Next steps[​](#next-steps "Direct link to Next steps") * [Architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md) — understand control plane deployment options, worker node models, and the syncer * [Build for Production](https://vcluster.com/docs/vcluster/production-guide/overview.md) — choose the path for managed Kubernetes, managed inference endpoints, internal AI platforms, or distributed GPU compute * [Inference Provider](https://vcluster.com/docs/vcluster/production-guide/inference-provider.md) — build managed model-serving endpoints on GPU infrastructure * [Building an inference platform](https://vcluster.com/docs/vcluster/introduction/inference-platform.md) — tenancy models, GPU capacity classes, and endpoint readiness for managed inference * [GPU and accelerator support](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md) — understand how vCluster works with NVIDIA, AMD, DRA, and other accelerator devices * [Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md) — standardize the applications that make up each tenant's GPU environment * [Certified Stacks](https://vcluster.com/docs/platform/integrations/certified-stacks/.md) — browse tested integrations bundled with vCluster Platform * [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) — deploy dedicated GPU infrastructure per tenant * [vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md) — bootstrap your control plane cluster on bare metal * [vCluster Platform](https://vcluster.com/docs/platform/.md) — centralized management, access control, and node automation * [vNode](https://www.vnode.com/docs/) — workload isolation layer for GPU tenants --- # Building an inference platform A managed inference platform serves models to customers through your product API and endpoints, not through direct Kubernetes access. This page covers the design concepts behind that platform. It explains how vCluster provides the tenant boundary, which tenancy model fits each customer tier, how to size GPU capacity, and what endpoint readiness actually depends on. For the step-by-step build and operations path, see the [Inference Provider production path](https://vcluster.com/docs/vcluster/production-guide/inference-provider.md). ## How vCluster enables inference[​](#how-vcluster-enables-inference "Direct link to How vCluster enables inference") An inference provider needs an isolated tenant boundary, GPU capacity, and an inference serving stack reachable over a public URL. vCluster and Platform implement the tenant boundary and GPU capacity orchestration. Platform can deliver components individually through the Argo CD integration or use a [Stack](https://vcluster.com/docs/platform/understand/what-are-stacks.md) when the GPU and serving components need dependency ordering, health gates, or output handoff. [Inference serving stacks](https://vcluster.com/docs/vcluster/integrations/model-serving.md) explains how execution engines, distributed serving systems, and inference control planes compose. It includes direct vLLM and KServe-managed endpoint patterns. [GPU and inference autoscaling](https://vcluster.com/docs/vcluster/integrations/gpu-hpa-dcgm.md) covers the HPA and KEDA examples. [vMetal inference fleet capacity](https://www.vmetal.ai/docs/operate/gpu-fleet#inference-provider-capacity) covers the bare metal layer behind Private Nodes. ## Choose an inference tenancy model[​](#choose-an-inference-tenancy-model "Direct link to Choose an inference tenancy model") Inference providers usually need more than one tenancy model. Use the model that matches the customer's trust level, performance expectation, and isolation requirement. | Model | Use when | Tradeoff | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Tenant cluster per customer | Customers need their own Kubernetes API boundary, custom resources, secrets, and lifecycle. | Strong control plane isolation without automatically dedicating every GPU node. | | Private GPU nodes | Customers need dedicated worker-node infrastructure, predictable performance, separate CNI/CSI, or compliance isolation. | Stronger isolation and performance predictability, with more capacity planning. | | vNode runtime isolation | Workloads need a stronger runtime boundary than standard containers, such as custom containers, adapters, or dynamic execution. | Adds a workload sandbox on top of your tenancy model. For untrusted or external customer code, use private nodes and add vNode, not shared nodes. | | Shared serving tier | Customers call provider-owned, trusted models and don't run their own containers, plugins, or models. | Highest utilization. Isolation is application level, not infrastructure level. Use it only for provider-owned models, never for customer-supplied workloads. | Tenant clusters are a strong default when each customer or endpoint tier needs its own Kubernetes API boundary, custom resources, secrets, lifecycle, or compliance requirements. Without that boundary, two customers can collide over competing CRDs in a shared control plane. For example, one tenant running KServe and another running a different serving framework's CRDs can conflict and affect other tenants. A single tenant cluster can host multiple serving endpoints, so one tenant cluster doesn't have to mean one model. An endpoint can use direct Kubernetes resources or an inference control plane such as KServe. Most inference providers default to tenant clusters with private GPU nodes for dedicated performance and isolation. Reserve the shared serving tier for trusted, provider-owned models where utilization matters more than isolation. ## Map the tenancy model to vcluster.yaml[​](#map-the-tenancy-model-to-vclusteryaml "Direct link to Map the tenancy model to vcluster.yaml") A tenant cluster per customer with dedicated GPU nodes starts with [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) enabled at creation: vcluster.yaml ``` privateNodes: enabled: true ``` Restrict which GPU node types a customer's project can claim with the project's `spec.allowedNodeTypes`, so an `l40s-shared` tier can't claim `h100-dedicated` capacity: ``` apiVersion: management.loft.sh/v1 kind: Project metadata: name: customer-h100-dedicated spec: allowedNodeTypes: - name: metal3.gpu-h100 ``` Deliver the runtime for that tenant cluster from the tenant cluster template by enabling the [Argo CD connector](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md) and referencing an `ArgoCDApplicationTemplate`: vcluster.yaml ``` integrations: argoCD: enabled: true connector: argocd-main deploy: argoCD: applications: - name: vllm-inference target: vcluster template: name: vllm-inference ``` For the shared serving tier, one tenant cluster hosts multiple provider-owned endpoints instead of dedicating nodes per customer. See [Inference serving stacks](https://vcluster.com/docs/vcluster/integrations/model-serving.md) for direct and control-plane-managed deployment patterns. When customers run their own inference code, use private nodes and add [vNode](https://www.vnode.com/docs/configuration/) for a stronger runtime boundary. ## Size GPU capacity classes[​](#size-gpu-capacity-classes "Direct link to Size GPU capacity classes") Define node types by GPU model, GPU count, region, provider, rack, and reservation model. Use [vMetal](https://www.vmetal.ai/docs/operate/gpu-fleet#inference-provider-capacity) for owned bare metal fleets, or cloud GPU instances and Auto Nodes for cloud capacity. For NVIDIA fleets that support MIG, use MIG to give external or SLA-backed tenants hardware-isolated GPU instances. MIG partitions memory and compute at the hardware level, so each tenant gets predictable, isolated performance. Offer a small set of standard MIG profiles per GPU model instead of fully custom sizes. Beyond MIG, keep these capacity boundaries in mind: * Keep training and inference on separate node pools, since co-locating them causes latency spikes for inference. * Keep the underlying driver, vGPU, and Dynamic Resource Allocation configuration in the vendor stack. ## Endpoint readiness[​](#endpoint-readiness "Direct link to Endpoint readiness") Endpoint readiness depends on more than Kubernetes pod readiness. A new endpoint might wait for: * Argo CD to sync the Application * GPU capacity and node bootstrap * GPU Operator reconciliation * Image pulls * Model download, model load, and cache warmup * Route attachment, DNS, and certificate issuance Plan for minutes, not seconds, for the Argo CD sync itself. The first image pull on a new GPU node is also slow for multi-gigabyte runtime images, though later endpoints scheduled on that same node start fast from the cache. See [Sequence the GPU stack and the serving stack](https://vcluster.com/docs/vcluster/integrations/model-serving.md#sequence-the-gpu-stack-and-the-serving-stack) for why the sync itself takes minutes rather than seconds. Model weight download and load time scales with model size, and for large models it can add more to cold start than the runtime image pull does. As with the image, endpoints that land on a node where the weights are already cached load faster. On bare metal, the wait for capacity is not instant. Provisioning a server includes BMC verification, hardware inspection, PXE boot, and OS install (the bare metal provisioning steps) before the node joins. See [vMetal inference fleet capacity](https://www.vmetal.ai/docs/operate/gpu-fleet#inference-provider-capacity) for capacity planning at the hardware layer. ## Integrate with your product control plane[​](#integrate-with-your-product-control-plane "Direct link to Integrate with your product control plane") Customers call your product API and inference endpoints, not Platform. Your product decides which actions to expose, such as endpoint creation, model selection, GPU tier selection, quota changes, and deletion. It then maps those actions to Platform and tenant resources. An endpoint tier such as `h100-dedicated` or `l40s-shared` maps to template parameters, node type constraints, and quota policy. An endpoint instance maps to a tenant cluster and serving resources. These resources can include an `InferenceService` or direct workload, Service, route, autoscaler, and metrics resources. Dedicated private-node tiers and shared serving pools need different metering granularity, since many endpoints can share the same nodes in a pool. For the full mapping pattern and metering approach, see [Build a product control plane on the Platform API](https://vcluster.com/docs/platform/how-to/product-control-plane.md). It covers how to map product concepts to Platform resources, drive the Platform API, and keep billing labels stable. ## Next steps[​](#next-steps "Direct link to Next steps") * [Inference Provider production path](https://vcluster.com/docs/vcluster/production-guide/inference-provider.md) — the Day 0, Day 1, and Day 2 build and operations path * [Build a product control plane on the Platform API](https://vcluster.com/docs/platform/how-to/product-control-plane.md) — map endpoints to Platform resources and meter usage * [What are Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md) — coordinate the applications that make up an inference environment * [Inference serving stacks](https://vcluster.com/docs/vcluster/integrations/model-serving.md) — direct engines, distributed serving systems, inference control planes, and routing * [GPU and inference autoscaling](https://vcluster.com/docs/vcluster/integrations/gpu-hpa-dcgm.md) — hardware and serving metric scaling * [GPU and accelerator support](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md) — NVIDIA, AMD, DRA, and other accelerator devices * [Architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md) — control plane deployment options and worker node models --- # Compare open source and free tiers vCluster is available as open source under the Apache 2.0 license and in several product tiers with additional features. This page explains the difference between the open source (OSS) and Free tiers, how licensing works, and how to activate the Free tier. ## Overview[​](#overview "Direct link to Overview") **vCluster Open Source** is the foundation of the project. It provides full-stack tenant cluster isolation with dedicated Kubernetes control planes, resource syncing, and multiple backing stores. OSS requires no license and no Platform connectivity. **vCluster Free tier** adds capabilities on top of OSS at no cost, including embedded etcd, Private Nodes, custom resource syncing, sync patches, and HostPath Mapper. The Free tier requires connecting to the vCluster Platform for license validation but needs no credit card. Beyond the Free tier, paid Enterprise tiers (Dev, Prod, Scale) add features like sleep mode, external databases, SSO, audit logging, FIPS compliance, and air-gapped deployments. For pricing details, see the [vCluster Pricing](https://www.vcluster.com/pricing) page. Sidebar badges Documentation pages show tier badges in the sidebar: **FREE** (black) requires Platform activation, **ENTERPRISE** (orange) requires a paid license. Pages without badges are OSS features available to everyone. ## Feature comparison by license plan[​](#feature-comparison-by-license-plan "Direct link to Feature comparison by license plan") This table compares the vCluster license plans, Free, Dev, Prod, and Scale. It does not show open source features, which do not need a license or a Platform connection. | | | Enterprise | | | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ------ | ------ | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Advanced UI Customizations](https://vcluster.com/docs/platform/configure/platform-configs/custom-branding.md#advanced-custom-branding) | | | | | | [Air-Gapped Mode](https://vcluster.com/docs/platform/install/air-gapped.md) | | Add-On | Add-On | Add-On | | [Apps](https://vcluster.com/docs/platform/understand/what-are-apps.md) | | | | | | [Argo Integration](https://vcluster.com/docs/platform/integrations/argocd.md) | | | | | | [Audit Logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md) | | | | | | [Auto Sleep](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md) | | | | | | [Auto Sleep for Namespaces](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/sleep-mode.md) | | | | | | [Auto Sleep for Virtual Clusters](https://vcluster.com/docs/platform/use-platform/virtual-clusters/key-features/sleep-mode.md) | | | | | | [Auto Snapshots](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/snapshots.md#enable-auto-snapshots) | | | | | | [Built-In CoreDNS](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/coredns.md) | | | | | | [Central HostPath Mapper](https://vcluster.com/docs/platform/maintenance/monitoring/central-hostpath-mapper) | | | | | | [Centralized Admission Control](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/admission-control.md) | | | | | | [Cert Manager Integration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/cert-manager.md) | | | | | | [Cluster Access](https://vcluster.com/docs/platform/administer/clusters/cluster-roles.md) | | | | | | [Cluster Role Management](https://vcluster.com/docs/platform/administer/clusters/cluster-roles.md) | | | | | | [ClusterAPI Node Provider](https://vcluster.com/docs/platform/api/resources/nodeprovider.md) | | | | | | [Connected Clusters](https://vcluster.com/docs/platform/administer/clusters/connect-cluster.md) | | | | | | [Custom Branding](https://vcluster.com/docs/platform/configure/platform-configs/custom-branding.md) | | | | | | [Custom Resource Syncing](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/custom-resources.md) | | | | | | [Database Connector](https://vcluster.com/docs/platform/administer/connector/database.md) | | | | | | [DRA Sync](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md#dynamic-resource-allocation-and-device-classes) | | | | | | [EKS Pod Identity for External Database Connections](https://vcluster.com/docs/platform/administer/connector/database.md) | | | | | | [Embedded etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/embedded.md) | | | | | | [External Database](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md) | | | | | | [External Database Kerberos Authentication](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md#authenticate-to-postgresql-with-kerberos-gssapi) | | | | | | [External Database RDS IAM Authentication](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md) | | | | | | [External Secrets Integration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/eso.md) | | | | | | [FIPS](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/fips.md) | | | | | | [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md) | | | | | | [HashiCorp Vault Integration](https://vcluster.com/docs/platform/integrations/hashicorp-vault.md) | | | | | | [High-Availability Mode](https://vcluster.com/docs/platform/maintenance/reliability-scaling/high-availability.md) | | | | | | [Hybrid Scheduling](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/pods/hybrid-scheduling.md) | | | | | | Isolated Control Plane | | | | | | [Istio Integration](https://vcluster.com/docs/vcluster/integrations/istio.md) | | | | | | Kube-vip Integration | | | | | | [KubeVirt Integration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/kubeVirt.md) | | | | | | [KubeVirt Node Provider](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md) | | | | | | [Least Privilege Mode](https://vcluster.com/docs/platform/configure/agent-settings/least-privilege-mode.md) | | | | | | [Machine Management](https://vcluster.com/docs/platform/use-platform/machines/overview.md) | | | | | | [Metal3 Integration](https://vcluster.com/docs/platform/administer/bare-metal/overview.md) | | | | | | [Metal3 Node Provider](https://vcluster.com/docs/platform/administer/node-providers/metal3.md) | | | | | | [Multi-Region Mode](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md) | | | | | | [Multi-Region Platform](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md) | | | | | | [Multiple SSO Providers](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md#multiple-sso-providers) | | | | | | [Namespace Management](https://vcluster.com/docs/platform/use-platform/namespaces/create/create-with-template.md) | | | | | | Netris | | | | | | NICo Node Provider | | | | | | [Node Profiles](https://vcluster.com/docs/platform/administer/node-profiles/overview.md) | | | | | | [Nvidia BCM Node Provider](https://vcluster.com/docs/platform/administer/node-providers/bcm.md) | | | | | | [Platform as OIDC Provider](https://vcluster.com/docs/platform/administer/authentication/oidc-provider.md) | | | | | | [Platform External Database](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md) | | | | | | [Private Nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes.md) | | | | | | [Private Nodes Auto Nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/auto-nodes.md) | | | | | | [Private Nodes VPN](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes.md) | | | | | | [Project Quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md) | | | | | | [Regional Cluster Endpoints](https://vcluster.com/docs/platform/administer/clusters/advanced/regional-cluster-endpoints.md) | | | | | | Resolve DNS | | | | | | [Secrets Sync](https://vcluster.com/docs/platform/use-platform/secrets/secret-sync.md) | | | | | | Security-Hardened vCluster Image | | | | | | [Single Sign-On](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md) | | | | | | [Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md) | | | | | | [Sync Namespaces toHost](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/namespaces.md) | | | | | | [Sync Patches](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) | | | | | | [Template Versioning](https://vcluster.com/docs/platform/administer/templates/versioning.md) | | | | | | [Terraform Node Provider](https://vcluster.com/docs/platform/administer/node-providers/terraform.md) | | | | | | Translate Patches | | | | | | vCluster Proxy Resources | | | | | | [Virtual Cluster Management](https://vcluster.com/docs/vcluster/introduction/what-are-virtual-clusters) | | | | | | [VM Management with KubeVirt](https://vcluster.com/docs/platform/administer/virtual-machines/overview.md) | | | | | | [vNode Runtime](https://www.vnode.com/docs) | | Add-On | Add-On | Add-On | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). ## How licensing works[​](#how-licensing-works "Direct link to How licensing works") The Free tier requires connecting your tenant cluster to the vCluster Platform for license validation. The Platform handles all license management automatically. You don't need to manually configure license keys. Tier-gated features need the Pro image Licensing has two parts. The license described below enables a feature, and the Pro image (`ghcr.io/loft-sh/vcluster-pro`, the chart default) contains the code that runs it. The open source image (`ghcr.io/loft-sh/vcluster-oss`) compiles these features out entirely. A Pro image without an attached license crashloops with a "vCluster pro feature" error that seems like a paywall but is really a configuration problem. See [Resolve Pro feature license errors](https://vcluster.com/docs/vcluster/troubleshoot/pro-feature-license-error.md). ### License validation flow[​](#license-validation-flow "Direct link to License validation flow") License validation happens at the Platform level, not within individual tenant clusters: 1. **Platform retrieves license**: The vCluster Platform periodically contacts the Loft License API (`admin.loft.sh`) to retrieve and validate its license. This determines which features are available across all connected tenant clusters. 2. **Tenant cluster connects to Platform**: Each tenant cluster authenticates with the Platform using an access key stored in a Kubernetes Secret. The Platform then provides license information to the tenant cluster. 3. **Features are enabled**: Based on the Platform's license tier (Free, Dev, Prod, or Scale), the tenant cluster enables the appropriate features. Network requirements The Platform requires outbound HTTPS access to `admin.loft.sh` for license retrieval. Tenant clusters only need connectivity to the Platform, not to the external license API. For air-gapped environments, see the [air-gapped installation guide](https://vcluster.com/docs/platform/install/air-gapped.md). ### What happens during validation[​](#what-happens-during-validation "Direct link to What happens during validation") When a tenant cluster starts or reconnects to the Platform: * The tenant cluster reads its access key from the configured Secret (defaults to `vcluster-platform-api-key` in the tenant cluster's namespace) * It establishes a secure connection to the Platform * The Platform validates the access key and returns the current license status * The tenant cluster enables or disables features based on the license tier For details on configuring the access key, see [Platform API Key configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/platform.md). ### Graceful degradation[​](#graceful-degradation "Direct link to Graceful degradation") note A tenant cluster that was previously licensed and then loses Platform connectivity does not stop functioning. In the worst case it falls back to the OSS feature set, ensuring your workloads continue running. This graceful fallback applies to a tenant cluster that has already been licensed. A Pro image that has never received a license Secret crashloops at startup instead. See [Resolve Pro feature license errors](https://vcluster.com/docs/vcluster/troubleshoot/pro-feature-license-error.md). If license validation fails: * **Temporary connectivity issues**: The tenant cluster continues operating with cached license information during brief outages. * **Expired or invalid license**: Features revert to the OSS tier. Your workloads continue running, but tier-specific features become unavailable. * **No Platform connection**: Tenant clusters deployed without Platform connectivity operate as OSS from the start. ## Activate the free tier[​](#activate-the-free-tier "Direct link to Activate the free tier") Activate the vCluster Free tier using these steps: ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * A Kubernetes cluster (Control Plane Cluster) where you'll deploy tenant clusters * kubectl installed and configured * The vCluster CLI installed ### Step 1: Install the vCluster Platform[​](#step-1-install-the-vcluster-platform "Direct link to Step 1: Install the vCluster Platform") Install the vCluster Platform in your Kubernetes cluster. See the [Platform installation guide](https://vcluster.com/docs/platform/install/quick-start-guide.md) for detailed instructions. No credit card is required for the Free tier. ### Step 2: Connect your tenant cluster[​](#step-2-connect-your-tenant-cluster "Direct link to Step 2: Connect your tenant cluster") After installing the Platform, connect existing tenant clusters or create new ones. **For existing tenant clusters:** ``` # Log in to the Platform vcluster platform login # Add an existing vCluster to the Platform vcluster platform add vcluster my-vcluster ``` **For new tenant clusters**, create them through the Platform UI or CLI - Free tier features are automatically enabled. ### Step 3: Deploy with Free tier features[​](#step-3-deploy-with-free-tier-features "Direct link to Step 3: Deploy with Free tier features") When creating tenant clusters through the Platform, Free tier features like embedded etcd are automatically available: vcluster.yaml ``` # Use embedded etcd (Free tier feature) controlPlane: backingStore: etcd: embedded: enabled: true ``` Deploy your tenant cluster: ``` vcluster create my-vcluster --values vcluster.yaml ``` ### Step 4: Verify Free tier activation[​](#step-4-verify-free-tier-activation "Direct link to Step 4: Verify Free tier activation") Check that your tenant cluster is connected and licensed in the Platform UI, or use the CLI: ``` # List virtual clusters managed by Platform vcluster platform list vclusters ``` You should see your tenant cluster listed with the Free tier license status. ## Frequently asked questions[​](#frequently-asked-questions "Direct link to Frequently asked questions") ### Is the Free tier really free?[​](#is-the-free-tier-really-free "Direct link to Is the Free tier really free?") Yes, the Free tier is completely free. Creating a vCluster Platform account is required, but no credit card or payment is necessary. ### Can OSS be used in production?[​](#can-oss-be-used-in-production "Direct link to Can OSS be used in production?") Yes, vCluster OSS is production-ready and deployed by platform teams worldwide. Paid tiers add high availability, external database support, SSO, audit logging, and enterprise support — capabilities that larger-scale platform deployments typically require. ### What happens if a Free tier license expires?[​](#what-happens-if-a-free-tier-license-expires "Direct link to What happens if a Free tier license expires?") Free tier licenses don't expire as long as the Platform account remains active. If connectivity issues prevent license validation, tenant clusters continue operating with a grace period. ### Can you upgrade from Free to a paid tier?[​](#can-you-upgrade-from-free-to-a-paid-tier "Direct link to Can you upgrade from Free to a paid tier?") Yes, upgrades are possible at any time from the vCluster Platform account settings. Existing tenant clusters automatically receive the new license without requiring redeployment. ### Does OSS require platform connectivity?[​](#does-oss-require-platform-connectivity "Direct link to Does OSS require platform connectivity?") No, vCluster OSS operates completely independently without requiring any connection to the vCluster Platform. Only Free and paid tiers require Platform connectivity for license validation. ## Next steps[​](#next-steps "Direct link to Next steps") * [Deploy your first tenant cluster](https://vcluster.com/docs/vcluster/.md) * [Understand vCluster architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md) * [Configure embedded etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/embedded.md) * [Explore the vCluster Platform](https://www.vcluster.com/platform) --- # What is vCluster? vCluster helps AI cloud providers deliver highly scalable, resilient compute clusters as self-service tenant products, including managed Kubernetes, Slurm, inference, Ray, and other cluster types. Each tenant gets a dedicated tenant cluster, provisioned on your infrastructure or directly on bare metal, with its own control plane and RBAC. Tenants get an isolated cluster experience, with the isolation level set by the worker node model you choose. Customer-facing, resold, and regulated offerings run on private nodes, which require a vCluster license. vCluster OSS defaults to shared nodes, which suit internal, trusted tenants. The control plane is completely invisible to tenants. There are no shared control plane nodes, no in-cluster agent pods, and no lateral path between environments. vCluster suits any environment where isolation is a hard requirement, from developer platforms and CI/CD pipelines to GPU cloud infrastructure serving paying tenants. Ready to deploy? See the [Quick Start](https://vcluster.com/docs/vcluster/.md) to choose the path that fits your environment. ## How it works[​](#how-it-works "Direct link to How it works") vCluster works across a range of infrastructure configurations. Each tenant cluster is backed by a dedicated virtualized control plane, built on a certified Kubernetes distribution. That's what lets tools built for the standard Kubernetes API, such as `kubectl`, Helm, Argo, and Crossplane, work against a tenant cluster without modification. It's the same foundation that higher-level cluster types like Slurm, Ray, and inference-serving stacks run on as workloads. **Where the control plane runs** * **On an existing Kubernetes cluster.** The control plane runs as a pod in a dedicated namespace. Your existing cluster becomes the Control Plane Cluster and no additional infrastructure is required. * **Standalone.** A complete, zero-dependency Kubernetes distribution that runs as a self-contained binary on bare metal or VMs. Use it as a full Kubernetes environment for any workload. When connected to vCluster Platform, it can act as a Control Plane Cluster for tenant isolation. * **vind.** The full stack runs in Docker containers with no Kubernetes dependency. Suited for local development and CI/CD pipelines. **How tenant workloads run** On cluster-based deployments, you choose how tenant workloads land on compute: * **Private nodes.** Each tenant cluster gets dedicated nodes enrolled through a token-based process. Network, storage, and compute are fully isolated per tenant, with no cross-tenant visibility at the infrastructure level. This model requires a vCluster license and is the isolation model for GPU workloads, regulated industries, and AI cloud platforms serving paying tenants. Nodes can come from bare metal ([vMetal](https://vcluster.com/docs/platform/administer/bare-metal/overview.md)), cloud VMs, or any Linux machine. * **Shared nodes.** Tenants share the existing node pool, with multiple tenant clusters running on the same physical nodes under full API-level isolation. Each tenant still gets a dedicated virtualized control plane, and its workloads run on the control plane cluster's node pool. Shared nodes are the vCluster OSS default and the only worker node model OSS provides. They aren't a security boundary for untrusted tenants, so they suit internal, trusted tenants such as developer platforms, CI environments, and testing rather than externally facing or resold offerings. Building a customer-facing or other production offering? [Choose a worker node model](https://vcluster.com/docs/vcluster/production-guide/choose-worker-node-model.md) walks through the questions that decide which one your offering requires. ## Features and plans[​](#features-and-plans "Direct link to Features and plans") vCluster is available as open source and as an enterprise-grade platform with additional features across paid tiers: | | | Enterprise | | | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ------ | ------ | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Advanced UI Customizations](https://vcluster.com/docs/platform/configure/platform-configs/custom-branding.md#advanced-custom-branding) | | | | | | [Air-Gapped Mode](https://vcluster.com/docs/platform/install/air-gapped.md) | | Add-On | Add-On | Add-On | | [Apps](https://vcluster.com/docs/platform/understand/what-are-apps.md) | | | | | | [Argo Integration](https://vcluster.com/docs/platform/integrations/argocd.md) | | | | | | [Audit Logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md) | | | | | | [Auto Sleep](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md) | | | | | | [Auto Sleep for Namespaces](https://vcluster.com/docs/platform/use-platform/namespaces/key-features/sleep-mode.md) | | | | | | [Auto Sleep for Virtual Clusters](https://vcluster.com/docs/platform/use-platform/virtual-clusters/key-features/sleep-mode.md) | | | | | | [Auto Snapshots](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/snapshots.md#enable-auto-snapshots) | | | | | | [Built-In CoreDNS](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/coredns.md) | | | | | | [Central HostPath Mapper](https://vcluster.com/docs/platform/maintenance/monitoring/central-hostpath-mapper) | | | | | | [Centralized Admission Control](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/admission-control.md) | | | | | | [Cert Manager Integration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/cert-manager.md) | | | | | | [Cluster Access](https://vcluster.com/docs/platform/administer/clusters/cluster-roles.md) | | | | | | [Cluster Role Management](https://vcluster.com/docs/platform/administer/clusters/cluster-roles.md) | | | | | | [ClusterAPI Node Provider](https://vcluster.com/docs/platform/api/resources/nodeprovider.md) | | | | | | [Connected Clusters](https://vcluster.com/docs/platform/administer/clusters/connect-cluster.md) | | | | | | [Custom Branding](https://vcluster.com/docs/platform/configure/platform-configs/custom-branding.md) | | | | | | [Custom Resource Syncing](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/custom-resources.md) | | | | | | [Database Connector](https://vcluster.com/docs/platform/administer/connector/database.md) | | | | | | [DRA Sync](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md#dynamic-resource-allocation-and-device-classes) | | | | | | [EKS Pod Identity for External Database Connections](https://vcluster.com/docs/platform/administer/connector/database.md) | | | | | | [Embedded etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/embedded.md) | | | | | | [External Database](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md) | | | | | | [External Database Kerberos Authentication](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md#authenticate-to-postgresql-with-kerberos-gssapi) | | | | | | [External Database RDS IAM Authentication](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md) | | | | | | [External Secrets Integration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/eso.md) | | | | | | [FIPS](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/fips.md) | | | | | | [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md) | | | | | | [HashiCorp Vault Integration](https://vcluster.com/docs/platform/integrations/hashicorp-vault.md) | | | | | | [High-Availability Mode](https://vcluster.com/docs/platform/maintenance/reliability-scaling/high-availability.md) | | | | | | [Hybrid Scheduling](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/pods/hybrid-scheduling.md) | | | | | | Isolated Control Plane | | | | | | [Istio Integration](https://vcluster.com/docs/vcluster/integrations/istio.md) | | | | | | Kube-vip Integration | | | | | | [KubeVirt Integration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/integrations/kubeVirt.md) | | | | | | [KubeVirt Node Provider](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md) | | | | | | [Least Privilege Mode](https://vcluster.com/docs/platform/configure/agent-settings/least-privilege-mode.md) | | | | | | [Machine Management](https://vcluster.com/docs/platform/use-platform/machines/overview.md) | | | | | | [Metal3 Integration](https://vcluster.com/docs/platform/administer/bare-metal/overview.md) | | | | | | [Metal3 Node Provider](https://vcluster.com/docs/platform/administer/node-providers/metal3.md) | | | | | | [Multi-Region Mode](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md) | | | | | | [Multi-Region Platform](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md) | | | | | | [Multiple SSO Providers](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md#multiple-sso-providers) | | | | | | [Namespace Management](https://vcluster.com/docs/platform/use-platform/namespaces/create/create-with-template.md) | | | | | | Netris | | | | | | NICo Node Provider | | | | | | [Node Profiles](https://vcluster.com/docs/platform/administer/node-profiles/overview.md) | | | | | | [Nvidia BCM Node Provider](https://vcluster.com/docs/platform/administer/node-providers/bcm.md) | | | | | | [Platform as OIDC Provider](https://vcluster.com/docs/platform/administer/authentication/oidc-provider.md) | | | | | | [Platform External Database](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md) | | | | | | [Private Nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes.md) | | | | | | [Private Nodes Auto Nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/auto-nodes.md) | | | | | | [Private Nodes VPN](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes.md) | | | | | | [Project Quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md) | | | | | | [Regional Cluster Endpoints](https://vcluster.com/docs/platform/administer/clusters/advanced/regional-cluster-endpoints.md) | | | | | | Resolve DNS | | | | | | [Secrets Sync](https://vcluster.com/docs/platform/use-platform/secrets/secret-sync.md) | | | | | | Security-Hardened vCluster Image | | | | | | [Single Sign-On](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md) | | | | | | [Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md) | | | | | | [Sync Namespaces toHost](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/namespaces.md) | | | | | | [Sync Patches](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) | | | | | | [Template Versioning](https://vcluster.com/docs/platform/administer/templates/versioning.md) | | | | | | [Terraform Node Provider](https://vcluster.com/docs/platform/administer/node-providers/terraform.md) | | | | | | Translate Patches | | | | | | vCluster Proxy Resources | | | | | | [Virtual Cluster Management](https://vcluster.com/docs/vcluster/introduction/what-are-virtual-clusters) | | | | | | [VM Management with KubeVirt](https://vcluster.com/docs/platform/administer/virtual-machines/overview.md) | | | | | | [vNode Runtime](https://www.vnode.com/docs) | | Add-On | Add-On | Add-On | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). ## Next steps[​](#next-steps "Direct link to Next steps") * [Deploy vCluster](https://vcluster.com/docs/vcluster/deploy.md) — control plane modes, worker node types, and settings locked at deploy time * [Architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md) — control plane internals, syncer behavior, and networking * [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) — dedicated infrastructure for GPU tenants and regulated workloads * [vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) — zero-dependency Kubernetes for bare metal and edge * [Building a GPU cloud platform](https://vcluster.com/docs/vcluster/introduction/gpu-cloud-platform.md) — deployment models for AI cloud providers offering managed Kubernetes * [Building an inference platform](https://vcluster.com/docs/vcluster/introduction/inference-platform.md) — tenancy models and capacity planning for managed inference * [vcluster.yaml reference](https://vcluster.com/docs/vcluster/configure/what-is-vcluster-yaml.md) — full configuration reference --- # Custom Links vCluster Platform lets you associate custom links to tenant cluster instances, for example to point to a github pull request or jira issue. * Platform UI * CLI 1. Go to the Virtual Clusters view using the menu on the left 2. Edit an existing Tenant Cluster or create a new one 3. Click Add Link to add a new link or Change to edit existing ones. 4. Enter a URL and optionally a label for your link. 5. Save your links by clicking Save. 6. Click on the Create Virtual Cluster, or Save Changes if you're editing an existing tenant cluster, to apply all changes. ``` vcluster create vcluster-1 --link 'my-issue=https://github.com/kubernetes/kubernetes/pull/1234' vcluster create vcluster-1 --link 'https://loft.sh' vcluster create vcluster-1 --link 'my-issue=https://github.com/kubernetes/kubernetes/pull/1234,documentation=https://example.com/docs' ``` --- # Gateway API Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Gateway API is the recommended path for new vCluster endpoint and tenant routing deployments. It provides a Kubernetes-native model for shared traffic infrastructure, explicit route attachment, and controller-owned status conditions. vCluster uses Gateway API in two main ways: * Expose the tenant cluster API server with a `TLSRoute`. * Sync tenant workload routes, such as `HTTPRoute` resources, to the control plane cluster. vCluster Platform also uses Gateway API for Gateway-backed endpoint and auto sleep workflows. These workflows require Gateway API CRDs and a compatible controller on the connected control plane cluster. ## Relationship to ingress-nginx[​](#relationship-to-ingress-nginx "Direct link to Relationship to ingress-nginx") The upstream ingress-nginx project is deprecated. Existing ingress-based vCluster and Platform configurations continue to work in vCluster v0.35 and vCluster Platform v4.10, but Gateway API is the forward path for new deployments. There is no automatic ingress-to-Gateway migration in vCluster v0.35 or Platform v4.10. Plan the move explicitly: keep existing ingress configurations in place, validate a Gateway controller, create a test tenant cluster with Gateway-backed endpoints, and then migrate selected workloads. For a step-by-step adoption path, see [Migrate from ingress-nginx to Gateway API](https://vcluster.com/docs/vcluster/manage/ingress-nginx-to-gateway-api.md). ## Gateway API uses in vCluster[​](#gateway-api-uses-in-vcluster "Direct link to Gateway API uses in vCluster") | Use case | Gateway API resource | Where to configure | | -------------------------------------------------------------------------------------------- | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | | Expose the tenant cluster API server with TLS passthrough | `TLSRoute` | [Access and expose vCluster](https://vcluster.com/docs/vcluster/manage/accessing-vcluster.md#expose-vcluster) | | Let tenants publish HTTP workloads through a Gateway controller in the control plane cluster | `HTTPRoute` | [Gateway API sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md) | | Let tenants create their own Gateway resources | `Gateway` | [Gateway API sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md) | | Let tenants attach routes to an admin-approved shared Gateway | Imported `Gateway` and `GatewayClass` | [Imported Gateways and GatewayClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/gateways.md) | | Wake sleeping tenant clusters through HTTP traffic | `HTTPRoute` with sleep-time route handling | [Configure auto sleep](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md#gateway-api-httproute-activity-detection) | ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before adopting Gateway API, verify: * Version [v1.5.0 or later](https://gateway-api.sigs.k8s.io/docs/implementations/versions/v1.5/) Gateway API CRDs are installed on the control plane cluster. * A Gateway API-compatible controller is installed and healthy. * At least one `GatewayClass` is available for the controller you plan to use. * DNS and certificates are planned for the Gateway listener hostnames. * The selected controller supports the route kinds you need, such as `TLSRoute` for tenant cluster API server exposure and `HTTPRoute` for application traffic. * If awake HTTP traffic must reset the inactivity timer, the selected controller supports `HTTPRoute` request mirroring or is explicitly allowed by Platform configuration. Gateway API support is controller-specific. A controller can support Gateway endpoints without supporting every feature, including auto sleep request mirroring. ## Use shared gateways for tenancy[​](#use-shared-gateways-for-tenancy "Direct link to Use shared gateways for tenancy") Platform teams often want a small number of administrator-owned Gateways and many tenant-owned Routes. In that model, administrators create and operate the Gateway in the control plane cluster. vCluster then imports a read-only representation into tenant clusters. Tenants attach `HTTPRoute` resources to that approved Gateway instead of creating their own Gateway. Use this model when DNS, certificates, listener policy, and infrastructure ownership should stay with the platform team. Use tenant-created Gateways when the tenant is allowed to own Gateway lifecycle and the controller supports that operating model. For the configuration model, see [Imported Gateways and GatewayClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/gateways.md). ## Validate routes[​](#validate-routes "Direct link to Validate routes") Use these checks when validating a Gateway API rollout: Check Gateway API CRDs ``` kubectl get crd gatewayclasses.gateway.networking.k8s.io gateways.gateway.networking.k8s.io httproutes.gateway.networking.k8s.io ``` Modify the following with your specific values to generate a copyable command: GATEWAY\_CLASS\_NAMEmy-gateway-class Check GatewayClasses ``` kubectl get gatewayclass kubectl describe gatewayclass my-gateway-class ``` Modify the following with your specific values to generate a copyable command: ROUTE\_NAMEmy-route NAMESPACEmy-namespace Check route status ``` kubectl get httproute,tlsroute -A kubectl describe httproute my-route -n my-namespace kubectl describe tlsroute my-route -n my-namespace ``` Look for accepted and programmed status conditions on Routes. If a Route is not accepted, check the referenced Gateway, Gateway listener settings, route kind support, cross-namespace attachment policy, and any required `ReferenceGrant` resources. --- # Ingress Access Typically, tenant clusters are accessed through the platform proxy, that is, requests to the tenant cluster API server, are proxied through the platform itself. This behavior allows for the platform to act as a single endpoint for all tenant clusters in the platform deployment. Because of this behavior, the platform is also able to act as a central point of authentication and authorization, and to log all interactions (if [vCluster Platform Auditing](https://vcluster.com/docs/platform/configure/platform-configs/audit.md) is licensed and enabled). In some situations you may prefer to access a tenant cluster API server directly, that is, not through the platform proxy. This behavior can be enabled with the tenant cluster `AccessPoint` feature. Enabling `AccessPoint` on a tenant cluster requires that the control plane cluster has a valid ingress controller deployed, and the `Cluster` object has the `loft.sh/ingress-suffix` annotation set with a valid domain. Gateway API adoption Gateway API is the recommended path for new endpoint and tenant routing deployments. Existing ingress access remains supported in vCluster Platform v4.10, but Gateway API migration is explicit and should be planned. See [Gateway API](https://vcluster.com/docs/vcluster/key-features/gateway-api.md) and [Migrate from ingress-nginx to Gateway API](https://vcluster.com/docs/vcluster/manage/ingress-nginx-to-gateway-api.md). The hostname used to access a tenant cluster that has the `AccessPoint` feature enabled, will be of the following format: `-.` Where the `` is the name of the tenant cluster instance, the `` is the name of the project the tenant cluster instance is created in, and the `` is the value from the `loft.sh/ingress-suffix` annotation on the cluster. ## Enable ingress access when creating the tenant cluster[​](#enable-ingress-access-when-creating-the-tenant-cluster "Direct link to Enable ingress access when creating the tenant cluster") Enabling the `AccessPoint` feature can be done during tenant cluster creation in the Platform UI. * Platform UI 1. From the project drop-down menu (top left corner), select the project you'd like to create the tenant cluster in. 2. Click on Virtual Clusters. 3. Click the New Virtual Cluster button. 4. Click the Configure button to skip selecting a tenant cluster template. 5. Click the Advanced Options. 6. Click the Ingress Access to expand the configuration section. 7. Slide the Enable Ingress Access slider to enable the ingress action. 8. Finish configuring anything else you'd like on your tenant cluster, then click the Create Virtual Cluster button. ``` vcluster create vcluster-1 --link 'my-issue=https://github.com/kubernetes/kubernetes/pull/1234' vcluster create vcluster-1 --link 'https://loft.sh' vcluster create vcluster-1 --link 'my-issue=https://github.com/kubernetes/kubernetes/pull/1234,documentation=https://example.com/docs' ``` Virtual Cluster Templates The `AccessPoint` feature can also be enabled on the template. --- # Isolation The vCluster project provides several configuration options pertaining to tenant cluster isolation and security. This section briefly covers the primary configuration options. For detailed information, see the [security overview](https://vcluster.com/docs/vcluster/security/.md) and [policies configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies.md). vCluster can isolate workloads by combining a Pod Security Standard, ResourceQuota, LimitRange, and NetworkPolicy. These settings are configured explicitly in `vcluster.yaml`. Enable these settings in your tenant cluster or tenant cluster template values: ``` policies: podSecurityStandard: "baseline" limitRange: enabled: true resourceQuota: enabled: true networkPolicy: enabled: true ``` All vCluster isolation settings can be customized through the policy configuration pages. CNIs and Network Policies Not all CNIs will support all network policies. Make sure you understand what capabilities your CNI supports when investigating tenant cluster isolation. --- # Kubectl shell Platform administrators and users alike often find themselves in a situation where they just need to execute a couple of kubectl commands against a cluster to troubleshoot or get a specific piece of information from it. The Kubectl Shell feature is a terminal that is connected to a pod containing kubectl, among other tools. The pod exists in the target vCluster within the namespace `vcluster-platform-shell`. A kubeconfig is mounted to the pod with a vCluster scoped AccessKey. This means the AccessKey cannot be used to access other resources in vCluster Platform. The Kubectl Shell Pod and AccessKey will automatically be cleaned up after 15 minutes of inactivity. To learn more about the security implications and options for security hardening, read the [Security Considerations section](#security-considerations). ## How to use[​](#how-to-use "Direct link to How to use") The Kubectl Shell feature can be accessed in the Project Overview page under the three-dot menu for any vCluster row. The Kubectl Shell feature can also be accessed on the vCluster Config page under the three-dot menu. ## Who can use it[​](#who-can-use-it "Direct link to Who can use it") Any user who has access to the VirtualClusterInstance, VirtualClusterInstances/Shell, and pod/exec permissions for the `vcluster-browser-shell` namespace within the vCluster can use the Kubectl Shell feature. Read the [Security Consideration section](#security-considerations) to understand the implications of granting these aforementioned permissions. The following YAML can be used to create a project role that grants the VirtualClusterInstance permissions needed for the Kubectl Shell feature: ``` kind: ClusterRoleTemplate apiVersion: management.loft.sh/v1 metadata: name: kubectlshell labels: loft.sh/project-role: 'true' spec: displayName: kubectlshell owner: user: admin management: true clusterRoleTemplate: metadata: {} rules: - verbs: - '*' apiGroups: - management.loft.sh resources: - virtualclusterinstances - virtualclusterinstances/shell ``` By default, users with VirtualClusterInstance access are granted the `cluster-admin` ClusterRole in the vCluster, which will grant the necessary pod/exec permission for the Kubectl Shell feature. Any ClusterRole that grants or Role that grants the following permission will suffice instead if you configure custom RBAC settings: ``` rules: - apiGroups: - "" resources: - pods/attach - pods/exec verbs: - create ``` If using a RoleBinding and Role, you would need to target the `vcluster-platform-shell` namespace. ## Security considerations[​](#security-considerations "Direct link to Security considerations") Required permissions for this feature should only be given to users who are otherwise vCluster admins. vCluster admins, not to be confused with vCluster Platform admins, are any users who have full access or effectively full access to perform sensitive actions within the targeted vCluster. The Kubectl Shell pods exist within the vCluster in the `vcluster-platform-shell` namespace. The kubeconfig is mounted from a secret that also exists in the `vcluster-platform-shell` namespace. Any user that has access to either exec into pods or get secrets in the `vcluster-platform-shell` have access to the kubeconfig and the AccessKey token embedded within the kubeconfig. The AccessKey is scoped to the vCluster, meaning it can only be used to access vCluster related resources and cannot be used to access any other resources in the vCluster Platform. Consequently, these permissions should only be granted to users who would be considered admins for the targeted vCluster. It is common for all users with access to a vCluster to effectively be admins for that vCluster. Pod exec permissions are a sensitive privilege and should only be given to vCluster admins regardless. Use normal Kubernetes RBAC to configure who has access to the `vcluster-platform-shell` namespace. --- # Sleep tenant clusters | | | Enterprise | | | | -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---- | ----- | | Available in these plans | [Free](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview)[](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview "Requires vCluster Platform. Open source users do not need Platform.") | Dev | Prod | Scale | | [Auto Sleep](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md) | | | | | Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See [Compare open source and free tiers](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md#overview). Sleep reduces the compute resources that idle tenant clusters consume. You can pause and resume a tenant cluster on demand, or configure auto sleep to respond to inactivity, schedules, and supported HTTP traffic. ## Choose a sleep method[​](#choose-a-sleep-method "Direct link to Choose a sleep method") | Method | Behavior | Use when | | --------------------------- | --------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | | Manual pause and resume | The `vcluster` CLI pauses or resumes the tenant cluster when you run a command. | You need an on-demand sleep trigger instead of inactivity or schedules. | | Standalone auto sleep | The tenant cluster scales down its workloads but keeps its control plane running to detect activity and wake the workloads. | The tenant cluster isn't managed by vCluster Platform. | | Platform-managed auto sleep | The platform agent manages sleep and wakeup and can scale down the tenant cluster control plane. | You need control plane sleep, Platform-proxied activity detection, Platform web-routing integrations, or auto-delete. | When a tenant cluster with auto sleep connects to a compatible platform agent, the agent takes over sleep management automatically. See [How auto sleep is managed](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md#how-auto-sleep-is-managed) for the version requirements and the `workloads-only` exception. Pre-production workloads Auto sleep is intended for pre-production workloads. Sleeping interrupts workloads, and recreated Pods lose temporary filesystem data. ## What happens during standalone auto sleep[​](#what-happens-during-standalone-auto-sleep "Direct link to What happens during standalone auto sleep") Without a platform agent, the tenant cluster control plane remains running. Native auto sleep scales down and restores the following workload types: * Deployments * StatefulSets * ReplicaSets * ReplicationControllers * DaemonSets, which sleep by using a node selector that doesn't match a node Native auto sleep deletes Pods that don't have a workload controller. Because it deletes the Pod object from the tenant cluster API, wakeup doesn't restore these Pods. You can exempt selected workloads from auto sleep. For example, keep an ingress controller running so it can receive traffic that wakes the tenant cluster. ## What happens during platform-managed auto sleep[​](#what-happens-during-platform-managed-auto-sleep "Direct link to What happens during platform-managed auto sleep") The platform sleeps the tenant cluster's namespaces on the control plane cluster. It scales down Deployments, StatefulSets, and standalone ReplicaSets, prevents new physical Pods from starting, and can scale down the tenant cluster control plane. The tenant cluster's Kubernetes API objects remain in its data store. After the control plane and [syncer](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync) wake, vCluster recreates physical workload Pods, including Pods without a Kubernetes workload controller. Recreated Pods lose temporary filesystem data and can receive new IP addresses. ## What wakes a tenant cluster[​](#what-wakes-a-tenant-cluster "Direct link to What wakes a tenant cluster") Depending on the deployment and configuration, the following activity can wake a tenant cluster or reset its inactivity timer: * Kubernetes API requests routed through vCluster Platform * Tenant cluster API requests handled by standalone auto sleep * Requests to supported Ingress, Istio, or Gateway API endpoints * A configured wakeup schedule * A manual wakeup or CLI connection HTTP activity detection depends on the routing integration. Safe activity-based auto sleep with Gateway API requires a controller that supports `HTTPRoute` request mirroring. ## Next steps[​](#next-steps "Direct link to Next steps") * [Pause and resume a tenant cluster](https://vcluster.com/docs/vcluster/manage/sleep-wakeup.md) for manual CLI operations. * [Configure auto sleep](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md) for `vcluster.yaml`, vCluster Platform, activity detection, schedules, exclusions, and auto-delete. * [Review platform sleep-mode annotations](https://vcluster.com/docs/platform/reference/platform-annotations.md#sleepmode-annotations) for request filtering and advanced platform behavior. --- # Snapshots vCluster Platform allows you to configure taking snapshots of the vCluster at specific intervals. For detailed vCluster-specific auto snapshots configuration and examples, see the [vCluster auto snapshots documentation](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/snapshots.md). ## View snapshots[​](#view-snapshots "Direct link to View snapshots") After enabling automatic snapshots, you can view the list of snapshots for each tenant cluster in the details of the tenant cluster. This menu is only available for Project admins and Platform admins users. ### Snapshot name[​](#snapshot-name "Direct link to Snapshot name") Snapshots are identified by a generated name formatted as `-.tar.gz`. ### Snapshot schedule lifecycle[​](#snapshot-schedule-lifecycle "Direct link to Snapshot schedule lifecycle") Snapshots transition through different statuses as part of their execution lifecycle. A typical flow starts with the snapshot being in the `Scheduled` state, then moves to the `In Progress` state while it is being stored. If the snapshot and storage is successful, the snapshot moves to the `Completed` state. If an error occurs, the snapshot will be moved to a `Failed` state. If a snapshot was removed from the storage backend, but not in vCluster Platform, the snapshot appears in the `Not Found` state. | Snapshot Status | Description | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `Scheduled` | A snapshot that is scheduled to be stored in the storage backend service. Only one snapshot can have the Scheduled status at a time. | | `In Progress` | A snapshot that is currently being stored in the storage backend service. | | `Completed` | A snapshot that has been successfully stored in the storage backend service. | | `Failed` | A snapshot that failed to be taken. | | `Not Found` | A snapshot that appears in the list but has been removed from the storage backend service. | --- # Deploy vCluster in an air-gapped environment using a private web server Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** For installing vCluster Standalone or Private Nodes in air-gapped environments, follow these tutorials: * [vCluster Standalone Air-gapped](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/security/air-gapped.md) * [Private Nodes Air-gapped](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/security/air-gapped.md) The standard method for installing vCluster is using Helm with the vCluster Labs charts repository. For air-gapped environments without internet access, you'll need to host the Helm chart locally. This guide demonstrates how to use a private GitLab instance as a web server to host the vCluster Helm chart. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * A web server accessible through HTTP or HTTPS to store the Helm chart and `index.yaml` * A private GitLab instance (used in this example) * A machine with internet access that has the Helm CLI installed ## Procedure[​](#procedure "Direct link to Procedure") ### Download and host the Helm chart[​](#download-and-host-the-helm-chart "Direct link to Download and host the Helm chart") 1. Add and update the vCluster Helm repository on a machine with internet access: ``` helm repo add loft https://charts.loft.sh helm repo update ``` 2. Download the Helm chart to your local disk: ``` helm pull loft/vcluster --version 0.23.0 ``` 3. Store the chart on your web server: ``` git clone cd mkdir vcluster mv vcluster-0.23.0.tgz vcluster/ ``` 4. Create an `index.yaml` file and push it to the GitLab repository: ``` helm repo index vcluster --url https://your-private-server.com/your-username/your-private-project/path-to-vcluster-chart git add -A git commit -m "Add vCluster Helm chart" git push ``` note Ensure the URL you specify points to the location where `vcluster-0.23.0.tgz` can be accessible to the target cluster. ### Deploy vCluster using the private Helm chart[​](#deploy-vcluster-using-the-private-helm-chart "Direct link to Deploy vCluster using the private Helm chart") After hosting the chart, deploy vCluster using either the Helm CLI or the vCluster platform #### Use the Helm CLI[​](#use-the-helm-cli "Direct link to Use the Helm CLI") ``` helm upgrade --install my-vcluster vcluster \ --values vcluster.yaml \ --repo https://your-private-server.com/your-username/your-private-project/path-to-vcluster-chart-directory \ --version 0.23.0 \ --namespace my-namespace \ --repository-config='' ``` #### Use the platform[​](#use-the-platform "Direct link to Use the platform") You can specify the private repository in your vCluster configuration: ##### Use a template[​](#use-a-template "Direct link to Use a template") ``` kind: VirtualClusterTemplate apiVersion: management.loft.sh/v1 metadata: name: my-template spec: template: helmRelease: chart: repo: https://your-private-server.com/your-username/your-private-project/path-to-vcluster-chart-directory version: 0.23.0 ``` ##### Use directly in the VirtualClusterInstance resource[​](#use-directly-in-the-virtualclusterinstance-resource "Direct link to Use directly in the VirtualClusterInstance resource") ``` kind: VirtualClusterInstance apiVersion: management.loft.sh/v1 metadata: name: my-vcluster spec: template: helmRelease: chart: repo: https://your-private-server.com/your-username/your-private-project/path-to-vcluster-chart-directory version: 0.23.0 ``` After creating the configuration, verify the deployment by creating the tenant cluster through the platform UI. --- # Use vCluster CLI as a kubectl plugin This guide demonstrates how to set up the vCluster CLI as a kubectl plugin and allows you to manage tenant clusters directly through kubectl commands. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * [Install the vCluster CLI](https://www.vcluster.com/docs/vcluster/#deploy-vcluster) * Ensure kubectl is installed and properly configured ## Set up the plugin[​](#set-up-the-plugin "Direct link to Set up the plugin") * Mac/Linux * Windows 1. Locate the installed vCluster binary: ``` which vcluster ``` The output typically displays `/usr/local/bin/vcluster`. 2. Create a symlink with the name `kubectl-vcluster` that points to the vCluster binary: ``` sudo ln -s $(which vcluster) /usr/local/bin/kubectl-vcluster ``` 1) Locate the `vcluster.exe` binary using File Explorer. It should be in `C:\Users\\AppData\Roaming` 2) Copy the vCluster binary to the same location and rename it to `kubectl-vcluster.exe` ## Test the plugin[​](#test-the-plugin "Direct link to Test the plugin") Verify the installation by running the following command: ``` kubectl vcluster --version ``` The output displays the installed vCluster version: ``` vcluster version 0.21.2 ``` ## Use the plugin[​](#use-the-plugin "Direct link to Use the plugin") After installation, you can run vCluster commands through kubectl: ``` # List vClusters kubectl vcluster list # Create a vCluster kubectl vcluster create my-vcluster # Connect to a vCluster kubectl vcluster connect my-vcluster ``` ## Troubleshoot kubectl plugin issues[​](#troubleshoot-kubectl-plugin-issues "Direct link to Troubleshoot kubectl plugin issues") ### Plugin not found[​](#plugin-not-found "Direct link to Plugin not found") If kubectl cannot find the plugin: * Verify the renamed binary is in a directory included in your system's PATH * Check the paths with `echo $PATH` (macOS/Linux) or `echo %PATH%` (Windows) * Ensure the binary name begins with `kubectl-` and has no file extension on macOS/Linux ### Permission denied[​](#permission-denied "Direct link to Permission denied") If you get permission errors: * **For macOS/Linux**: Ensure the binary has executable permissions with `chmod +x /usr/local/bin/kubectl-vcluster`. * **For Windows**: Verify the file permissions and PATH configuration. --- # Capture pprof for vCluster This guide shows how to use Performance Profiling in Go (pprof) to capture CPU and memory profiles from vCluster. Capturing [pprof](https://github.com/google/pprof) is important for identifying performance bottlenecks and optimizing resource usage. When vCluster experiences high CPU or memory usage, use profiling to identify performance bottlenecks and optimize resource consumption. ## Setup and capture pprof on vCluster[​](#setup-and-capture-pprof-on-vcluster "Direct link to Setup and capture pprof on vCluster") ### Enable verbose logging and pprof endpoint[​](#enable-verbose-logging-and-pprof-endpoint "Direct link to Enable verbose logging and pprof endpoint") To make the logging more verbose and to enable the endpoint that serves pprof data, add the following environment variable in the Helm chart values: ``` controlPlane: statefulSet: env: - name: "DEBUG" value: "true" ``` note You must be on the kube-context of the control plane cluster where the vCluster is running. ### Port forwarding[​](#port-forwarding "Direct link to Port forwarding") If you're not using the templates from the UI, you can execute the following commands from your terminal: ``` kubectl port-forward -n vclusternamespace pod/vclustername-0 10443:8443 ``` This command allows you to access services running inside the tenant cluster from your local machine without exposing them publicly through the control plane cluster’s services or ingress. If a template is not used, the values should be added directly to `.spec.template.helmRelease.values` in the `VirtualClusterInstance`. ### Capture CPU profile[​](#capture-cpu-profile "Direct link to Capture CPU profile") Run the following command locally: ``` curl -k "https://localhost:10443/debug/pprof/profile?second=60" > profile60.prof ``` This command fetches a CPU profile of the vCluster pod's processes (such as the virtual Kubernetes API server) for the past 60 seconds. A `profile60.prof` file is created on your local system. ### Capture memory profile[​](#capture-memory-profile "Direct link to Capture memory profile") In the terminal from your local, execute: ``` curl -k https://localhost:10443/debug/pprof/heap -o heapprofile.prof ``` This command fetches a heap profile from the vCluster pod by sending a request to the `/debug/pprof/heap` endpoint. You should see a `heapprofile.prof` file created on your local system. ## Next steps[​](#next-steps "Direct link to Next steps") After you've captured these files, you can [send them to us](https://loft-sh.slack.com/). This enables us to identify opportunities for optimization and enhance the performance of your application. --- # Check connection to a vCluster or control plane cluster When working with Kubernetes clusters, you can determine if you're connected to a vCluster or control plane cluster using the following methods: * **Method 1**: [Check node labels](#check-node-labels) - Examine node labels for vCluster-specific indicators * **Method 2**: [Examine the current context](#examine-context) - Look for vCluster naming patterns in your kubectl context * **Method 3**: [List all contexts](#parse-contexts) - Get a comprehensive view of available vCluster contexts ## Check for vCluster node labels[​](#check-node-labels "Direct link to Check for vCluster node labels") Retrieve the nodes and check for a vCluster node label: ``` kubectl get nodes --show-labels | grep vcluster ``` If this command returns results, you're likely connected to a vCluster. This method works in most cases but might not be entirely reliable. ## Examine the current context[​](#examine-context "Direct link to Examine the current context") For scripting purposes, you can check the current context: ``` kubectl config current-context ``` If the output contains `vcluster` (for example, `vcluster_vcluster-xx5gk`), you're connected to a vCluster. ## Parse all contexts[​](#parse-contexts "Direct link to Parse all contexts") To get a more comprehensive view, you can list all contexts and search for `vcluster`: ``` kubectl config get-contexts | grep vcluster ``` This shows all vCluster contexts available. ## Implement check in a script[​](#implement-check-in-a-script "Direct link to Implement check in a script") Here's a simple bash script that combines these methods: ``` #!/bin/bash # Check current context current_context=$(kubectl config current-context) if [[ $current_context == *"vcluster"* ]]; then echo "Connected to vCluster: $current_context" exit 0 fi # Check for vCluster node labels vcluster_nodes=$(kubectl get nodes --show-labels | grep vcluster) if [ ! -z "$vcluster_nodes" ]; then echo "Connected to vCluster (detected via node labels)" exit 0 fi echo "Connected to host cluster" exit 1 ``` This script first checks the current context name. If that doesn't indicate a vCluster, it then checks for vCluster node labels. If neither method detects a vCluster, it assumes you're connected to the control plane cluster. ## Additional tools[​](#additional-tools "Direct link to Additional tools") While not directly applicable for scripting, the following tools can help with manual context switching and management: * Install kubectl plugins using [Krew](https://krew.sigs.k8s.io/) * Use [kubectx](https://github.com/ahmetb/kubectx) for easier context switching These tools are particularly useful when working with multiple clusters or contexts in a development environment. --- # Configure custom CoreDNS Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Customizing CoreDNS enables tailored DNS management for specific requirements, such as handling custom zones, forwarding DNS queries, and integrating with external DNS providers. This guide explains how to configure and use custom CoreDNS setups in a vCluster. ## Common CoreDNS customizations[​](#common-coredns-customizations "Direct link to Common CoreDNS customizations") The following are typical customizations for CoreDNS in vCluster: * **Custom DNS zones**: Define specific domains that resolve to selected IP addresses. * **DNS forwarding**: Route specific domain queries to external DNS servers. * **Integration with external DNS providers**: Connect to cloud provider DNS services. * **Local DNS overrides**: Define host entries for development or testing environments. For more advanced configurations, see to the [CoreDNS documentation](https://coredns.io/manual/toc/). ## Define CoreDNS configuration[​](#define-coredns-configuration "Direct link to Define CoreDNS configuration") Create or modify the `vcluster.yaml` file to include a custom CoreDNS configuration using the `overwriteManifests` field: note All Corefile server blocks must bind on port **1053**, not the standard DNS port 53. The tenant cluster's CoreDNS process listens on port 1053 to avoid privilege requirements. A custom Corefile that binds on port 53 does not match the listener and silently breaks DNS resolution. UDP packets are dropped with no error in pod logs. ``` controlPlane: backingStore: etcd: deploy: enabled: true coredns: overwriteManifests: |- apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: |- .:1053 { errors health ready rewrite name regex .*\.nodes\.vcluster\.com kubernetes.default.svc.cluster.local kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } hosts /etc/NodeHosts { ttl 60 reload 15s fallthrough } prometheus :9153 forward . /etc/resolv.conf file /etc/coredns/dg.db dg.in cache 30 loop loadbalance } import /etc/coredns/custom/*.server NodeHosts: "" dg.db: | dg.in. IN SOA ns3.dg.in cto.dg.in 20132601 10800 900 1209600 86400 ; dg.in file *.dg.in. IN A 10.211.9.11 *.dg.in. IN A 10.211.9.12 *.dg.in. IN A 10.211.9.13 *.dg.in. IN A 10.211.9.14 *.dg.in. IN A 10.211.9.15 *.dg.in. IN A 10.211.9.16 ``` ## Apply your CoreDNS configuration[​](#apply-your-coredns-configuration "Direct link to Apply your CoreDNS configuration") After defining your CoreDNS configuration, choose one of these approaches based on your situation: ### Deploy a new vCluster[​](#deploy-a-new-vcluster "Direct link to Deploy a new vCluster") If you're creating a new vCluster, deploy it with your custom CoreDNS configuration using Helm: ``` helm install corednsissue loft/vcluster --version 0.20.0 -f vcluster.yaml ``` ### Update an existing vCluster[​](#update-an-existing-vcluster "Direct link to Update an existing vCluster") If you already have a running vCluster: 1. Update your `vcluster.yaml` with the new CoreDNS configuration or add additional entries: ``` da.db: | dada.in. IN SOA ns3.dada.in cto.da.in 20132601 10800 900 1209600 86400 ; da.in file *.da.in. IN A 10.311.9.11 *.da.in. IN A 10.311.9.12 *.da.in. IN A 10.311.9.13 *.da.in. IN A 10.311.9.14 *.da.in. IN A 10.311.9.15 *.da.in. IN A 10.311.9.16 ``` 2. Apply the changes using Helm: ``` helm upgrade corednsissue loft/vcluster --version 0.20.0 -f vcluster.yaml ``` ## Test the DNS resolution[​](#test-the-dns-resolution "Direct link to Test the DNS resolution") After configuring custom CoreDNS settings, verify that the setup works correctly: ### Verify ConfigMap changes[​](#verify-configmap-changes "Direct link to Verify ConfigMap changes") Ensure the CoreDNS ConfigMap reflects your custom configuration: ``` kubectl get cm coredns -n kube-system -o yaml ``` ### Confirm CoreDNS pod status[​](#confirm-coredns-pod-status "Direct link to Confirm CoreDNS pod status") Check that CoreDNS pods are running and using the updated configuration: ``` kubectl get pods -n kube-system -l k8s-app=kube-dns ``` ### Test DNS resolution[​](#test-dns-resolution "Direct link to Test DNS resolution") Use a test pod to validate that DNS resolution works for your custom zones: ``` kubectl run dns-test --image=busybox --restart=Never --rm -it -- nslookup myapp.da.in ``` --- # Configure CPU and memory limits for vCluster ## Error message[​](#error-message "Direct link to Error message") When using quota limits in your vCluster project, you might encounter this error: ``` FailedCreate create Pod vclusterpod-0 in StatefulSet vclusterpod failed error: admission webhook "quota.loft.sh" denied the request: pods "vclusterpod-0" is forbidden: failed quota: project-quota-test: must specify limits.cpu ``` This error occurs when the vCluster pod doesn't have CPU limits specified while running in a namespace with resource quotas. ## Solution[​](#solution "Direct link to Solution") You can configure CPU and memory limits for vCluster using either the Platform UI or CLI. ### Configure limits using the platform UI[​](#configure-limits-using-the-platform-ui "Direct link to Configure limits using the platform UI") 1. **Log in to the vCluster platform**. Access your platform dashboard. 2. **Start the vCluster creation process**. Click **Create vCluster**. 3. **Configure resource limits**. In `vcluster.yaml`, add the following configuration and modify values as needed: ``` controlPlane: statefulSet: resources: limits: cpu: "1" memory: "2Gi" ``` 4. **Create the vCluster**. Complete the creation process by clicking Create. ### Configure limits using the vCluster CLI[​](#configure-limits-using-the-vcluster-cli "Direct link to Configure limits using the vCluster CLI") 1. **Create a configuration file**. Create a file named `values.yaml` with the following content and edit values as needed: ``` controlPlane: statefulSet: resources: limits: cpu: "1" memory: "2Gi" ``` 2. **Deploy the vCluster with the configuration**. Use the vCluster CLI to create a new vCluster with the limits: ``` vcluster create my-vcluster --values values.yaml ``` ### Verify the configuration[​](#verify-the-configuration "Direct link to Verify the configuration") Since vCluster is deployed as a `StatefulSet`, verify the resource limits with the following: ``` kubectl get statefulset -n -o jsonpath="{.spec.template.spec.containers[*].resources.limits}" ``` This command displays the configured CPU and memory limits for your vCluster. ## Additional resources[​](#additional-resources "Direct link to Additional resources") * [vCluster resource management documentation](https://www.vcluster.com/docs/architecture/resources) * [Kubernetes resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) --- # Configure webhook token authentication Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. This guide shows how to configure the vCluster API server to use [webhook token authentication](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#webhook-token-authentication), enabling integration with external identity providers and custom authentication systems. ## Overview[​](#overview "Direct link to Overview") Webhook token authentication allows the Kubernetes API server to delegate token validation to an external service. When the API server receives a request with a bearer token it does not recognize, it sends a `TokenReview` request to the configured webhook service, which validates the token and returns user identity information. Use cases for webhook token authentication include: * Integrating with enterprise identity providers (LDAP, SAML, OAuth2/OIDC) that issue custom tokens * Implementing centralized authentication across multiple clusters * Supporting custom token formats with specific expiration or validation requirements * Enabling authentication through external systems like the Cluster API Provider for vCluster (CAPIVC) ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you begin, ensure you have: * A running vCluster using the K8s distribution (`controlPlane.distro.k8s.enabled: true`) * An external webhook authentication service accessible from the vCluster * TLS certificates for secure communication between the API server and webhook service note Webhook token authentication uses kube-apiserver, which supports the `--authentication-token-webhook-config-file` flag. ## Configure webhook token authentication[​](#configure-webhook-token-authentication "Direct link to Configure webhook token authentication") 1. Create a kubeconfig-style file that describes how to connect to your webhook authentication service: webhook-config.yaml ``` apiVersion: v1 kind: Config clusters: - name: authn-webhook cluster: # CA certificate for verifying the webhook service certificate-authority: /etc/kubernetes/pki/webhook-ca.crt # URL of your webhook authentication service server: https://auth-service.example.com/authenticate users: - name: authn-webhook-client user: # Client certificate for mutual TLS (optional) client-certificate: /etc/kubernetes/pki/webhook-client.crt client-key: /etc/kubernetes/pki/webhook-client.key contexts: - context: cluster: authn-webhook user: authn-webhook-client name: authn-webhook current-context: authn-webhook ``` 2. Create a ConfigMap in the vCluster namespace containing the webhook configuration: ``` kubectl create configmap webhook-token-auth-config \ --from-file=webhook-config.yaml \ -n vcluster-my-vcluster ``` If your webhook requires TLS certificates, create a Secret: ``` kubectl create secret generic webhook-auth-certs \ --from-file=webhook-ca.crt=/path/to/ca.crt \ --from-file=webhook-client.crt=/path/to/client.crt \ --from-file=webhook-client.key=/path/to/client.key \ -n vcluster-my-vcluster ``` 3. Configure vCluster to pass the webhook configuration to the API server: vcluster.yaml ``` controlPlane: distro: k8s: enabled: true apiServer: extraArgs: # Point to the webhook configuration file - --authentication-token-webhook-config-file=/etc/kubernetes/webhook/webhook-config.yaml # Cache successful authentications for 2 minutes (default) - --authentication-token-webhook-cache-ttl=2m statefulSet: persistence: addVolumes: # Mount the webhook config ConfigMap - name: webhook-config configMap: name: webhook-token-auth-config # Mount TLS certificates if needed - name: webhook-certs secret: secretName: webhook-auth-certs addVolumeMounts: - name: webhook-config mountPath: /etc/kubernetes/webhook - name: webhook-certs mountPath: /etc/kubernetes/pki ``` 4. Create or update your vCluster with the configuration: ``` vcluster create my-vcluster -f vcluster.yaml --connect=false ``` 5. Verify the API server is using the webhook configuration: ``` kubectl exec -n vcluster-my-vcluster my-vcluster-0 -c syncer -- \ ps -ef | grep authentication-token-webhook ``` You should see `--authentication-token-webhook-config-file` in the kube-apiserver arguments. ## Configuration options[​](#configuration-options "Direct link to Configuration options") ### Cache time-to-live[​](#cache-time-to-live "Direct link to Cache time-to-live") The `--authentication-token-webhook-cache-ttl` flag controls how long successful authentication responses are cached. The default is 2 minutes. Adjust this based on your security requirements: vcluster.yaml ``` controlPlane: distro: k8s: apiServer: extraArgs: - --authentication-token-webhook-config-file=/etc/kubernetes/webhook/webhook-config.yaml # Shorter cache for stricter security - --authentication-token-webhook-cache-ttl=30s ``` ### Webhook service implementation[​](#webhook-service-implementation "Direct link to Webhook service implementation") Your webhook service must implement the Kubernetes `TokenReview` API. When the API server receives an unrecognized token, it sends a POST request with a `TokenReview` object: ``` { "apiVersion": "authentication.k8s.io/v1", "kind": "TokenReview", "spec": { "token": "" } } ``` The webhook must respond with the authentication result: ``` { "apiVersion": "authentication.k8s.io/v1", "kind": "TokenReview", "status": { "authenticated": true, "user": { "username": "user@example.com", "uid": "12345", "groups": ["developers", "team-a"] } } } ``` ## Use with CAPIVC[​](#use-with-capivc "Direct link to Use with CAPIVC") When using vCluster with [Cluster API Provider for vCluster (CAPIVC)](https://github.com/loft-sh/cluster-api-provider-vcluster), you can configure webhook token authentication through the `VCLUSTER_YAML` environment variable: ``` cat > /tmp/values.yaml < loft/vcluster -n -f vcluster.yaml ``` * Replace `` and `` with your actual values. ## View debug logs[​](#view-debug-logs "Direct link to View debug logs") After enabling debug logging, view the logs with the following command: ``` kubectl logs -n ``` You'll see more detailed logs that include `DEBUG` level information. --- Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** # Override the Alpine sidecar image in vCluster vCluster automatically creates a small Alpine Linux container (called a "sidecar") alongside your workloads to manage network routing. This sidecar modifies the `/etc/hosts` file inside your tenant cluster's pods so they can communicate properly with the control plane cluster. By default, vCluster downloads this Alpine image from Docker Hub, but you can configure it to use a different Alpine image from your own container registry instead. You might need this for: * **Docker Hub rate limiting**: Avoid hitting Docker Hub's download limits in busy environments. * **Security requirements**: Use only approved, internal registries. * **Air-gapped environments**: No internet access requires local image hosting. * **Custom Alpine images**: Use modified images with additional tools or security patches. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you begin, ensure you have: **Required tools and access:** * A running Kubernetes cluster with vCluster installed (or ready to install) * `kubectl` installed and configured to access your cluster * `docker` command-line tool installed for image management * Permission to create secrets in your vCluster's namespace **Registry requirements:** * Access to a container registry (local, cloud, or public) * Credentials for private registries (username/password or access tokens) * Network connectivity from your Kubernetes cluster to the registry ## Set up your container registry[​](#set-up-your-container-registry "Direct link to Set up your container registry") Before configuring vCluster, you need a container registry with an Alpine image. You can choose the option that fits your environment from the following: ### Local testing or air-gapped environments[​](#local-testing-or-air-gapped-environments "Direct link to Local testing or air-gapped environments") Create a simple local registry for development or environments without internet access: ``` # Start a basic registry server docker run -d -p 5000:5000 --name local-registry registry:2 # Verify it's running curl http://localhost:5000/v2/_catalog ``` ### Cloud environments[​](#cloud-environments "Direct link to Cloud environments") Use existing managed registries for production deployments: * AWS ECR (Elastic Container Registry) * Google Container Registry (GCR) * Azure Container Registry (ACR) * Other cloud providers with managed registry services ### Complex air-gapped setups[​](#complex-air-gapped-setups "Direct link to Complex air-gapped setups") For enterprise air-gapped environments, vCluster provides official automation scripts that automatically pull and push all required images with proper version compatibility. These scripts handle dependencies and version management for you. For more information and detailed setup instructions, refer to the [vCluster air-gapped installation guide](https://www.vcluster.com/docs/platform/install/advanced/air-gapped#scripts-to-pull-and-push-images). ## Upload the Alpine image[​](#upload-the-alpine-image "Direct link to Upload the Alpine image") After you have a registry, upload an Alpine image to it. Start by pulling the official Alpine image and pushing it to your registry: ``` # Pull the Alpine image docker pull alpine:latest # Tag for your registry docker tag alpine:latest your-registry.com/alpine:latest # Push to your registry docker push your-registry.com/alpine:latest ``` The commands vary by registry type:
* Local registry * AWS ECR * GitHub Container Registry ``` docker tag alpine:latest localhost:5000/alpine:latest docker push localhost:5000/alpine:latest ``` ``` # First authenticate: aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 123456789.dkr.ecr.us-west-2.amazonaws.com docker tag alpine:latest 123456789.dkr.ecr.us-west-2.amazonaws.com/alpine:latest docker push 123456789.dkr.ecr.us-west-2.amazonaws.com/alpine:latest ``` ``` # First authenticate: echo $GITHUB_TOKEN | docker login ghcr.io -u your-username --password-stdin docker tag alpine:latest ghcr.io/your-username/alpine:latest docker push ghcr.io/your-username/alpine:latest ``` ## Configure vCluster to use your image[​](#configure-vcluster-to-use-your-image "Direct link to Configure vCluster to use your image") Add the image configuration to your `vcluster.yaml` file. This enables vCluster's pod sync to use your custom Alpine image instead of the default one when it creates the sidecar container that manages host file entries. The `sync.toHost.pods.rewriteHosts.initContainer.image` setting specifically controls which image vCluster uses for the network routing sidecar.
* vCluster 0.20 and later * vCluster 0.19 and earlier ``` sync: toHost: pods: rewriteHosts: initContainer: image: registry: your-registry.com repository: alpine tag: latest ``` ``` syncer: extraArgs: - '--override-hosts-container-image=your-registry.com/alpine:latest' ``` The following are examples for different registry types:
* Local registry * AWS ECR * GitHub Container Registry * Private registry ``` sync: toHost: pods: rewriteHosts: initContainer: image: registry: localhost:5000 repository: alpine tag: latest ``` ``` sync: toHost: pods: rewriteHosts: initContainer: image: "123456789.dkr.ecr.us-west-2.amazonaws.com/alpine:latest" ``` ``` sync: toHost: pods: rewriteHosts: initContainer: image: "ghcr.io/your-username/alpine:latest" ``` ``` sync: toHost: pods: rewriteHosts: initContainer: image: "registry.mycompany.com/alpine:3.18" ``` ## Set up authentication for private registries[​](#set-up-authentication-for-private-registries "Direct link to Set up authentication for private registries") note Skip this section if you're using a public registry, such as a local registry without authentication. Private registries require authentication. Most cloud registries (AWS ECR, Google Container Registry, GitHub Container Registry) and enterprise registries need credentials to pull images. Create a Kubernetes secret with your registry credentials. Replace the placeholder values with your actual registry URL, username, and password:
* Generic private registry * AWS ECR * GitHub Container Registry ``` kubectl create secret docker-registry my-registry-secret \ --docker-server=your-registry.com \ --docker-username=your-username \ --docker-password=your-password \ --docker-email=your-email@example.com \ --namespace=vcluster-namespace ``` ``` kubectl create secret docker-registry ecr-secret \ --docker-server=123456789.dkr.ecr.us-west-2.amazonaws.com \ --docker-username=AWS \ --docker-password=$(aws ecr get-login-password --region us-west-2) \ --namespace=vcluster-namespace ``` ``` kubectl create secret docker-registry ghcr-secret \ --docker-server=ghcr.io \ --docker-username=your-github-username \ --docker-password=your-github-token \ --namespace=vcluster-namespace ``` Then you can configure vCluster to use the secret by adding it to your `vcluster.yaml` file. This configures the service account that vCluster uses to run workloads. This gives it permission to pull images from your private registry using the credentials you created:
* vCluster 0.20 and later * vCluster 0.19 and earlier ``` controlPlane: advanced: workloadServiceAccount: imagePullSecrets: - name: my-registry-secret ``` ``` serviceAccount: imagePullSecrets: - name: my-registry-secret ``` ## Configuration examples[​](#configuration-examples "Direct link to Configuration examples") The following are complete `vcluster.yaml` configurations for different scenarios:
* Public registry (no authentication) * Private registry * AWS ECR ``` sync: toHost: pods: rewriteHosts: initContainer: image: "localhost:5000/alpine:latest" ``` ``` sync: toHost: pods: rewriteHosts: initContainer: image: "registry.mycompany.com/alpine:3.18" controlPlane: advanced: workloadServiceAccount: imagePullSecrets: - name: mycompany-registry-secret ``` ``` sync: toHost: pods: rewriteHosts: initContainer: image: "123456789.dkr.ecr.us-west-2.amazonaws.com/alpine:latest" controlPlane: advanced: workloadServiceAccount: imagePullSecrets: - name: ecr-secret ``` ## Deploy your changes[​](#deploy-your-changes "Direct link to Deploy your changes") After updating your `vcluster.yaml` file, apply the configuration:
* New vCluster installation * Existing vCluster ``` vcluster create my-vcluster --values vcluster.yaml ``` This creates a new vCluster using your custom Alpine image configuration. ``` vcluster create my-vcluster --upgrade --values vcluster.yaml ``` This applies your new Alpine image configuration to an existing vCluster. New pods use the custom image immediately, while existing pods use it when they restart. ## Troubleshoot[​](#troubleshoot "Direct link to Troubleshoot") If you encounter problems, check these common issues: * **Image pull errors**: Verify the image exists at the specified URL and the URL format is correct. Test manually with `docker pull your-registry.com/alpine:latest` to confirm the image is accessible. * **Authentication failures**: Ensure your secret is in the correct namespace and contains valid credentials. For cloud registries like ECR, check that tokens haven't expired and recreate the secret if needed. * **Registry connectivity**: Confirm your Kubernetes cluster can reach the registry URL. Test with `kubectl run test-pod --image=curlimages/curl --rm -it --restart=Never -- curl -I https://your-registry.com/v2/` to verify network access. * **Version compatibility**: Use Alpine images that include basic networking tools like `iptables` and `ip` that vCluster requires. Standard `alpine:latest` works well, but minimal custom images may be missing required utilities. --- # Retrieve the deployed vCluster configuration You can deploy vCluster using various tools including Helm charts, the vCluster CLI, ArgoCD, Flux, or other GitOps workflows. After deployment, you may need to inspect the actual configuration of a running vCluster instance when the original Helm values or deployment files are no longer available. This helps with debugging issues, understanding configuration drift, or replicating the setup in another environment. The following methods provide different levels of detail depending on your deployment approach and available tools: * **Helm (*Recommended*)**: Use when the vCluster is Helm-compatible (deployed with Helm or vCluster CLI) and structured output distinguishing custom and default values is needed. This method is also recommended when planning to replicate the deployment in another environment or when working with Helm-managed deployments and consistency is required. * **vCluster CLI**: Use when the complete effective configuration as vCluster interprets it is required or when troubleshooting vCluster-specific behavior or issues. This method is also helpful when the original deployment method is unknown or mixed, or when the most comprehensive view of the cluster state is needed. * **kubectl**: Use when vCluster CLI and Helm are not available in the environment or when working in a restricted environment with limited tool access. This method provides direct access to the raw internal configuration file when other methods are failing or when only kubectl access to the control plane cluster is available.
* Helm * vCluster CLI * kubectl For Helm-managed deployments, you can retrieve the configuration values used during installation and upgrades. #### Get user-defined values[​](#get-user-defined-values "Direct link to Get user-defined values") Use the following command to get only user-defined values: note The following command shows only the values you explicitly set. Modify the following with your specific values to generate a copyable command: VCLUSTER NAMEmy-vcluster VCLUSTER NAMESPACEvcluster-my-vcluster ``` helm get values my-vcluster -n vcluster-my-vcluster -o yaml ```
#### Get default and user-defined values[​](#get-default-and-user-defined-values "Direct link to Get default and user-defined values") Use the following command to get merged default and user values: note The following command provides the complete configuration including defaults that Helm applied to your deployment. Modify the following with your specific values to generate a copyable command: VCLUSTER NAMEmy-vcluster VCLUSTER NAMESPACEvcluster-my-vcluster ``` helm get values my-vcluster -n vcluster-my-vcluster -a -o yaml ``` The vCluster CLI provides a comprehensive view of your cluster configuration, showing both user-defined settings and applied defaults. Use the following command when you need the complete effective configuration that vCluster uses to manage your tenant cluster: Modify the following with your specific values to generate a copyable command: VCLUSTER NAMEmy-vcluster VCLUSTER NAMESPACEvcluster-my-vcluster ``` vcluster describe my-vcluster --namespace vcluster-my-vcluster -o yaml ``` When you don't have access to the vCluster CLI or Helm, you can read the configuration directly from the running vCluster pod. The following method accesses the actual configuration file that vCluster uses internally, though it requires more manual interpretation: Modify the following with your specific values to generate a copyable command: HOST CONTEXTmy-host-cluster VCLUSTER NAMESPACEvcluster-my-vcluster VCLUSTER POD NAMEmy-vcluster-0 ``` kubectl exec --context=my-host-cluster -n vcluster-my-vcluster my-vcluster-0 -it -- cat /var/lib/vcluster/config.yaml ``` Version note In vCluster versions before v0.26, the configuration file path is `/var/vcluster/config.yaml`. Starting in v0.26, the file path has moved to `/var/lib/vcluster/config.yaml` to align with common Kubernetes conventions. --- # Safely upgrade ETCD v3.5 -> v3.6 ## Upgrade ETCD from v3.5 to v3.6 in vCluster[​](#upgrade-etcd-from-v35-to-v36-in-vcluster "Direct link to Upgrade ETCD from v3.5 to v3.6 in vCluster") In vCluster, etcd can serve as the underlying storage backend, similar to vanilla Kubernetes. Its version can be configured either explicitly using a `controlPlane.backingStore.etcd.deploy.statefulSet.image.tag` section in the `vcluster.yaml` or implicitly derived from the vCluster version. Before upgrading, back up your data. Deployed etcd doesn't support `vcluster snapshot create`, so back it up with its own tooling instead. For embedded etcd, take a [vCluster snapshot](https://vcluster.com/docs/vcluster/manage/backup-restore/backup.md). See [Back up and restore](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md#back-up-and-restore) for details. etcd has identified and fixed an additional scenario that may cause upgrade failures when moving from etcd v3.5 to v3.6, specifically during rolling replacement upgrades where a new learner may receive an incorrect snapshot from an older member (≤ v3.5.19). This issue, detailed in [issue #20793](https://github.com/etcd-io/etcd/issues/20793), was addressed in etcd v3.5.24 using [PR #20797](https://github.com/etcd-io/etcd/pull/20797).
The recommended solution is to first upgrade to etcd v3.5.24 (or higher) before proceeding to v3.6. For users on v3.5.20-v3.5.23 from a fresh install, a direct upgrade to v3.6 is safe.
To safely upgrade etcd in vCluster, continue reading this page for detailed steps.

For more details, refer to the [etcd upgrade guide](https://etcd.io/docs/latest/upgrades/upgrade_3_6/). ## Version matrix for embedded ETCD and vCluster[​](#version-matrix-for-embedded-etcd-and-vcluster "Direct link to Version matrix for embedded ETCD and vCluster") | vCluster Version | ETCD Version | | ---------------- | ------------ | | v0.26.4 | v3.5.21 | | v0.26.5 | v3.5.25 | | v0.27.2 | v3.5.21 | | v0.27.3 | v3.5.25 | | v0.28.1 | v3.5.21 | | v0.28.2 | v3.5.25 | | v0.29.\*+ | v3.6.\* | **Important**: Before upgrading, read the [ETCD blog post about upgrading from 3.5 to 3.6](https://etcd.io/blog/2025/upgrade_from_3.5_to_3.6_issue_followup/). * Embedded ETCD * Deployed ETCD - vCluster v0.26.\[0-4] - vCluster v0.27.\[0-2] - vCluster v0.28.\[0-1] First upgrade to v0.26.5, it uses ETCD v3.5.25. Then, your ETCD data will be saved in a format compatible with upgrade to v3.6 First upgrade to v0.27.3, it uses ETCD v3.5.25. Then, your ETCD data will be saved in a format compatible with upgrade to v3.6 First upgrade to v0.28.2, it uses ETCD v3.5.25. Then, your ETCD data will be saved in a format compatible with upgrade to v3.6 Follow the ETCD-recommended upgrade path: 1. **v3.5.21 → v3.5.25** 2. **v3.5.25 → v3.6.x** You may set ETCD image tag via: vcluster.yaml ``` controlPlane: backingStore: etcd: deploy: statefulSet: image: registry: quay.io repository: coreos/etcd tag: v3.5.25 ``` ## Config reference[​](#config-reference "Direct link to Config reference") ## `backingStore` object [​](#backingStore "Direct link to backingStore")BackingStore defines which backing store to use for virtual cluster. If not defined will use embedded database as a default backing store. ### `etcd` object [​](#backingStore-etcd "Direct link to backingStore-etcd")Etcd defines that etcd should be used as the backend for the virtual cluster #### `embedded` object [​](#backingStore-etcd-embedded "Direct link to backingStore-etcd-embedded")Embedded defines to use embedded etcd as a storage backend for the virtual cluster ##### `enabled` boolean false [​](#backingStore-etcd-embedded-enabled "Direct link to backingStore-etcd-embedded-enabled")Enabled defines if the embedded etcd should be used. ##### `migrateFromDeployedEtcd` boolean false [​](#backingStore-etcd-embedded-migrateFromDeployedEtcd "Direct link to backingStore-etcd-embedded-migrateFromDeployedEtcd")MigrateFromDeployedEtcd signals that vCluster should migrate from the deployed external etcd to embedded etcd. ##### `snapshotCount` integer [​](#backingStore-etcd-embedded-snapshotCount "Direct link to backingStore-etcd-embedded-snapshotCount")SnapshotCount defines the number of snapshots to keep for the embedded etcd. Defaults to 10000 if less than 1. ##### `extraArgs` string\[] \[] [​](#backingStore-etcd-embedded-extraArgs "Direct link to backingStore-etcd-embedded-extraArgs")ExtraArgs are additional arguments to pass to the embedded etcd. #### `deploy` object [​](#backingStore-etcd-deploy "Direct link to backingStore-etcd-deploy")Deploy defines to use an external etcd that is deployed by the helm chart ##### `enabled` boolean false [​](#backingStore-etcd-deploy-enabled "Direct link to backingStore-etcd-deploy-enabled")Enabled defines that an external etcd should be deployed. ##### `statefulSet` object [​](#backingStore-etcd-deploy-statefulSet "Direct link to backingStore-etcd-deploy-statefulSet")StatefulSet holds options for the external etcd statefulSet. ##### `enabled` boolean true [​](#backingStore-etcd-deploy-statefulSet-enabled "Direct link to backingStore-etcd-deploy-statefulSet-enabled")Enabled defines if the statefulSet should be deployed ##### `enableServiceLinks` boolean true [​](#backingStore-etcd-deploy-statefulSet-enableServiceLinks "Direct link to backingStore-etcd-deploy-statefulSet-enableServiceLinks")EnableServiceLinks for the StatefulSet pod ##### `image` object [​](#backingStore-etcd-deploy-statefulSet-image "Direct link to backingStore-etcd-deploy-statefulSet-image")Image is the image to use for the external etcd statefulSet ##### `registry` string registry.k8s.io [​](#backingStore-etcd-deploy-statefulSet-image-registry "Direct link to backingStore-etcd-deploy-statefulSet-image-registry")Registry is the registry of the container image, e.g. my-registry.com or ghcr.io. This setting can be globally overridden via the controlPlane.advanced.defaultImageRegistry option. Empty means docker hub. ##### `repository` string etcd [​](#backingStore-etcd-deploy-statefulSet-image-repository "Direct link to backingStore-etcd-deploy-statefulSet-image-repository")Repository is the repository of the container image, e.g. my-repo/my-image ##### `tag` string 3.6.16-0 [​](#backingStore-etcd-deploy-statefulSet-image-tag "Direct link to backingStore-etcd-deploy-statefulSet-image-tag")Tag is the tag of the container image, and is the default version. ##### `imagePullPolicy` string [​](#backingStore-etcd-deploy-statefulSet-imagePullPolicy "Direct link to backingStore-etcd-deploy-statefulSet-imagePullPolicy")ImagePullPolicy is the pull policy for the external etcd image ##### `env` object\[] \[] [​](#backingStore-etcd-deploy-statefulSet-env "Direct link to backingStore-etcd-deploy-statefulSet-env")Env are extra environment variables ##### `extraArgs` string\[] \[] [​](#backingStore-etcd-deploy-statefulSet-extraArgs "Direct link to backingStore-etcd-deploy-statefulSet-extraArgs")ExtraArgs are appended to the etcd command. ##### `resources` object [​](#backingStore-etcd-deploy-statefulSet-resources "Direct link to backingStore-etcd-deploy-statefulSet-resources")Resources the etcd can consume ##### `limits` object [​](#backingStore-etcd-deploy-statefulSet-resources-limits "Direct link to backingStore-etcd-deploy-statefulSet-resources-limits")Limits are resource limits for the container ##### `requests` object map\[cpu:20m memory:150Mi] [​](#backingStore-etcd-deploy-statefulSet-resources-requests "Direct link to backingStore-etcd-deploy-statefulSet-resources-requests")Requests are minimal resources that will be consumed by the container ##### `pods` object [​](#backingStore-etcd-deploy-statefulSet-pods "Direct link to backingStore-etcd-deploy-statefulSet-pods")Pods defines extra metadata for the etcd pods. ##### `annotations` object {} [​](#backingStore-etcd-deploy-statefulSet-pods-annotations "Direct link to backingStore-etcd-deploy-statefulSet-pods-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#backingStore-etcd-deploy-statefulSet-pods-labels "Direct link to backingStore-etcd-deploy-statefulSet-pods-labels")Labels are extra labels for this resource. ##### `highAvailability` object [​](#backingStore-etcd-deploy-statefulSet-highAvailability "Direct link to backingStore-etcd-deploy-statefulSet-highAvailability")HighAvailability are high availability options ##### `replicas` integer 1 [​](#backingStore-etcd-deploy-statefulSet-highAvailability-replicas "Direct link to backingStore-etcd-deploy-statefulSet-highAvailability-replicas")Replicas are the amount of pods to use. ##### `scheduling` object [​](#backingStore-etcd-deploy-statefulSet-scheduling "Direct link to backingStore-etcd-deploy-statefulSet-scheduling")Scheduling options for the etcd pods. ##### `nodeSelector` object {} [​](#backingStore-etcd-deploy-statefulSet-scheduling-nodeSelector "Direct link to backingStore-etcd-deploy-statefulSet-scheduling-nodeSelector")NodeSelector is the node selector to apply to the pod. ##### `affinity` object {} [​](#backingStore-etcd-deploy-statefulSet-scheduling-affinity "Direct link to backingStore-etcd-deploy-statefulSet-scheduling-affinity")Affinity is the affinity to apply to the pod. ##### `tolerations` object\[] \[] [​](#backingStore-etcd-deploy-statefulSet-scheduling-tolerations "Direct link to backingStore-etcd-deploy-statefulSet-scheduling-tolerations")Tolerations are the tolerations to apply to the pod. ##### `priorityClassName` string [​](#backingStore-etcd-deploy-statefulSet-scheduling-priorityClassName "Direct link to backingStore-etcd-deploy-statefulSet-scheduling-priorityClassName")PriorityClassName is the priority class name for the the pod. ##### `podManagementPolicy` string Parallel [​](#backingStore-etcd-deploy-statefulSet-scheduling-podManagementPolicy "Direct link to backingStore-etcd-deploy-statefulSet-scheduling-podManagementPolicy")PodManagementPolicy is the statefulSet pod management policy. ##### `topologySpreadConstraints` object\[] \[] [​](#backingStore-etcd-deploy-statefulSet-scheduling-topologySpreadConstraints "Direct link to backingStore-etcd-deploy-statefulSet-scheduling-topologySpreadConstraints")TopologySpreadConstraints are the topology spread constraints for the pod. ##### `security` object [​](#backingStore-etcd-deploy-statefulSet-security "Direct link to backingStore-etcd-deploy-statefulSet-security")Security options for the etcd pods. ##### `podSecurityContext` object {} [​](#backingStore-etcd-deploy-statefulSet-security-podSecurityContext "Direct link to backingStore-etcd-deploy-statefulSet-security-podSecurityContext")PodSecurityContext specifies security context options on the pod level. ##### `containerSecurityContext` object {} [​](#backingStore-etcd-deploy-statefulSet-security-containerSecurityContext "Direct link to backingStore-etcd-deploy-statefulSet-security-containerSecurityContext")ContainerSecurityContext specifies security context options on the container level. ##### `persistence` object [​](#backingStore-etcd-deploy-statefulSet-persistence "Direct link to backingStore-etcd-deploy-statefulSet-persistence")Persistence options for the etcd pods. ##### `volumeClaim` object [​](#backingStore-etcd-deploy-statefulSet-persistence-volumeClaim "Direct link to backingStore-etcd-deploy-statefulSet-persistence-volumeClaim")VolumeClaim can be used to configure the persistent volume claim. ##### `enabled` boolean true [​](#backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-enabled "Direct link to backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-enabled")Enabled enables deploying a persistent volume claim. ##### `accessModes` string\[] \[ReadWriteOnce] [​](#backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-accessModes "Direct link to backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-accessModes")AccessModes are the persistent volume claim access modes. ##### `retentionPolicy` string Retain [​](#backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-retentionPolicy "Direct link to backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-retentionPolicy")RetentionPolicy is the persistent volume claim retention policy. ##### `size` string 5Gi [​](#backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-size "Direct link to backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-size")Size is the persistent volume claim storage size. ##### `storageClass` string [​](#backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-storageClass "Direct link to backingStore-etcd-deploy-statefulSet-persistence-volumeClaim-storageClass")StorageClass is the persistent volume claim storage class. ##### `volumeClaimTemplates` object\[] \[] [​](#backingStore-etcd-deploy-statefulSet-persistence-volumeClaimTemplates "Direct link to backingStore-etcd-deploy-statefulSet-persistence-volumeClaimTemplates")VolumeClaimTemplates defines the volumeClaimTemplates for the statefulSet ##### `addVolumes` object\[] \[] [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumes "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumes")AddVolumes defines extra volumes for the pod ##### `addVolumeMounts` object\[] [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts")AddVolumeMounts defines extra volume mounts for the container ##### `name` string [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-name "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-name")This must match the Name of a Volume. ##### `readOnly` boolean [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-readOnly "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-readOnly")Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false. ##### `mountPath` string [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-mountPath "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-mountPath")Path within the container at which the volume should be mounted. Must not contain ':'. ##### `subPath` string [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-subPath "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-subPath")Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root). ##### `mountPropagation` string [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-mountPropagation "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-mountPropagation")mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10. ##### `subPathExpr` string [​](#backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-subPathExpr "Direct link to backingStore-etcd-deploy-statefulSet-persistence-addVolumeMounts-subPathExpr")Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR\_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. ##### `annotations` object {} [​](#backingStore-etcd-deploy-statefulSet-annotations "Direct link to backingStore-etcd-deploy-statefulSet-annotations")Annotations are extra annotations for this resource. ##### `labels` object {} [​](#backingStore-etcd-deploy-statefulSet-labels "Direct link to backingStore-etcd-deploy-statefulSet-labels")Labels are extra labels for this resource. ##### `service` object [​](#backingStore-etcd-deploy-service "Direct link to backingStore-etcd-deploy-service")Service holds options for the external etcd service. ##### `enabled` boolean true [​](#backingStore-etcd-deploy-service-enabled "Direct link to backingStore-etcd-deploy-service-enabled")Enabled defines if the etcd service should be deployed ##### `annotations` object {} [​](#backingStore-etcd-deploy-service-annotations "Direct link to backingStore-etcd-deploy-service-annotations")Annotations are extra annotations for the external etcd service ##### `headlessService` object [​](#backingStore-etcd-deploy-headlessService "Direct link to backingStore-etcd-deploy-headlessService")HeadlessService holds options for the external etcd headless service. ##### `annotations` object {} [​](#backingStore-etcd-deploy-headlessService-annotations "Direct link to backingStore-etcd-deploy-headlessService-annotations")Annotations are extra annotations for the external etcd headless service #### `external` object [​](#backingStore-etcd-external "Direct link to backingStore-etcd-external")External defines to use a self-hosted external etcd that is not deployed by the helm chart ##### `enabled` boolean false [​](#backingStore-etcd-external-enabled "Direct link to backingStore-etcd-external-enabled")Enabled defines if the external etcd should be used. ##### `endpoint` string [​](#backingStore-etcd-external-endpoint "Direct link to backingStore-etcd-external-endpoint")Endpoint holds the endpoint of the external etcd server, e.g. my-example-service:2379 ##### `tls` object [​](#backingStore-etcd-external-tls "Direct link to backingStore-etcd-external-tls")TLS defines the tls configuration for the external etcd server ##### `caFile` string [​](#backingStore-etcd-external-tls-caFile "Direct link to backingStore-etcd-external-tls-caFile")CaFile is the path to the ca file ##### `certFile` string [​](#backingStore-etcd-external-tls-certFile "Direct link to backingStore-etcd-external-tls-certFile")CertFile is the path to the cert file ##### `keyFile` string [​](#backingStore-etcd-external-tls-keyFile "Direct link to backingStore-etcd-external-tls-keyFile")KeyFile is the path to the key file ### `database` object [​](#backingStore-database "Direct link to backingStore-database")Database defines that a database backend should be used as the backend for the virtual cluster. This uses a project called kine under the hood which is a shim for bridging Kubernetes and relational databases. #### `embedded` object [​](#backingStore-database-embedded "Direct link to backingStore-database-embedded")Embedded defines that an embedded database (sqlite) should be used as the backend for the virtual cluster ##### `enabled` boolean false [​](#backingStore-database-embedded-enabled "Direct link to backingStore-database-embedded-enabled")Enabled defines if the database should be used. ##### `dataSource` string [​](#backingStore-database-embedded-dataSource "Direct link to backingStore-database-embedded-dataSource")DataSource is the kine dataSource to use for the database. This depends on the database format. This is optional for the external database. Examples:- mysql: mysql://username:password\@tcp(hostname:3306)/vcluster - postgres: postgres\://username:password\@hostname:5432/vcluster ##### `identityProvider` string [​](#backingStore-database-embedded-identityProvider "Direct link to backingStore-database-embedded-identityProvider")IdentityProvider is the kine identity provider to use when generating temporary authentication tokens for enhanced security. This is optional for the external database. Examples:- aws: RDS IAM Authentication ##### `keyFile` string [​](#backingStore-database-embedded-keyFile "Direct link to backingStore-database-embedded-keyFile")KeyFile is the key file to use for the database. This is optional. ##### `certFile` string [​](#backingStore-database-embedded-certFile "Direct link to backingStore-database-embedded-certFile")CertFile is the cert file to use for the database. This is optional. ##### `caFile` string [​](#backingStore-database-embedded-caFile "Direct link to backingStore-database-embedded-caFile")CaFile is the ca file to use for the database. This is optional. ##### `extraArgs` string\[] \[] [​](#backingStore-database-embedded-extraArgs "Direct link to backingStore-database-embedded-extraArgs")ExtraArgs are additional arguments to pass to Kine. #### `external` object [​](#backingStore-database-external "Direct link to backingStore-database-external")External defines that an external database should be used as the backend for the virtual cluster ##### `enabled` boolean false [​](#backingStore-database-external-enabled "Direct link to backingStore-database-external-enabled")Enabled defines if the database should be used. ##### `dataSource` string [​](#backingStore-database-external-dataSource "Direct link to backingStore-database-external-dataSource")DataSource is the kine dataSource to use for the database. This depends on the database format. This is optional for the external database. Examples:- mysql: mysql://username:password\@tcp(hostname:3306)/vcluster - postgres: postgres\://username:password\@hostname:5432/vcluster ##### `identityProvider` string [​](#backingStore-database-external-identityProvider "Direct link to backingStore-database-external-identityProvider")IdentityProvider is the kine identity provider to use when generating temporary authentication tokens for enhanced security. This is optional for the external database. Examples:- aws: RDS IAM Authentication ##### `keyFile` string [​](#backingStore-database-external-keyFile "Direct link to backingStore-database-external-keyFile")KeyFile is the key file to use for the database. This is optional. ##### `certFile` string [​](#backingStore-database-external-certFile "Direct link to backingStore-database-external-certFile")CertFile is the cert file to use for the database. This is optional. ##### `caFile` string [​](#backingStore-database-external-caFile "Direct link to backingStore-database-external-caFile")CaFile is the ca file to use for the database. This is optional. ##### `extraArgs` string\[] \[] [​](#backingStore-database-external-extraArgs "Direct link to backingStore-database-external-extraArgs")ExtraArgs are additional arguments to pass to Kine. ##### `connector` string [​](#backingStore-database-external-connector "Direct link to backingStore-database-external-connector")Connector specifies a secret located in a connected vCluster Platform that contains database server connection information to be used by Platform to create a database and database user for the vCluster. and non-privileged user. A kine endpoint should be created using the database and user on Platform registration. This is optional. --- # Use custom schedulers with Hybrid scheduling This guide shows how to: * Run a custom scheduler in the control plane cluster. * Run a custom scheduler in the tenant cluster. * Use vCluster Hybrid scheduling, so Pods from the tenant cluster can use schedulers from both control plane and tenant cluster. How pods are scheduled with Hybrid scheduling enabled: * Pods without `spec.schedulerName` use the default scheduler inside the tenant cluster. * Pods whose `spec.schedulerName` matches any scheduler from `sync.toHost.pods.hybridScheduling.hostSchedulers` in your vCluster config are scheduled by that matching host scheduler. * Pods whose `spec.schedulerName` matches a scheduler running inside the tenant cluster are scheduled by that virtual scheduler. See [Hybrid scheduling](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/pods/hybrid-scheduling.md) docs for more details. Enterprise feature with node synced from host to virtual Hybrid scheduling feature that is used in this guide is an Enterprise feature that is available in vCluster v0.26.0 and newer. It requires shared host nodes and host nodes have to be synced into the tenant cluster. This guide enables node syncing in the `my-vcluster.yaml` configuration you create in [Create control plane and tenant cluster](#create-control-plane-and-tenant-cluster). ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you begin, ensure you have: * kind, * container runtime that kind can use (e.g. Docker), * kubectl, * vCluster CLI. ## Create control plane and tenant cluster[​](#create-control-plane-and-tenant-cluster "Direct link to Create control plane and tenant cluster") Create kind control plane cluster: ``` kind create cluster --name scheduler-demo ``` Create a vCluster config file with Hybrid scheduling enabled and node syncing from the control plane cluster: ``` cat > my-vcluster.yaml <`. containerd uses HTTPS for registry endpoints by default. If the registry serves plain HTTP or its certificate doesn't match the node hostname, use the containerd host configuration in the next section. Pod spec ``` image: :30500/my-image:tag ``` Kubernetes exposes the same NodePort on every eligible node. Use a stable node address or hostname that the nodes themselves can resolve. This option changes every image reference to a node address and a non-standard port. ### Redirect containerd with hosts.toml[​](#redirect-containerd-with-hoststoml "Direct link to Redirect containerd with hosts.toml") Keep the FQDN in image references, and redirect containerd to the registry at the OS layer instead of through DNS. First, verify that the containerd CRI plugin loads registry host configurations from `/etc/containerd/certs.d`. For containerd 2.x, `/etc/containerd/config.toml` needs this configuration: containerd 2.x ``` version = 3 [plugins."io.containerd.cri.v1.images".registry] config_path = "/etc/containerd/certs.d" ``` For containerd 1.x, use the following configuration instead: containerd 1.x ``` version = 2 [plugins."io.containerd.grpc.v1.cri".registry] config_path = "/etc/containerd/certs.d" ``` Restart containerd after adding or changing `config_path`. Follow your node maintenance procedure when restarting the runtime. You can skip the restart if `config_path` already has the correct value. Create `/etc/containerd/certs.d/registry.ns.svc.cluster.local/hosts.toml` on each node: hosts.toml ``` server = "https://registry.ns.svc.cluster.local" [host."http://10.96.0.50:5000"] capabilities = ["pull", "resolve"] ``` Point the `host` entry at the registry's `ClusterIP` or `NodePort` address instead of its FQDN. The mirror entry needs an address the node can reach without resolving the registry FQDN. Deploy a DaemonSet that writes this file to each node, so the configuration also reaches newly scaled nodes. After you configure `config_path`, containerd loads `hosts.toml` changes without a restart. This approach preserves the FQDN in image references and follows containerd's [registry host model](https://github.com/containerd/containerd/blob/main/docs/hosts.md). ### Ingress with a resolvable hostname[​](#ingress-with-a-resolvable-hostname "Direct link to Ingress with a resolvable hostname") Expose the registry through an Ingress controller, using a hostname that resolves from the node's own DNS, such as a public or internal DNS zone the node already trusts. Reference that hostname in image specs instead of the in-cluster FQDN. This avoids per-node file management, at the cost of routing registry traffic through the ingress path. ## Private nodes and vind[​](#private-nodes-and-vind "Direct link to Private nodes and vind") The kubelet DNS ceiling above is specific to shared nodes, where tenant workloads run on the control plane cluster's own nodes and share its kubelet. With private nodes, each tenant cluster has its own kubelet running on its own dedicated nodes, so image pull redirection is a node-join setting rather than a DNS or DaemonSet workaround. Configure it through [`privateNodes.joinNode.containerd.registry`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes.md#privateNodes-joinNode-containerd-registry) for nodes joined through kubeadm. vind doesn't use that setting. Its Docker driver serves images to worker node containers through a host-cache registry proxy instead. See [Deploy vind with restricted network access](https://vcluster.com/docs/vcluster/deploy/control-plane/docker-container/air-gapped.md) for how vind handles image pulls in a restricted-network environment. --- # Set up the resource proxy Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. vCluster Platform required This feature requires [vCluster Platform](https://vcluster.com/docs/platform/.md). Both the client and target tenant clusters must be managed as [VirtualClusterInstance](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance/.md) within the platform. The resource proxy lets a client tenant cluster transparently forward custom resource requests to a target tenant cluster, which stores the objects and can run controllers against them. Set it up to centralize management of selected custom resources across tenant clusters, expose target-hosted custom resources across projects, or build cross-cluster automation. This page walks through setting it up for common scenarios. For the mechanism, see [How the resource proxy works](https://vcluster.com/docs/vcluster/understand/resource-proxy.md). For the `vcluster.yaml` fields themselves, see [Configure the resource proxy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/proxy.md). ## Platform RBAC requirements[​](#platform-rbac "Direct link to Platform RBAC requirements") Platform checks access to the target at two points, using two different principals: * When someone creates or updates the client VirtualClusterInstance, the user or automation submitting that request must have `use` access to the target VirtualClusterInstance. * When the client tenant cluster registers itself, the client's Platform identity must have the same `use` access. Its identity has the format `loft:vcluster::`. These checks require RBAC on the platform's management cluster, in addition to the resource RBAC you configure inside the target tenant cluster. ### Platform RBAC configuration[​](#platform-rbac-config "Direct link to Platform RBAC configuration") Create a Role and RoleBinding in the target tenant cluster's project namespace on the platform's management cluster. With the default project namespace prefix, this namespace is `p-`. Installations with a customized prefix, including installations upgraded from older versions, can use a different project namespace. The following RoleBinding grants access to the client identity used during registration: platform-proxy-rbac.yaml ``` # Platform RBAC for resource proxy # This grants the client cluster identity access during registration. # The identity creating or updating the client also needs use access. # Apply to the platform's management cluster in the target project namespace. apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: vcluster-proxy-target-access namespace: p-default rules: - apiGroups: ["management.loft.sh"] resources: ["virtualclusterinstances"] resourceNames: ["target"] # Target vCluster name verbs: ["use"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: client-vcluster-proxy-access namespace: p-default subjects: - kind: User name: "loft:vcluster:p-default:client" # Client vCluster identity apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: vcluster-proxy-target-access apiGroup: rbac.authorization.k8s.io ``` Apply this configuration to the platform's management cluster (not the tenant clusters): Apply platform RBAC ``` kubectl apply -f platform-proxy-rbac.yaml --context ``` The user or automation that creates or updates the client may already receive `use` access through its Platform project role. Check the submitting identity separately: Check access for the current Platform identity ``` kubectl auth can-i use virtualclusterinstances.management.loft.sh/target \ -n p-default \ --context ``` If the command returns `no`, a Platform administrator must bind that user, group, or service account to the `vcluster-proxy-target-access` Role, or grant equivalent `use` access. This must happen before it creates or updates the client. ## Examples[​](#examples "Direct link to Examples") ### Basic proxy setup[​](#basic-setup "Direct link to Basic proxy setup") This example demonstrates a simple two-cluster setup where a client tenant cluster proxies MyResource resources to a target tenant cluster. 1. **Create the target tenant cluster**. Create a tenant cluster to serve as the target. The target doesn't need any proxy configuration - it just stores the resources and enforces RBAC: Create target tenant cluster ``` vcluster create target --driver platform --project default ``` 2. **Install the CustomResourceDefinition in the target tenant cluster**. The CustomResourceDefinition must exist in the target tenant cluster: myresource-crd.yaml ``` apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: myresources.example.com spec: group: example.com names: kind: MyResource listKind: MyResourceList plural: myresources singular: myresource scope: Namespaced versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: name: type: string priority: type: string ``` Apply the CustomResourceDefinition to the target tenant cluster: Apply CustomResourceDefinition to target ``` vcluster connect target --driver platform --project default -- kubectl apply -f myresource-crd.yaml ``` 3. **Configure RBAC in the target tenant cluster**. Create RBAC rules to allow the client tenant cluster to access resources. The client tenant cluster authenticates using its vCluster Platform identity in the format `loft:vcluster::`. For example, a client named `client` whose project namespace is `p-default` uses `loft:vcluster:p-default:client`: target-rbac.yaml ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: vcluster-proxy-client rules: - apiGroups: ["example.com"] resources: ["myresources"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["example.com"] resources: ["myresources/status"] verbs: ["get", "update", "patch"] - apiGroups: [""] resources: ["namespaces"] verbs: ["get", "list", "watch", "create"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: vcluster-proxy-client roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: vcluster-proxy-client subjects: - kind: User # vCluster identity format: loft:vcluster:: name: "loft:vcluster:p-default:client" apiGroup: rbac.authorization.k8s.io ``` Apply RBAC to the target tenant cluster: Apply RBAC to target ``` vcluster connect target --driver platform --project default -- kubectl apply -f target-rbac.yaml ``` Keep the `get` verb when granting `update`, `patch`, or `delete` in `owned` mode. The proxy performs a GET before those named mutations to verify ownership. Without `get`, it masks the forbidden ownership check as a 404. 4. **Configure platform RBAC**. Grant the client identity permission to access the target when the tenant cluster registers with vCluster Platform. Apply this to the platform's management cluster: platform-rbac.yaml ``` # Platform RBAC for resource proxy # This grants the client cluster identity access during registration. # The identity creating or updating the client also needs use access. # Apply to the platform's management cluster in the target project namespace. apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: vcluster-proxy-target-access namespace: p-default rules: - apiGroups: ["management.loft.sh"] resources: ["virtualclusterinstances"] resourceNames: ["target"] # Target vCluster name verbs: ["use"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: client-vcluster-proxy-access namespace: p-default subjects: - kind: User name: "loft:vcluster:p-default:client" # Client vCluster identity apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: vcluster-proxy-target-access apiGroup: rbac.authorization.k8s.io ``` Apply to the platform's management cluster: Apply platform RBAC ``` kubectl apply -f platform-rbac.yaml --context ``` Before creating the client in the next step, also verify that your current Platform identity can use the target: Check creator access to target ``` kubectl auth can-i use virtualclusterinstances.management.loft.sh/target \ -n p-default \ --context ``` 5. **Create the client tenant cluster with proxy configuration**. Configure the client tenant cluster to proxy MyResource resources to the target: client-vcluster.yaml ``` experimental: proxy: customResources: myresources.example.com/v1: enabled: true targetVirtualCluster: name: "target" ``` Deploy the client tenant cluster: Deploy client tenant cluster ``` vcluster create client --driver platform --project default -f client-vcluster.yaml ``` 6. **Test the proxy**. Create a MyResource in the client tenant cluster: Create MyResource in client ``` vcluster connect client --driver platform --project default -- kubectl apply -f - < ``` 4. **Test isolation**. Test multi-client isolation ``` # Team A creates a resource vcluster connect team-a --driver platform --project default -- kubectl apply -f - < { const selector = context.virtualObject.spec && context.virtualObject.spec.selector; const scope = `vcluster.loft.sh/namespace == '${context.virtualObject.metadata.namespace}'`; if (!value.spec) value.spec = {}; value.spec.selector = selector ? `(${selector}) && ${scope}` : scope; return value; })(value) reverseExpression: | (value => { if (value.spec) { delete value.spec.selector; delete value.spec.ingress; delete value.spec.egress; } return value; })(value) # Scopes the positive rule-level selectors the same way. - path: spec.ingress[*].source.selector expression: | value ? `(${value}) && vcluster.loft.sh/namespace == '${context.virtualObject.metadata.namespace}'` : value - path: spec.ingress[*].destination.selector expression: | value ? `(${value}) && vcluster.loft.sh/namespace == '${context.virtualObject.metadata.namespace}'` : value - path: spec.egress[*].source.selector expression: | value ? `(${value}) && vcluster.loft.sh/namespace == '${context.virtualObject.metadata.namespace}'` : value - path: spec.egress[*].destination.selector expression: | value ? `(${value}) && vcluster.loft.sh/namespace == '${context.virtualObject.metadata.namespace}'` : value rbac: role: extraRules: # Calico's API server checks tier access before it accepts a policy write. - apiGroups: ["projectcalico.org"] resources: ["tier.networkpolicies"] resourceNames: ["default.*"] verbs: ["create", "delete", "patch", "update", "get", "list", "watch"] clusterRole: extraRules: - apiGroups: ["projectcalico.org"] resources: ["tiers"] resourceNames: ["default"] verbs: ["get"] ``` This configuration: * Restricts the policy's target endpoints and positive rule endpoint selectors to the tenant namespace where the policy was created. The empty-path patch sets `spec.selector` for both a selector the tenant wrote and one it omitted. A path-specific patch can't supply a field that isn't there. The four `spec.ingress`/`spec.egress` entries scope the rule-level `source` and `destination` selectors the same way. The brackets around `value` keep the tenant's own selector intact, and are required. See [Limitations](#limitations). * Removes the composed fields from control plane cluster changes before they reach the tenant. The empty-path patch's `reverseExpression` deletes `spec.selector`, `spec.ingress`, and `spec.egress` from the change set, so a composed selector never lands back in the tenant object. The four rule-selector entries omit `reverseExpression` themselves, which is safe here only because the empty-path patch already removes their parent field first. * Grants the permissions Calico's API server needs for the `default` tier. It authorizes a policy write against related `tier` resources that you never configure for sync. To use another tier, replace both `default` values with that tier's name and set `spec.tier` on the policy. See [Authorization on the control plane cluster](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/custom-resources.md#authorization-on-the-control-plane-cluster). ### Sync a policy and check its scope[​](#sync-a-policy-and-check-its-scope "Direct link to Sync a policy and check its scope") 1. Tenant Cluster Create two namespaces. Each namespace has a web server and a client with identical labels: Create the workloads ``` kubectl --context="${VCLUSTER_CTX}" create namespace demo kubectl --context="${VCLUSTER_CTX}" create namespace demo2 kubectl --context="${VCLUSTER_CTX}" -n demo run web --image=nginx --labels=app=web kubectl --context="${VCLUSTER_CTX}" -n demo run probe --image=busybox:1.36 \ --labels=role=client -- sleep 3600 kubectl --context="${VCLUSTER_CTX}" -n demo2 run web --image=nginx --labels=app=web kubectl --context="${VCLUSTER_CTX}" -n demo2 run probe --image=busybox:1.36 \ --labels=role=client -- sleep 3600 kubectl --context="${VCLUSTER_CTX}" wait --for=condition=Ready pod --all -n demo --timeout=2m kubectl --context="${VCLUSTER_CTX}" wait --for=condition=Ready pod --all -n demo2 --timeout=2m ``` 2. #### Apply a policy in one namespace[​](#apply-a-policy-in-one-namespace "Direct link to Apply a policy in one namespace") Tenant Cluster Apply a policy in `demo` only: deny-web-ingress.yaml ``` apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: deny-web-ingress namespace: demo spec: selector: app == 'web' types: - Ingress ingress: - action: Allow source: selector: role == 'client' ``` Apply the policy ``` kubectl --context="${VCLUSTER_CTX}" create -f deny-web-ingress.yaml ``` 3. #### Check the selector on the control plane cluster[​](#check-the-selector-on-the-control-plane-cluster "Direct link to Check the selector on the control plane cluster") Control Plane Cluster vCluster rewrites object names, so list the policies instead of looking one up by name: Read the synced selectors ``` kubectl --context="${HOST_CTX}" -n "${HOST_NAMESPACE}" \ get networkpolicies.projectcalico.org \ -o custom-columns='NAME:.metadata.name,TARGET:.spec.selector,SOURCE:.spec.ingress[0].source.selector' ``` The synced copy carries the scope term: Composed selector ``` NAME TARGET SOURCE v1mn70meafc7je (app == 'web') && vcluster.loft.sh/namespace == 'demo' (role == 'client') && vcluster.loft.sh/namespace == 'demo' ``` 4. #### Confirm the policy works within its namespace[​](#confirm-the-policy-works-within-its-namespace "Direct link to Confirm the policy works within its namespace") Tenant Cluster The client in `demo` can reach the selected web server in the same namespace: Allow the client in demo ``` DEMO_WEB_IP=$(kubectl --context="${VCLUSTER_CTX}" -n demo get pod web -o jsonpath='{.status.podIP}') kubectl --context="${VCLUSTER_CTX}" -n demo exec probe -- \ wget -q -T 5 -O- "http://${DEMO_WEB_IP}:80/" ``` 5. #### Confirm the rule selector doesn't cross namespaces[​](#confirm-the-rule-selector-doesnt-cross-namespaces "Direct link to Confirm the rule selector doesn't cross namespaces") Tenant Cluster The identically labeled client in `demo2` can't reach the protected web server in `demo`: Calico programs policy asynchronously. If this check reports unexpected access immediately after you create the policy, wait a few seconds and run it again. Block the client in demo2 ``` DEMO_WEB_IP=$(kubectl --context="${VCLUSTER_CTX}" -n demo get pod web -o jsonpath='{.status.podIP}') if kubectl --context="${VCLUSTER_CTX}" -n demo2 exec probe -- \ wget -q -T 5 -O- "http://${DEMO_WEB_IP}:80/"; then echo "unexpectedly reached the protected workload" else echo "blocked as expected" fi ``` 6. #### Confirm the other namespace is unaffected[​](#confirm-the-other-namespace-is-unaffected "Direct link to Confirm the other namespace is unaffected") Tenant Cluster The web server in `demo2` keeps serving traffic because its namespace has no policy: Reach the unprotected workload in demo2 ``` DEMO2_WEB_IP=$(kubectl --context="${VCLUSTER_CTX}" -n demo2 get pod web -o jsonpath='{.status.podIP}') kubectl --context="${VCLUSTER_CTX}" -n demo2 exec probe -- \ wget -q -T 5 -O- "http://${DEMO2_WEB_IP}:80/" ``` 7. #### Confirm the tenant object is unchanged[​](#confirm-the-tenant-object-is-unchanged "Direct link to Confirm the tenant object is unchanged") Tenant Cluster The tenant object still reads the selectors it was created with, not the composed values from the control plane cluster: Read the tenant selectors ``` kubectl --context="${VCLUSTER_CTX}" -n demo get networkpolicies.projectcalico.org \ deny-web-ingress \ -o jsonpath='{.spec.selector}{"\n"}{.spec.ingress[0].source.selector}{"\n"}' ``` Tenant selectors ``` app == 'web' role == 'client' ``` ### Limitations[​](#limitations "Direct link to Limitations") * Keep the brackets around `value`. Calico gives `||` the lowest precedence, so `app == 'web' || all()` composed without brackets becomes `app == 'web' || (all() && )`, and the first half escapes the scope term. Selector grammar belongs to the extension API server, so check its precedence rules. * Don't patch `notSelector` by appending the scope term. Calico negates the whole `notSelector`, which turns the result into "not X, or outside this namespace" and admits other namespaces' workloads as peers. This guide doesn't translate `notSelector` semantics. * This configuration doesn't translate `namespaceSelector`. It selects over namespace labels, but tenant namespace objects don't exist separately on the control plane cluster in single-namespace mode. Don't use `namespaceSelector` in policies synced with this configuration. * This configuration doesn't translate `serviceAccountSelector`, rule-level `serviceAccounts`, or `services` references. Don't use those fields unless you add resource-specific translations and verify the resulting policy on the control plane cluster. * The positive rule-selector patches limit matches to synced workload endpoints that carry `vcluster.loft.sh/namespace`. They don't preserve selectors intended to match a Calico NetworkSet or HostEndpoint. * A rule with no `source` or `destination` selector at all isn't scoped either. Calico matches an empty entity rule against every endpoint in scope. For a namespaced policy, that scope is its own namespace, so an unscoped rule admits the same cross-tenant match this guide exists to prevent. A path-specific patch can't add a selector to a rule that doesn't have one, so avoid rules that rely on an implicit match-all peer. * Roll out or restart the vCluster control plane after adding these patches. The [empty-path patch](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md#empty-paths) runs unconditionally, so reconciliation retrofits every top-level `spec.selector`, including an omitted selector. Existing rule selectors are patched only when that rule changes; update or recreate those policies after the rollout. ### Apply this to other resources[​](#apply-this-to-other-resources "Direct link to Apply this to other resources") Any aggregated API resource that decides its scope through an opaque string field needs the same treatment, using the terms and grammar of that resource's own selector language. See [Patching synced resources](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/patching.md) for the full expression syntax. --- # Sync a namespaced custom resource from the control plane cluster Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** This guide walks through syncing a namespaced custom resource from the control plane cluster to a tenant cluster, using an example CRD. For the config reference, see [Custom resources from the control plane cluster](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/custom-resources.md). ### Set up cluster contexts[​](#set-up-cluster-contexts "Direct link to Set up cluster contexts") Setting up the control plane and tenant cluster contexts makes it easier to switch between them. set up kubectl contexts ``` export HOST_CTX="your-host-context" export VCLUSTER_CTX="vcluster-ctx" ``` Then, create a namespace in your control plane cluster. This example uses `foobar2`: create namespace ``` kubectl --context="${HOST_CTX}" create namespace foobar2 ``` tip You can find your contexts by running `kubectl config get-contexts` ### Create a CustomResourceDefinition in the control plane cluster[​](#create-a-customresourcedefinition-in-the-control-plane-cluster "Direct link to Create a CustomResourceDefinition in the control plane cluster") Save the following CustomResourceDefinition: example-crd.yaml ``` apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: examples.demo.loft.sh spec: group: demo.loft.sh versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: image: type: string replicas: type: integer additionalPrinterColumns: - name: Image type: string description: The image of an example jsonPath: .spec.image - name: Replicas type: integer description: The number of replicas in example jsonPath: .spec.replicas scope: Namespaced names: plural: examples singular: example kind: Example ``` Save this file locally and apply it to the control plane cluster: Create Example CRD in the host ``` kubectl --context="${HOST_CTX}" create -f example-crd.yaml ``` ### Enable from-host syncing for your custom resource[​](#enable-from-host-syncing-for-your-custom-resource "Direct link to Enable from-host syncing for your custom resource") Enable from-host syncing for `Example` custom resources in your tenant cluster configuration: Enable from host syncing for CustomResource ``` sync: fromHost: customResources: examples.demo.loft.sh: enabled: true scope: Namespaced mappings: byName: "foobar2/*": "default/*" ``` This configuration: * Enables from-host syncing of `examples.demo.loft.sh` from the `foobar2` namespace you created earlier. * Automatically configures RBAC permissions for vCluster, so it can get, watch, and list `Example` resources in the `foobar2` namespace. * Syncs all `Examples` from the `foobar2` namespace to the `default` namespace in the tenant cluster. create tenant cluster Create or update a tenant cluster following the [vCluster quick start guide](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md). ### Sync a namespaced custom resource to the tenant cluster[​](#sync-a-namespaced-custom-resource-to-the-tenant-cluster "Direct link to Sync a namespaced custom resource to the tenant cluster") 1. First, create an example resource in the control plane cluster. Copy this file and save it locally as `example-cr.yaml`: ``` apiVersion: demo.loft.sh/v1 kind: Example metadata: name: my-example namespace: foobar2 spec: image: "my-image:latest" replicas: 2 ``` Then create the example in the control plane cluster: Create example in the host ``` kubectl --context="${HOST_CTX}" create -f example-cr.yaml ``` 2. #### Ensure the custom resource synced to the tenant cluster[​](#ensure-the-custom-resource-synced-to-the-tenant-cluster "Direct link to Ensure the custom resource synced to the tenant cluster") The custom resource should now be accessible in the tenant cluster. If you edit a field on the tenant object, it stays until the control plane cluster changes that same field, which then overwrites it. Check the custom resource in the tenant cluster: Get synced CustomResource ``` kubectl --context="${VCLUSTER_CTX}" get examples.demo.loft.sh --namespace default ``` You should see similar output: examples in vCluster ``` NAME IMAGE REPLICAS my-example my-image:latest 2 ``` 3. #### Edit the custom resource in the control plane cluster[​](#edit-the-custom-resource-in-the-control-plane-cluster "Direct link to Edit the custom resource in the control plane cluster") Now edit the example in the control plane cluster and see that the change syncs to the tenant cluster. To set replicas to 4, run: Patch Example CR ``` kubectl --context="${HOST_CTX}" patch examples.demo.loft.sh my-example --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": 4}]' --namespace foobar2 ``` 4. #### Verify the tenant cluster copy updated[​](#verify-the-tenant-cluster-copy-updated "Direct link to Verify the tenant cluster copy updated") Tenant Cluster Check the number of replicas: Check example ``` kubectl --context="${VCLUSTER_CTX}" get --namespace default examples.demo.loft.sh ``` You should see the number of replicas updated from the control plane cluster object: Example updated in vCluster ``` NAME IMAGE REPLICAS my-example my-image:latest 4 ``` --- # Control node IP visibility Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** When vCluster syncs real nodes from the control plane cluster, it obfuscates node IP addresses by default. This behavior protects sensitive information in tenant isolation environments while still providing node features. Node IP obfuscation provides several benefits: * **Security**: Prevents exposure of control plane cluster network topology in tenant isolation environments. * **Privacy**: Protects sensitive infrastructure information from tenant cluster users. * **Isolation**: Maintains logical separation between tenant and control plane clusters. ## Default behavior[​](#default-behavior "Direct link to Default behavior") By default, when you enable real node syncing: Enable real node syncing ``` sync: fromHost: nodes: enabled: true ``` vCluster automatically replaces the actual node IP addresses with obfuscated values. For example, if a host node has IP `192.168.1.100`, it appears in the tenant cluster with a generated IP like `10.0.0.1`. ## Control IP visibility with patches[​](#control-ip-visibility-with-patches "Direct link to Control IP visibility with patches") ### Remove IP information completely[​](#remove-ip-information-completely "Direct link to Remove IP information completely") To remove IP addresses entirely from synced nodes: Remove IP addresses ``` sync: fromHost: nodes: enabled: true patches: - path: status.addresses expression: | [] ``` ## Advanced node information control[​](#advanced-node-information-control "Direct link to Advanced node information control") You can use patches to control other sensitive node information beyond IP addresses: ### Hide system information[​](#hide-system-information "Direct link to Hide system information") Hide system information ``` sync: fromHost: nodes: enabled: true patches: - path: status.nodeInfo.bootID expression: '""' - path: status.nodeInfo.systemUUID expression: '""' - path: status.nodeInfo.machineID expression: '""' ``` ### Selective field obfuscation[​](#selective-field-obfuscation "Direct link to Selective field obfuscation") Selective obfuscation ``` sync: fromHost: nodes: enabled: true patches: - path: status.nodeInfo.osImage expression: '"Linux"' - path: status.nodeInfo.kernelVersion expression: '"Hidden"' ``` ## Use node selectors[​](#use-node-selectors "Direct link to Use node selectors") When using node selectors with obfuscated IPs, focus on labels rather than IP-based selection: Node selector with labels ``` sync: fromHost: nodes: enabled: true selector: labels: node-role.kubernetes.io/worker: "true" environment: "production" ``` This approach ensures your workloads can still be scheduled appropriately without relying on specific IP addresses. ## Troubleshoot node IP issues[​](#troubleshoot-node-ip-issues "Direct link to Troubleshoot node IP issues") ### Verify obfuscation[​](#verify-obfuscation "Direct link to Verify obfuscation") To check if IP obfuscation is working: 1. Compare node information between control plane and tenant clusters: Check control plane cluster node ``` kubectl get nodes -o wide ``` Check tenant cluster node ``` vcluster connect my-vcluster -- kubectl get nodes -o wide ``` 2. Inspect node details: Inspect node details ``` vcluster connect my-vcluster -- kubectl describe node ``` ### Common issues[​](#common-issues "Direct link to Common issues") * **Networking problems**: Some network policies or CNI plugins may require real IP addresses. Test thoroughly when modifying IP obfuscation. * **Monitoring tools**: Tools that rely on node IPs may need adjustment when IPs are obfuscated. * **Service discovery**: Services that use node IPs directly may require alternative approaches. ## Best practices[​](#best-practices "Direct link to Best practices") * **Default to obfuscation**: Keep IP obfuscation enabled unless you have specific requirements for real IPs. * **Use labels**: Rely on node labels rather than IPs for workload scheduling and management. * **Document changes**: Clearly document any modifications to default obfuscation behavior. * **Test thoroughly**: Verify that your applications work correctly with obfuscated IPs before production deployment. --- # Access and expose vCluster ## Access vCluster for control plane running as a container[​](#access-vcluster-for-control-plane-running-as-a-container "Direct link to Access vCluster for control plane running as a container") Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** There are multiple ways how you can access a vCluster with an external application like `kubectl`. #### Connect directly using the CLI[​](#connect-directly-using-the-cli "Direct link to Connect directly using the CLI") Ensure you install the vCluster CLI. * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster-experimental ``` If you installed the CLI using `brew install vcluster`, you should `brew uninstall vcluster` and then install the experimental version. The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download//vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Replace `` with the version you want to download. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download//vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Replace `` with the version you want to download. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download//vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Replace `` with the version you want to download. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download//vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Replace `` with the version you want to download. Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/download//vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Replace `` with the version you want to download. Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Confirm that you've installed the correct version of the vCluster CLI. ``` vcluster --version ``` ``` # Connect and switch the current context to the vCluster vcluster connect my-vcluster -n my-vcluster # Print the kube-config vcluster connect my-vcluster --print # Switch back context vcluster disconnect # Create a separate kube config to use instead of changing the current context vcluster connect my-vcluster --print > kubeconfig.yaml # Execute a command directly with vCluster context without changing the current context vcluster connect my-vcluster -- kubectl get namespaces vcluster connect my-vcluster -- bash ``` Depending on if the vCluster was created within a local Kubernetes cluster or with the `--expose` flag, the CLI either starts port-forwarding or creates a context that can be used directly. If you have manually [exposed the vCluster](#expose-vcluster), you can specify the domain where the vCluster is reachable with the `--server` flag: ``` # Create a kube context that uses https://my-domain.org as endpoint vcluster connect my-vcluster -n my-vcluster --server my-domain.org ``` #### Connect using service accounts[​](#connect-using-service-accounts "Direct link to Connect using service accounts") By default, vCluster updates the current kubeconfig to access the vCluster that contains the default admin client certificate and client key to authenticate to the vCluster. This means that all kubeconfig files generated have cluster admin access within the vCluster. Often this might not be desired. Instead of giving a user admin access to the tenant cluster, you can also use [service account authentication](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#service-account-tokens) to the tenant cluster. Say we want to create a kubeconfig file that only has view access in the tenant cluster. Then you would create a new service account inside the vCluster and assign it the cluster role `view` through a cluster role binding. Then we would generate a service account token and use that instead of the client-cert and client-key inside the kubeconfig. ``` vcluster connect my-vcluster -n my-vcluster --service-account kube-system/my-user --cluster-role view # OR: create a kube context for a cluster admin vcluster connect my-vcluster -n my-vcluster --service-account kube-system/my-user --cluster-role cluster-admin # OR: create a kube context that expires after an hour vcluster connect my-vcluster -n my-vcluster --service-account kube-system/my-user --cluster-role view --token-expiration 3600 ``` This creates a kube context similar to this as well as create the needed service account and cluster role binding: ``` apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0t... server: https://localhost:8443 name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: token: eyJhbGc... ``` As you can see the service account token is used in this kubeconfig here instead of the client-cert and client-key that is used by default. Trying to create a namespace with this config yields the following: ``` export KUBECONFIG=kubeconfig.yaml # This works as we have view access kubectl get namespaces # This won't work kubectl create namespace test Error from server (Forbidden): namespaces is forbidden: User "system:serviceaccount:kube-system:my-user" cannot create resource "namespaces" in API group "" at the cluster scope ``` You can replace the token field in the kubeconfig with any other service account token from inside the vCluster to act as this service account against the vCluster. For more information about service accounts and tokens, refer to the [official Kubernetes documentation](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#service-account-tokens). ## Retrieve the kubeconfig from the vCluster secret[​](#retrieve-the-kubeconfig-from-the-vcluster-secret "Direct link to Retrieve the kubeconfig from the vCluster secret") Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** There might be cases where connecting to a vCluster with the CLI is not feasible or the CLI cannot be installed. For such cases, you can retrieve the vCluster kubeconfig from a secret that is created automatically in the vCluster namespace. The secret is prefixed with `vc-` and ends with the vCluster name, so a vCluster called `my-vcluster` in namespace `test` would create a secret called `vc-my-vcluster` in the namespace `test`. You can retrieve the kubeconfig after the vCluster has started with: ``` kubectl get secret vc-my-vcluster -n test --template={{.data.config}} | base64 --decode ``` The secret holds a kubeconfig in this format: ``` apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0t... server: https://localhost:8443 name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config users: - name: user user: client-certificate-data: LS0tLS... client-key-data: LS0tLS... ``` By default, the server `https://localhost:8443` is used that would work if you port forward the vCluster with: ``` kubectl port-forward my-vcluster-0 -n test 8443 ``` tip You can specify a different namespace where the kubeconfig secret gets created with the `exportKubeConfig.additionalSecrets` field in `vcluster.yaml`. This lets you emit multiple kubeconfig secrets, each with its own namespace. Keep in mind that you have to manually apply RBAC permissions for the vCluster to allow creation and retrieving of secrets in that namespace. See [Access vCluster externally](#access-vcluster-externally) below. ### Access vCluster externally[​](#access-vcluster-externally "Direct link to Access vCluster externally") If you have [exposed the vCluster](#expose-vcluster), you can also tell the vCluster to create the kubeconfig secret with another server endpoint through the `exportKubeConfig` field. For example, if you want to expose a vCluster at `https://my-domain.org`, you can create a `vcluster.yaml` like this: ``` # Make sure vCluster signs the server certs for my-domain.org # and use it in the generated kube config secret. controlPlane: proxy: extraSANs: - my-domain.org statefulSet: scheduling: podManagementPolicy: OrderedReady exportKubeConfig: server: https://my-domain.org:443 ``` `exportKubeConfig` also supports these fields. * `context` sets the context name and `current-context` in the generated kubeconfig. * `insecure` sets `insecure-skip-tls-verify: true` and removes the CA data from the generated kubeconfig. * `serviceAccount.name`, `serviceAccount.namespace` (defaults to `kube-system`), and `serviceAccount.clusterRole` generate a service account token instead of the default admin client certificate. vCluster creates the service account and a cluster role binding to the given `clusterRole` inside the tenant cluster. * `additionalSecrets` emits multiple kubeconfig secrets, each with its own `name`, `namespace`, `server`, `context`, `insecure`, and `serviceAccount`. info `exportKubeConfig.secret` is deprecated in favor of `additionalSecrets`. For example, to export three kubeconfig secrets with different access levels: ``` exportKubeConfig: additionalSecrets: - name: kubeconfig-admin namespace: my-vcluster server: https://my-domain.org:443 - name: kubeconfig-viewer namespace: my-vcluster server: https://my-domain.org:443 serviceAccount: name: my-viewer namespace: kube-system clusterRole: view - name: kubeconfig-insecure namespace: my-vcluster server: https://my-domain.org:443 insecure: true ``` Then you can create or upgrade the vCluster with: ``` vcluster create my-vcluster -n my-vcluster --upgrade --connect=false -f values.yaml ``` Wait until the vCluster has started and you can retrieve the kubeconfig with: ``` kubectl get secret vc-my-vcluster -n my-vcluster --template={{.data.config}} | base64 --decode ``` [YouTube video player](https://www.youtube.com/embed/zI-ekVfHufI) ## Expose vCluster[​](#expose-vcluster "Direct link to Expose vCluster") Exposing vCluster is different, depending on whether the control-plane is deployed [as container](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/basics.md) or [as a binary](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md). * As Container * As Binary For vCluster Standalone, you can use the method of your choice for exposing API Server endpoint, as described in the [Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/). If you decide to use vCluster Standalone as a control plane cluster for tenant clusters, then you can set your current kube context to the vCluster Standalone and create and interact with tenant clusters using vCluster CLI. By default, vCluster is only reachable via port-forwarding in remote clusters. However, this means that you need access to the control plane cluster, where the vCluster is running, in order to access it. To directly access vCluster without port-forwarding, you can use one of the following methods. Local Kubernetes Clusters If you are using a local Kubernetes cluster, such as docker-desktop, rancher-desktop, kind or minikube, vCluster automatically connects to it without the need of port-forwarding. * Gateway API (TLSRoute) * Ingress (Deprecated) * Ingress without SSL (Deprecated) * Load Balancer Service * NodePort Service * From Control Plane Cluster The Gateway API [TLSRoute](https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io/v1alpha2.TLSRoute) is the recommended way to expose the tenant cluster's API server with TLS passthrough. It requires a Gateway API-compatible controller, such as Envoy Gateway, Istio, or Cilium, installed on the Control Plane Cluster with a `Gateway` resource configured for TLS passthrough. For adoption planning, migration caveats, and Gateway API troubleshooting checks, see [Gateway API](https://vcluster.com/docs/vcluster/key-features/gateway-api.md) and [Migrate from ingress-nginx to Gateway API](https://vcluster.com/docs/vcluster/manage/ingress-nginx-to-gateway-api.md). Configure your `vcluster.yaml` to create a `TLSRoute` pointing at an existing `Gateway`: vcluster.yaml ``` controlPlane: proxy: extraSANs: - my-vcluster.example.com tlsRoute: enabled: true host: my-vcluster.example.com parentRefs: - name: my-gateway namespace: my-gateway-namespace ``` Create the tenant cluster: ``` vcluster create my-vcluster -n my-vcluster --connect=false -f vcluster.yaml ``` Retrieve the kube config: ``` vcluster connect my-vcluster -n my-vcluster --print --server=https://my-vcluster.example.com > kubeconfig.yaml ``` Access the tenant cluster: ``` export KUBECONFIG=./kubeconfig.yaml kubectl get ns ``` ingress-nginx is deprecated The [ingress-nginx project](https://kubernetes.github.io/ingress-nginx/) is deprecated upstream. Existing ingress-based configurations continue to work in vCluster v0.35 and vCluster Platform v4.10, but new deployments should use the **Gateway API (TLSRoute)** tab. Migration is manual; see [Migrate from ingress-nginx to Gateway API](https://vcluster.com/docs/vcluster/manage/ingress-nginx-to-gateway-api.md). An [Ingress Controller](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/) with SSL passthrough support provide the best user experience, but there is a workaround if this feature is not natively supported. * [Kubernetes Nginx (DEPRECATED)](https://kubernetes.github.io/ingress-nginx/user-guide/tls/#ssl-passthrough) * [Traefik Proxy](https://doc.traefik.io/traefik/routing/routers/#passthrough) * [Emissary](https://www.getambassador.io/docs/emissary/latest/topics/using/tcpmappings#tls-termination) Make sure your ingress controller is installed and healthy on the cluster that hosts your tenant clusters. Create the following `ingress.yaml` for a vCluster called `my-vcluster` in the namespace `my-vcluster`: ``` apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: # We need the ingress to pass through ssl traffic to the vCluster # This only works for the nginx-ingress (enable via --enable-ssl-passthrough # https://kubernetes.github.io/ingress-nginx/user-guide/tls/#ssl-passthrough ) # for other ingress controllers check their respective documentation. nginx.ingress.kubernetes.io/backend-protocol: HTTPS nginx.ingress.kubernetes.io/ssl-passthrough: "true" nginx.ingress.kubernetes.io/ssl-redirect: "true" name: vcluster-ingress namespace: my-vcluster spec: ingressClassName: nginx # use your ingress class name rules: - host: my-vcluster.example.com http: paths: - backend: service: name: my-vcluster port: number: 443 path: / pathType: ImplementationSpecific ``` Create the resource in the namespace via: ``` kubectl apply -f ingress.yaml ``` Enable SSL Passthrough Feature If you are using the ingress nginx controller, ensure you have [enabled the SSL passthrough feature](https://kubernetes.github.io/ingress-nginx/user-guide/tls/#ssl-passthrough) as it is disabled by default. To enable the SSL Passthrough Feature you can edit the nginx ingress deployment within the nginx namespace. The option that needs to be added is `- --enable-ssl-passthrough` under the container args within spec. It should end up looking something like: ``` spec: containers: - args: - /nginx-ingress-controller - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller - --election-id=ingress-nginx-leader - --controller-class=k8s.io/ingress-nginx - --ingress-class=nginx - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key - --enable-ssl-passthrough ``` SSL passthrough required For this ingress to work properly, enable SSL passthrough since TLS termination must occur at the vCluster level, not at the ingress controller level. If SSL passthrough is unavailable in your environment, refer to the following section for implementing ingress without SSL passthrough. Now create a `vcluster.yaml` to create the vCluster with: ``` controlPlane: proxy: extraSANs: - my-vcluster.example.com ``` Create the tenant cluster with: ``` vcluster create my-vcluster -n my-vcluster --connect=false -f values.yaml ``` Retrieve the kube config using: ``` vcluster connect my-vcluster -n my-vcluster --print --server=https://my-vcluster.example.com > kubeconfig.yaml ``` Access the vCluster: ``` export KUBECONFIG=./kubeconfig.yaml # Run any kubectl command kubectl get ns ``` ingress-nginx is deprecated The annotations in the example below are specific to ingress-nginx, which is deprecated upstream. If you are using a different ingress controller, refer to its documentation for the equivalent configuration. If you cannot configure your ingress controller to use ssl-passthrough, you can also create an ingress similar to this: ``` apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/backend-protocol: HTTPS nginx.ingress.kubernetes.io/ssl-redirect: "true" name: vcluster-ingress namespace: my-vcluster spec: ingressClassName: nginx # use your ingress class name rules: - host: my-vcluster.example.com http: paths: - backend: service: name: my-vcluster port: number: 443 path: / pathType: ImplementationSpecific tls: - hosts: - my-vcluster.example.com ``` With this configuration you must use service account authentication in order to connect as the ingress controller won't be able to resolve the client-cert and client-key which is used by default as authentication method. To create a kube config that uses a service account, run the following command: ``` vcluster connect my-vcluster -n my-vcluster --server=https://my-vcluster.example.com --service-account admin --cluster-role cluster-admin --insecure ``` Access the vCluster: ``` # Run any kubectl command kubectl get ns ``` The easiest way is to use the flag `--expose` in `vcluster create` to tell vCluster to use a LoadBalancer service: ``` # Create a new vCluster with a LoadBalancer vcluster create my-vcluster --expose # Run any kube command in the vCluster kubectl get ns ``` That's it, your vCluster is now externally reachable through a LoadBalancer service. Check the costs first Even though using a LoadBalancer is the easiest option, if you use a cloud provider it is costly to create one Loadbalancer per cluster. Check your cloud vendor about the cost of each LoadBalancer. In general using an Ingress is the most cost effective method. **Manual LoadBalancer service creation** Instead of using the built-in flag `--expose`, you can also create the following `load-balancer.yaml` for a vCluster called `my-vcluster` in the namespace `my-vcluster` yourself: ``` apiVersion: v1 kind: Service metadata: name: vcluster-loadbalancer namespace: my-vcluster spec: selector: app: vcluster release: my-vcluster ports: - name: https port: 443 targetPort: 8443 protocol: TCP type: LoadBalancer ``` Create the resource in the namespace via: ``` kubectl apply -f load-balancer.yaml ``` Find out the external ip via `kubectl get svc vCluster-loadbalancer -n my-vcluster`: ``` kubectl get svc vcluster-loadbalancer -n my-vcluster NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE vcluster-loadbalancer LoadBalancer 10.68.9.239 x.x.x.x 443:32678/TCP 7m15s ``` Now create a `values.yaml` to create the vCluster with: ``` controlPlane: proxy: extraSANs: - x.x.x.x ``` Create the tenant cluster with: ``` vcluster create my-vcluster -n my-vcluster --connect=false -f values.yaml ``` Update the current kube config via: ``` vcluster connect my-vcluster -n my-vcluster --server=https://x.x.x.x # OR: create a separate kube config file instead of updating the current context vcluster connect my-vcluster -n my-vcluster --server=https://x.x.x.x --print > kubeconfig.yaml ``` Access the vCluster: ``` # Run any kube context command kubectl get ns ``` You can also expose the vCluster via a NodePort service. Create the following `nodeport.yaml` for a vCluster called `my-vcluster` in the namespace `my-vcluster`: ``` apiVersion: v1 kind: Service metadata: name: vcluster-nodeport namespace: my-vcluster spec: selector: app: vcluster release: my-vcluster ports: - name: https port: 443 targetPort: 8443 protocol: TCP type: NodePort ``` Create the resource in the namespace via: ``` kubectl apply -f nodeport.yaml ``` Find out the external port via `kubectl get svc vcluster-nodeport -n my-vcluster`: ``` kubectl get svc vcluster-nodeport -n my-vcluster NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE vcluster-nodeport NodePort 10.68.9.75 443:31992/TCP 85s ``` Find out the node ips via `kubectl get nodes -o wide`: ``` NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME gke-cluster-1-default-pool-8f0bb8bb-p6wx Ready 6d v1.20.6-gke.1000 10.156.0.14 x.x.x.x Container-Optimized OS from Google 5.4.104+ containerd://1.4.3 gke-cluster-1-default-pool-8f0bb8bb-vl79 Ready 6d v1.20.6-gke.1000 10.156.0.15 y.y.y.y Container-Optimized OS from Google 5.4.104+ containerd://1.4.3 gke-cluster-1-default-pool-8f0bb8bb-wpkp Ready 6d v1.20.6-gke.1000 10.156.0.16 z.z.z.z Container-Optimized OS from Google 5.4.104+ containerd://1.4.3 ``` Now create a `values.yaml` to create the vCluster with: ``` controlPlane: proxy: extraSANs: - x.x.x.x - y.y.y.y - z.z.z.z ``` Create the tenant cluster with: ``` vcluster create my-vcluster -n my-vcluster --connect=false -f values.yaml ``` Retrieve the kube config via: ``` vcluster connect my-vcluster -n my-vcluster --server=https://x.x.x.x --print > kubeconfig.yaml ``` Access the vCluster: ``` export KUBECONFIG=./kubeconfig.yaml # Run any kube context command kubectl get ns ``` To access the tenant cluster from within the control plane cluster, you can directly connect to the vCluster service. Make sure you can access that service and then create a kube config in the following form: ``` vcluster connect my-vcluster -n my-vcluster --server=my-vcluster.my-vcluster --insecure --print > kubeconfig.yaml ``` Now access the tenant cluster with: ``` export KUBECONFIG=./kubeconfig.yaml # Run any kubectl command kubectl get ns ``` [YouTube video player](https://www.youtube.com/embed/0ic7oe8fd4o) --- # Create snapshots Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** There are multiple ways to back up and restore a tenant cluster. vCluster provides a built-in method to create and restore snapshots using its CLI. Scheduled snapshots with vCluster Platform Need automated, scheduled snapshots? [vCluster Platform AutoSnapshots](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/snapshots.md) lets you configure automatic snapshot creation on a cron schedule with retention policies. No need to write custom Kubernetes jobs. warning `vcluster snapshot create` exports the tenant cluster's Kubernetes key/value data through kine, including when kine uses an external MySQL or PostgreSQL database. This isn't a native database backup, and vCluster can't back up or roll back that external database if a later restore fails. Take a native database backup before you rely on `vcluster restore` with an external database. Refer to the relevant database documentation. ## Create a snapshot[​](#create-a-snapshot "Direct link to Create a snapshot") We recommend using the vCluster CLI `vcluster snapshot create` command to back up the [etcd](https://etcd.io/) datastore. info The vCluster snapshot feature does not back up persistent volumes. To back up persistent volumes, use the [Velero backup method](https://vcluster.com/docs/vcluster/manage/backup-restore/velero.md). When you run `vcluster snapshot create`, the CLI submits a snapshot request. The vCluster snapshot controller processes it in the background. Here is an example: ``` vcluster snapshot create myvcluster "container:///data/my-snapshot.tar.gz" 13:52:09 info Beginning snapshot creation... You can check the snapshot creation progress with the following command: vcluster snapshot get myvcluster "container:///data/my-snapshot.tar.gz"' ``` You can check the snapshot creation progress with the following command: ``` vcluster snapshot get myvcluster "container:///data/my-snapshot.tar.gz" SNAPSHOT | STATUS | AGE ---------------------------------------+-----------+------ container:///data/my-snapshot.tar.gz | Completed | 28s ``` The vCluster snapshot controller automatically determines the configured backing store and saves the snapshot at the specified location. The snapshot includes: * Backing store data (such as etcd or SQLite) * vCluster Helm release information * vCluster configuration (for example, `vcluster.yaml`) ### Standalone[​](#standalone "Direct link to Standalone") Standalone vCluster runs as a systemd service directly on a Linux host, without a Kubernetes cluster. Use the `--standalone` flag instead of a cluster name to target the local standalone installation. This flag can't be used with `--driver docker`. Modify the following with your specific values to generate a copyable command: BUCKETmy-s3-bucket SNAPSHOT\_KEYmy-bucket-key Create a snapshot on a standalone vCluster ``` vcluster snapshot create --standalone "s3://my-s3-bucket/my-bucket-key" ``` Check the snapshot status with: Modify the following with your specific values to generate a copyable command: BUCKETmy-s3-bucket SNAPSHOT\_KEYmy-bucket-key Check snapshot status on a standalone vCluster ``` vcluster snapshot get --standalone "s3://my-s3-bucket/my-bucket-key" ``` External storage for cross-node recovery The `container://` protocol stores the snapshot on the local filesystem of the standalone node. For disaster recovery scenarios where you need to restore to a different node, use an external storage backend such as [OCI](#store-snapshots-in-oci-image-registries), [S3](#store-snapshots-in-s3-buckets), or [Azure Blob Storage](#store-snapshots-in-azure-blob-storage). ### Snapshot URL[​](#snapshot-url "Direct link to Snapshot URL") vCluster uses a snapshot URL to save the snapshot to a specific location. The snapshot URL contains the following information: | Parameter | Description | Example | | ---------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Protocol | Defines the storage type for the snapshot | `oci`, `s3`, `container`, `https` (Azure Blob Storage) | | Storage location | Specifies where to save the snapshot | `oci://ghcr.io/my-user/my-repo:my-tag`, `s3://my-s3-bucket/my-snapshot-key`, `https://myaccount.blob.core.windows.net/my-container/my-snapshot.tar.gz`, `container:///data/my-snapshot.tar.gz` | | Optional flags | Additional options for snapshot storage | `skip-client-credentials=true` | ### Supported protocols[​](#supported-protocols "Direct link to Supported protocols") The following protocols are supported for storing snapshots: * [`oci`](#store-snapshots-in-oci-image-registries) – Stores snapshots in an OCI image registry, such as Docker Hub or GHCR. * [`s3`](#store-snapshots-in-s3-buckets) – Saves snapshots to an S3-compatible bucket, such as AWS S3 or MinIO. * [Azure Blob Storage](#store-snapshots-in-azure-blob-storage) – Saves snapshots to Azure Blob Storage. * [`container`](#store-snapshots-in-containers) – Stores snapshots as a local file inside a vCluster container or another persistent volume claim (PVC). For example, the following snapshot URL saves the snapshot to an OCI image registry: Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster OCI\_REGISTRYghcr.io OCI\_REPOmy-user/my-repo OCI\_TAGmy-tag Snapshot and push to an OCI image registry ``` vcluster snapshot create my-vcluster "oci://ghcr.io/my-user/my-repo:my-tag" ``` #### Store snapshots in OCI image registries[​](#store-snapshots-in-oci-image-registries "Direct link to Store snapshots in OCI image registries") You can save snapshots to OCI image registries. Authenticate using locally stored OCI credentials, or pass credentials directly in the snapshot URL. To authenticate with local credentials, log in to your OCI registry and create the snapshot: Modify the following with your specific values to generate a copyable command: OCI\_REGISTRYghcr.io VCLUSTER\_NAMEmy-vcluster OCI\_REPOmy-user/my-repo OCI\_TAGmy-tag Authenticate with local OCI credentials and create a snapshot ``` # Log in to the OCI registry using a password access token. echo $PASSWORD_ACCESS_TOKEN | docker login ghcr.io -u $USERNAME --password-stdin # Create a snapshot and push it to an OCI image registry. vcluster snapshot create my-vcluster "oci://ghcr.io/my-user/my-repo:my-tag" ``` Alternatively, pass credentials directly in the snapshot URL. The following parameters configure authentication: | Parameter | Description | Required | | ------------------------- | ---------------------------------------------------------------- | ------------------------------------- | | `username` | Username for authenticating with the OCI registry | Yes, when not using local credentials | | `password` | Base64-encoded password for authenticating with the OCI registry | Yes, when not using local credentials | | `skip-client-credentials` | When set to `true`, ignores local Docker credentials | No, defaults to `false` | Modify the following with your specific values to generate a copyable command: OCI\_USERNAMEmy-username OCI\_PASSWORDmy-password VCLUSTER\_NAMEmy-vcluster OCI\_REGISTRYghcr.io OCI\_REPOmy-user/my-repo OCI\_TAGmy-tag Pass OCI credentials directly in the snapshot URL ``` # Pass authentication credentials directly in the URL and create a snapshot. export OCI_USERNAME=my-username export OCI_PASSWORD=$(echo -n "my-password" | base64) vcluster snapshot create my-vcluster "oci://ghcr.io/my-user/my-repo:my-tag?username=$OCI_USERNAME&password=$OCI_PASSWORD&skip-client-credentials=true" ``` #### Store snapshots in S3 buckets[​](#store-snapshots-in-s3-buckets "Direct link to Store snapshots in S3 buckets") Store snapshots in an S3-compatible bucket using the `s3` protocol. This works with AWS S3 and S3-compatible providers such as MinIO or Ceph. These are common in on-premise and air-gapped environments where cloud storage isn't available. You can authenticate in two ways: by using local environment credentials or by passing credentials directly in the URL. To use local environment credentials, log in to AWS CLI, then create and save the snapshot: Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster BUCKETmy-s3-bucket SNAPSHOT\_KEYmy-bucket-key Create and store a snapshot in an S3 bucket using AWS CLI credentials ``` # Check if you are logged in. aws sts get-caller-identity # Create a snapshot and store it in an S3 bucket. vcluster snapshot create my-vcluster "s3://my-s3-bucket/my-bucket-key" ``` Alternatively, you can pass options directly in the snapshot URL. The following options are supported: | Flag | Description | Required | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | | `access-key-id` | Base64-encoded S3 access key ID for authentication | Yes, when not using local credentials | | `secret-access-key` | Base64-encoded S3 secret access key for authentication | Yes, when not using local credentials | | `session-token` | Base64-encoded temporary session token for authentication | Yes, when not using local credentials | | `region` | Region of the S3-compatible bucket | No | | `url` | Base64-encoded custom endpoint URL for S3-compatible providers (such as MinIO or Ceph) | No | | `force-path-style` | Use path-style addressing (`endpoint/bucket`) instead of virtual-hosted-style. Required for most S3-compatible providers. | No, defaults to `false` | | `profile` | AWS profile to use for authentication | No | | `skip-client-credentials` | Skips use of local credentials for authentication | No, defaults to `false` | | `server-side-encryption` | Server-side encryption method (`AES256` for SSE-S3, `aws:kms` for SSE-KMS) | No | | `kms-key-id` | KMS key ID for SSE-KMS encryption | No | Security Credentials passed in the snapshot URL are Base64-encoded, not encrypted. Avoid storing snapshot commands with inline credentials in shell history, scripts, or CI logs. For production environments, prefer local credential-based authentication with `aws configure` or a Kubernetes Secret, and set `skip-client-credentials=false`. Run the following command to create a snapshot and store it in an S3 bucket: * macOS * Linux Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster BUCKETmy-s3-bucket SNAPSHOT\_KEYmy-bucket-key Pass S3 credentials directly in the URL to create a snapshot ``` # Read the AWS credentials from files and encode them with base64 # This allows them to be safely included in the S3 URL export ACCESS_KEY_ID=$(cat my-access-key-id.txt | base64) export SECRET_ACCESS_KEY=$(cat my-secret-access-key.txt | base64) export SESSION_TOKEN=$(cat my-session-token.txt | base64) vcluster snapshot create my-vcluster "s3://my-s3-bucket/my-bucket-key?access-key-id=$ACCESS_KEY_ID&secret-access-key=$SECRET_ACCESS_KEY&session-token=$SESSION_TOKEN" ``` Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster BUCKETmy-s3-bucket SNAPSHOT\_KEYmy-bucket-key Pass S3 credentials directly in the URL to create a snapshot ``` # Read the AWS credentials from files and encode them with base64 # On Linux, the -w 0 flag prevents line wrapping of the encoded output export ACCESS_KEY_ID=$(cat my-access-key-id.txt | base64 -w 0) export SECRET_ACCESS_KEY=$(cat my-secret-access-key.txt | base64 -w 0) export SESSION_TOKEN=$(cat my-session-token.txt | base64 -w 0) vcluster snapshot create my-vcluster "s3://my-s3-bucket/my-bucket-key?access-key-id=$ACCESS_KEY_ID&secret-access-key=$SECRET_ACCESS_KEY&session-token=$SESSION_TOKEN" ``` ##### S3-compatible providers[​](#s3-compatible "Direct link to S3-compatible providers") S3-compatible providers such as MinIO or Ceph require two additional parameters: `url` to specify the custom endpoint and `force-path-style` to use path-style bucket addressing. These providers don't use session tokens, so you can omit that parameter. * macOS * Linux Modify the following with your specific values to generate a copyable command: ENDPOINThttps\://s3.example.com:9000 VCLUSTER\_NAMEmy-vcluster BUCKETmy-bucket SNAPSHOT\_KEYmy-snapshot-key REGIONus-east-1 Store a snapshot with an S3-compatible provider ``` # Base64-encode the credentials and the custom endpoint URL export ACCESS_KEY_ID=$(cat my-access-key-id.txt | base64) export SECRET_ACCESS_KEY=$(cat my-secret-access-key.txt | base64) export ENDPOINT=$(echo -n "https://s3.example.com:9000" | base64) vcluster snapshot create my-vcluster "s3://my-bucket/my-snapshot-key?access-key-id=$ACCESS_KEY_ID&secret-access-key=$SECRET_ACCESS_KEY&url=$ENDPOINT®ion=us-east-1&force-path-style=true" ``` Modify the following with your specific values to generate a copyable command: ENDPOINThttps\://s3.example.com:9000 VCLUSTER\_NAMEmy-vcluster BUCKETmy-bucket SNAPSHOT\_KEYmy-snapshot-key REGIONus-east-1 Store a snapshot with an S3-compatible provider ``` # Base64-encode the credentials and the custom endpoint URL # On Linux, the -w 0 flag prevents line wrapping of the encoded output export ACCESS_KEY_ID=$(cat my-access-key-id.txt | base64 -w 0) export SECRET_ACCESS_KEY=$(cat my-secret-access-key.txt | base64 -w 0) export ENDPOINT=$(echo -n "https://s3.example.com:9000" | base64 -w 0) vcluster snapshot create my-vcluster "s3://my-bucket/my-snapshot-key?access-key-id=$ACCESS_KEY_ID&secret-access-key=$SECRET_ACCESS_KEY&url=$ENDPOINT®ion=us-east-1&force-path-style=true" ``` ##### S3 encryption support[​](#s3-encryption "Direct link to S3 encryption support") vCluster supports server-side encryption for S3 snapshots to meet security requirements. See the [CLI reference](https://vcluster.com/docs/vcluster/cli/vcluster_snapshot) for all available flags. **SSE-S3 (AES256)** Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster BUCKETmy-bucket SNAPSHOT\_KEYmy-snapshot-key ``` vcluster snapshot create my-vcluster "s3://my-bucket/my-snapshot-key" --server-side-encryption AES256 ``` **SSE-KMS** Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster BUCKETmy-bucket SNAPSHOT\_KEYmy-snapshot-key KMS\_KEY\_ID12345678-1234-1234-1234-123456789012 ``` vcluster snapshot create my-vcluster "s3://my-bucket/my-snapshot-key" --kms-key-id "12345678-1234-1234-1234-123456789012" ``` **Example: Bucket policy requiring encryption** Enforce SSE-S3 encryption ``` { "Version": "2012-10-17", "Statement": [{ "Sid": "DenyUnencryptedUploads", "Effect": "Deny", "Principal": "*", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::my-bucket/*", "Condition": { "StringNotEquals": { "s3:x-amz-server-side-encryption": "AES256" } } }] } ``` #### Store snapshots in Azure Blob Storage[​](#store-snapshots-in-azure-blob-storage "Direct link to Store snapshots in Azure Blob Storage") Store snapshots in Azure Blob Storage using the standard Azure Blob HTTPS URL: ``` https://.blob.core.windows.net// ``` vCluster supports three ways to authenticate: * Azure CLI * SAS token * Storage account key Log in with `az login` before running the snapshot command. You must also provide the subscription ID and resource group for the storage account as CLI flags or environment variables (`AZURE_SUBSCRIPTION_ID` and `AZURE_RESOURCE_GROUP`): Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster STORAGE\_ACCOUNTmyaccount CONTAINERmy-container BLOB\_PATHmy-snapshot.tar.gz SUBSCRIPTION\_IDmy-subscription-id RESOURCE\_GROUPmy-resource-group Create and store a snapshot in Azure Blob Storage using Azure CLI credentials ``` # Log in to Azure. az login # Create a snapshot and store it in Azure Blob Storage. vcluster snapshot create my-vcluster "https://myaccount.blob.core.windows.net/my-container/my-snapshot.tar.gz" \ --azure-subscription-id my-subscription-id \ --azure-resource-group my-resource-group ``` If the blob URL contains a SAS token, no Azure CLI login, subscription ID, or resource group is needed: Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster STORAGE\_ACCOUNTmyaccount CONTAINERmy-container BLOB\_PATHmy-snapshot.tar.gz Create and store a snapshot in Azure Blob Storage using a SAS token ``` vcluster snapshot create my-vcluster "https://myaccount.blob.core.windows.net/my-container/my-snapshot.tar.gz?sv=2022-11-02&ss=b&srt=co&sp=rwlacuptfx&se=2027-01-01T00:00:00Z&sig=YOUR_SAS_SIGNATURE" ``` Set the `AZURE_STORAGE_KEY` environment variable to the storage account access key. No subscription ID or resource group is needed: Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster STORAGE\_ACCOUNTmyaccount CONTAINERmy-container BLOB\_PATHmy-snapshot.tar.gz Create and store a snapshot in Azure Blob Storage using a storage account key ``` export AZURE_STORAGE_KEY=my-storage-account-key vcluster snapshot create my-vcluster "https://myaccount.blob.core.windows.net/my-container/my-snapshot.tar.gz" ``` Security When using Azure CLI credentials or a storage account key, the vCluster CLI generates a short-lived SAS token and stores it in a Kubernetes Secret. The snapshot controller reads this token to upload the snapshot to Azure Blob Storage. vCluster deletes the Secret once the snapshot completes. #### Store snapshots in containers[​](#store-snapshots-in-containers "Direct link to Store snapshots in containers") Use the `container` protocol to save snapshots as local files inside a vCluster container. Run the following command to create a snapshot and store it in the specified path inside a container: Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster SNAPSHOT\_PATH/data/my-snapshot.tar.gz Create snapshots inside a vCluster container ``` # Create a snapshot to local vCluster PVC (if using embedded storage). vcluster snapshot create my-vcluster "container:///data/my-snapshot.tar.gz" ``` ## Limitations[​](#limitations "Direct link to Limitations") When taking snapshots and restoring tenant clusters, there are limitations: **Sleeping tenant clusters** * Snapshots require a running vCluster control plane and don't work with sleeping tenant clusters. **Tenant clusters using an external database** * vCluster snapshots export the tenant cluster's Kubernetes key/value data through kine, whether the external database is configured with `dataSource` or `connector`. This isn't a native database backup, and vCluster doesn't back up or roll back the external database itself, so take a native database backup before restoring. See [Restore limitations](https://vcluster.com/docs/vcluster/manage/backup-restore/restore.md#limitations). **Tenant clusters using deployed etcd** * Snapshots aren't supported with deployed etcd (`controlPlane.backingStore.etcd.deploy`), even though `vcluster snapshot create` can still complete against it. Back up etcd with its own tooling instead. See [Back up and restore](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md#back-up-and-restore). **Distribution** * Snapshots work only with [K8s](https://kubernetes.io/). **Cluster Certificates** * vCluster doesn't include cluster certificates in snapshots. --- # Restore snapshots Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** There are multiple ways to back up and restore a tenant cluster. vCluster provides a built-in method to create and restore snapshots using its CLI. A vCluster snapshot includes: * Backing store data (such as [etcd](https://etcd.io/) or SQLite) * vCluster Helm release information * vCluster configuration (for example, `vcluster.yaml`) Extra Steps for Private Nodes For tenant clusters with private nodes, you may need to follow [additional steps](#use-snapshots-with-private-nodes). ## Restore existing tenant cluster from a snapshot[​](#restore-existing-tenant-cluster-from-a-snapshot "Direct link to Restore existing tenant cluster from a snapshot") Restoring from a snapshot pauses the vCluster, scales down all workload pods to 0, and launches a temporary restore pod. Once the restore completes, vCluster resumes and scales all workload pods back up. This process results in temporary downtime while the restore is in progress. Restore failures vCluster pauses the tenant cluster and scales workload pods to 0 before the restore pod attempts to download the snapshot. An invalid snapshot URL or expired credential is therefore discovered after downtime begins. Before restoring, run `vcluster snapshot get` with the same URL and credential options. Continue only if it reports the snapshot as saved. This check doesn't pause the tenant cluster. If the restore fails for this or any other reason, the process stops. You must retry the restore to avoid leaving the tenant cluster in an inconsistent or broken state. Restore a tenant cluster using the following commands. You can use any snapshot option and set the snapshot URL with different credentials. Whichever credentials you provide, vCluster never writes them into the restore pod's spec. It mounts them from a Kubernetes Secret through `secretKeyRef`, and deletes that Secret along with the pod once the restore finishes. ### Restore tenant clusters from OCI[​](#restore-tenant-clusters-from-oci "Direct link to Restore tenant clusters from OCI") * Local credentials * URL credentials Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster OCI\_REGISTRYghcr.io OCI\_USERmy-user OCI\_REPOmy-repo OCI\_TAGmy-tag Restore from an OCI snapshot using local credentials ``` vcluster restore my-vcluster "oci://ghcr.io/my-user/my-repo:my-tag" ``` Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster OCI\_REGISTRYghcr.io OCI\_USERmy-user OCI\_REPOmy-repo OCI\_TAGmy-tag Restore from an OCI snapshot using URL credentials ``` export OCI_USERNAME=my-username export OCI_PASSWORD=$(echo -n "my-password" | base64) vcluster restore my-vcluster "oci://ghcr.io/my-user/my-repo:my-tag?username=$OCI_USERNAME&password=$OCI_PASSWORD&skip-client-credentials=true" ``` ### Restore tenant clusters from S3[​](#restore-tenant-clusters-from-s3 "Direct link to Restore tenant clusters from S3") Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster BUCKETmy-s3-bucket KEYmy-snapshot-key Restore from an S3 snapshot using local credentials ``` vcluster restore my-vcluster "s3://my-s3-bucket/my-snapshot-key" ``` ### Restore tenant clusters from Azure Blob[​](#restore-tenant-clusters-from-azure-blob "Direct link to Restore tenant clusters from Azure Blob") * Azure CLI * SAS token * Storage account key Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster STORAGE\_ACCOUNTmyaccount CONTAINERmy-container BLOB\_PATHmy-snapshot.tar.gz SUBSCRIPTION\_IDmy-subscription-id RESOURCE\_GROUPmy-resource-group Restore from an Azure Blob Storage snapshot using Azure CLI credentials ``` vcluster restore my-vcluster "https://myaccount.blob.core.windows.net/my-container/my-snapshot.tar.gz" \ --azure-subscription-id my-subscription-id \ --azure-resource-group my-resource-group ``` Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster STORAGE\_ACCOUNTmyaccount CONTAINERmy-container BLOB\_PATHmy-snapshot.tar.gz Restore from an Azure Blob Storage snapshot using a SAS token ``` vcluster restore my-vcluster "https://myaccount.blob.core.windows.net/my-container/my-snapshot.tar.gz?sv=2022-11-02&ss=b&srt=co&sp=rwlacuptfx&se=2027-01-01T00:00:00Z&sig=YOUR_SAS_SIGNATURE" ``` Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster STORAGE\_ACCOUNTmyaccount CONTAINERmy-container BLOB\_PATHmy-snapshot.tar.gz Restore from an Azure Blob Storage snapshot using a storage account key ``` export AZURE_STORAGE_KEY=my-storage-account-key vcluster restore my-vcluster "https://myaccount.blob.core.windows.net/my-container/my-snapshot.tar.gz" ``` ### Restore tenant clusters from container[​](#restore-tenant-clusters-from-container "Direct link to Restore tenant clusters from container") Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster PATH/data/my-snapshot.tar.gz Restore from a local container snapshot ``` vcluster restore my-vcluster "container:///data/my-snapshot.tar.gz" ``` ## Restore a standalone vCluster[​](#restore-a-standalone-vcluster "Direct link to Restore a standalone vCluster") For standalone vCluster running as a systemd service, use the `--standalone` flag instead of a cluster name. This flag can't be used with `--driver docker`. The CLI automatically stops the vCluster service, restores the backing store data from the snapshot, and restarts the service. Modify the following with your specific values to generate a copyable command: SNAPSHOT\_PATH/var/lib/vcluster/my-snapshot.tar.gz ``` vcluster restore --standalone "container:///var/lib/vcluster/my-snapshot.tar.gz" ``` You can also restore from any external storage backend: Modify the following with your specific values to generate a copyable command: OCI\_REGISTRYghcr.io OCI\_USERmy-user OCI\_REPOmy-repo OCI\_TAGmy-tag ``` vcluster restore --standalone "oci://ghcr.io/my-user/my-repo:my-tag" ``` Service account tokens invalidated with embedded etcd Restoring a standalone vCluster that uses the embedded etcd backing store can invalidate service account tokens for running system pods, such as CoreDNS, kube-proxy, and the konnectivity agent. The restored etcd data predates any tokens issued after the snapshot was taken, so the [API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) rejects them on the UID check. `kubectl get pods` still reports these pods as `Running`, but in-pod DNS resolution stops working. To restore full capability, manually restart the vCluster control plane again after the automatic restart from `vcluster restore --standalone` completes: ``` systemctl restart vcluster ``` Cross-node recovery Restoring to a different machine leaves stale node details in etcd that require manual cleanup. To make the snapshot available on another node, use an external storage backend (OCI, S3, or Azure Blob Storage). Plan to manually remove stale node resources after restore. ### Restore a standalone vCluster in HA mode[​](#restore-a-standalone-vcluster-in-ha-mode "Direct link to Restore a standalone vCluster in HA mode") Restoring a standalone vCluster configured with multiple etcd peers (HA mode) requires manual intervention. Shut down all nodes except the restore node, then run the restore on that node. The following steps assume a three-node setup where you run the restore on node A: 1. Stop the vCluster service on all nodes except the restore node: ``` # Run on nodes B and C systemctl stop vcluster ``` 2. Run the restore on node A: Modify the following with your specific values to generate a copyable command: SNAPSHOT\_URLsnapshot-url ``` vcluster restore --standalone "snapshot-url" ``` 3. Delete the stale node objects for the stopped nodes from the restored cluster: ``` kubectl delete node ``` 4. On nodes B and C, move the old vCluster state aside so they rejoin cleanly: ``` # Run on nodes B and C mv /var/lib/vcluster /var/lib/vcluster-bcp ``` 5. On node A, create a join token: ``` vcluster token create --control-plane ``` 6. Use the join script output to rejoin nodes B and C **one at a time**. Wait for each node to reach `Ready` before joining the next to rebuild etcd quorum sequentially. ## Clone a tenant cluster to a new tenant cluster[​](#clone-a-tenant-cluster-to-a-new-tenant-cluster "Direct link to Clone a tenant cluster to a new tenant cluster") You can use snapshots to clone an existing tenant cluster and create a new tenant cluster from that snapshot. When creating a new tenant cluster from a snapshot, it also restores all workloads in the tenant cluster. When you run `vcluster create --restore` without `-f`, `--values`, or `--set`, vCluster reads the Helm configuration from the snapshot before installing the tenant cluster. If it can't access the snapshot, the command exits before the new tenant cluster starts. Providing configuration through one of those flags skips this check because vCluster uses the supplied configuration instead. Restore Failures If the restore fails while using `vcluster create`, vCluster automatically deletes the new tenant cluster. Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster OCI\_REGISTRYghcr.io OCI\_USERmy-user OCI\_REPOmy-repo OCI\_TAGmy-tag ``` # Create a new tenant cluster from an OCI snapshot (uses local credentials). vcluster create my-vcluster --restore oci://ghcr.io/my-user/my-repo:my-tag ``` New name or namespace updates certificates vCluster certificates change when you create a tenant cluster with a new name or namespace. This is expected, as tenant clusters shouldn't share certificates. ## Migrate and override vCluster configuration options to create a new tenant cluster[​](#migrate-and-override-vcluster-configuration-options-to-create-a-new-tenant-cluster "Direct link to Migrate and override vCluster configuration options to create a new tenant cluster") When upgrading tenant clusters, there are a couple of configuration options that aren't supported to change on an existing tenant cluster. For example, changing the backing store type in place isn't supported. See [Backing store](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store.md) for details. To change other options, migrate the tenant cluster by creating a new one from a snapshot and applying updated configuration. The one exception is restoring an embedded etcd snapshot onto a different backing store type, which doubles as a migration path. See [Migrate from embedded etcd to a different backing store](#migrate-from-embedded-etcd-to-a-different-backing-store). When creating a new tenant cluster from a snapshot, it also restores all workloads in the tenant cluster. Restore failures If the restore fails while using `vcluster create`, vCluster automatically deletes the new tenant cluster. Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster OCI\_REGISTRYghcr.io OCI\_USERmy-user OCI\_REPOmy-repo OCI\_TAGmy-tag ``` # Upgrade an existing vCluster by restoring from a snapshot and applying a new vcluster.yaml. # Configuration options in the vcluster.yaml override the options from the snapshot. vcluster create my-vcluster --upgrade -f vcluster.yaml --restore oci://ghcr.io/my-user/my-repo:my-tag ``` New name or namespace updates certificates vCluster certificates change when you create a tenant cluster with a new name or namespace. This is expected, as tenant clusters shouldn't share certificates. ### Migrate from embedded etcd to a different backing store[​](#migrate-from-embedded-etcd-to-a-different-backing-store "Direct link to Migrate from embedded etcd to a different backing store") Restoring an embedded etcd snapshot onto a tenant cluster configured with a different backing store type converts the snapshot automatically. This lets you move a tenant cluster off embedded etcd onto an embedded database, [external database](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/database/external.md), or [deployed etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md), using the same snapshot and restore workflow as any other migration. This migration path is separate from deployed etcd's own snapshot and restore support. Deployed etcd is a supported *target* for a converted embedded-etcd snapshot, but taking a snapshot of a tenant cluster that already runs on deployed etcd, to back up or restore that same cluster, isn't validated. See [Back up and restore](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md#back-up-and-restore). vCluster can't change a tenant cluster's backing store type in place. Restore this into a new tenant cluster (a different name, namespace, or both) rather than the original one: Modify the following with your specific values to generate a copyable command: SOURCE\_VCLUSTER\_NAMEmy-vcluster OCI\_REGISTRYghcr.io OCI\_USERmy-user OCI\_REPOmy-repo OCI\_TAGmy-tag NEW\_VCLUSTER\_NAMEmy-vcluster-external-db NEW\_NAMESPACEvcluster-my-vcluster-external-db ``` # Take a snapshot of the source tenant cluster, which uses embedded etcd. vcluster snapshot create my-vcluster "oci://ghcr.io/my-user/my-repo:my-tag" # Restore it into a new tenant cluster whose vcluster.yaml configures a different backing store. # vCluster detects the etcd snapshot and converts it before restoring. vcluster create my-vcluster-external-db --namespace vcluster-my-vcluster-external-db -f vcluster.yaml --restore oci://ghcr.io/my-user/my-repo:my-tag ``` This guide covers migrating from embedded etcd only. Migrating in the other direction, such as from a database backing store to etcd, isn't documented or tested. ### Limitations[​](#limitations "Direct link to Limitations") When taking snapshots and restoring tenant clusters, there are limitations: **Sleeping tenant clusters** * Snapshots require a running vCluster control plane and don't work with sleeping tenant clusters. **Tenant clusters using an external database** * Restoring onto an external database, whether configured with `dataSource` or `connector`, deletes the existing tenant-cluster key/value records (through kine, key by key) before replaying the snapshot. vCluster doesn't back up or roll back the external database itself. Before running `vcluster restore`, the database owner or administrator must take a native database backup so that a native external database restore can be triggered if the restore fails. **Tenant clusters using deployed etcd** * Restoring a tenant cluster that already runs on deployed etcd (`controlPlane.backingStore.etcd.deploy`) isn't supported. Restore etcd with its own tooling instead. See [Back up and restore](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md#back-up-and-restore). Deployed etcd is a supported target when migrating an embedded etcd snapshot onto a new tenant cluster; see [Migrate from embedded etcd to a different backing store](#migrate-from-embedded-etcd-to-a-different-backing-store). **vcluster.yaml configuration** * Although the vcluster.yaml configuration is backed up, it isn't automatically restored. After restoring a tenant cluster, you must manually reapply your vcluster.yaml configuration. **Standalone: service account tokens invalidated after restore** * Restoring a standalone vCluster that uses the embedded etcd backing store can invalidate service account tokens for system pods, breaking DNS resolution even though the affected pods report as `Running`. Manually restart the vCluster control plane after the restore completes. See [Restore a standalone vCluster](#restore-a-standalone-vcluster). **Standalone: restore to a new node** * If you restore a snapshot on a different node, the standalone instance's etcd contains node details from the original machine. You must manually clean up stale node resources after restore (see [Use snapshots with private nodes](#use-snapshots-with-private-nodes)). **Standalone: tenant cluster workloads aren't restored** * A standalone snapshot only restores the standalone instance's own etcd data. Each tenant cluster running on the standalone instance has its own etcd, so workloads deployed inside tenant clusters aren't restored by a standalone snapshot. **Tenant clusters managed through the Argo CD integration** * The `ArgoCDApplication` object itself is unaffected by a restore, since it lives outside the tenant cluster's snapshot and restore boundary. The resources Argo CD deployed into the tenant cluster are reverted along with everything else. If the application has `selfHeal: true`, Argo CD reapplies the Git revision after it detects the drift, undoing the restore for those resources. `prune: true` is also required to remove any restored resources no longer defined in Git. See [Restoring a tenant cluster doesn't roll back what Argo CD manages](https://vcluster.com/docs/platform/integrations/argocd.md#argocdapplication-and-argocdapplicationtemplate) for details. ## Use snapshots with private nodes[​](#use-snapshots-with-private-nodes "Direct link to Use snapshots with private nodes") The snapshot also includes node resources. Restoring to a different set of nodes requires manual steps for tenant clusters using [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md). With [host nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes.md), vCluster automatically updates node information when nodes change between snapshot and restore. ### Nodes removed between snapshot and restore[​](#nodes-removed-between-snapshot-and-restore "Direct link to Nodes removed between snapshot and restore") When nodes exist in the snapshot but don't exist in the current tenant cluster, manually delete those nodes from the restored tenant cluster. The node shows up in `kubectl get nodes` but doesn't physically exist in the tenant cluster. Run `kubectl delete node [name]` for each removed node. Modify the following with your specific values to generate a copyable command: NODE\_NAMEmy-node ``` export NODE_NAME=my-node kubectl delete node $NODE_NAME ``` ### Nodes added between snapshot and restore[​](#nodes-added-between-snapshot-and-restore "Direct link to Nodes added between snapshot and restore") When nodes joined the cluster after the snapshot, they don't exist in the restored tenant cluster and aren't listed in `kubectl get nodes`. Re-join each private node with the `--force-join` flag. Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEmy-vcluster ``` export VCLUSTER_NAME=my-vcluster # Connect to your vcluster vcluster connect $VCLUSTER_NAME # Create a token vcluster token create --expires=1h ``` Append the `--force-join` flag to the output command before running it on the worker node. Modify the following with your specific values to generate a copyable command: VCLUSTER\_ENDPOINTvcluster-endpoint TOKENtoken ``` curl -sfLk https://vcluster-endpoint/node/join?token=token | sh - -- --force-join ``` --- # Using Velero ## Use Velero[​](#use-velero "Direct link to Use Velero") You can use [Velero](https://velero.io/) to back up tenant clusters. Ensure your cluster supports [volume snapshots](https://kubernetes.io/docs/concepts/storage/volume-snapshots/) to allow Velero to backup persistent volumes and persistent volume claims that save the tenant cluster state. Alternatively, you can use [Velero's file system backup](https://velero.io/docs/main/file-system-backup/) with the Node Agent to back up the tenant cluster state. ### Back up a vCluster[​](#back-up-a-vcluster "Direct link to Back up a vCluster") Install the [Velero CLI](https://velero.io/docs/main/basic-install/), [Velero server components](https://velero.io/docs/v1.8/supported-providers/) and run the following command: Create a Velero backup for a vCluster namespace ``` velero backup create --include-namespaces=my-vcluster-namespace ``` Verify a backup was successfully created with the following command: Verify the Velero backup status ``` velero backup describe ``` This should create an output similar to the following: Velero backup output ``` Name: Namespace: velero Labels: velero.io/storage-location=default Annotations: velero.io/source-cluster-k8s-gitversion=v1.24.0 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=24 Phase: Completed Errors: 0 Warnings: 0 Namespaces: Included: test Excluded: Resources: Included: * Excluded: Cluster-scoped: auto ... ``` ### Restore a vCluster[​](#restore-a-vcluster "Direct link to Restore a vCluster") After creating a backup with the Velero CLI or a scheduled backup, you can restore a vCluster using the Velero CLI: Restore a Velero backup ``` velero restore create --from-backup ``` Verify the restore process using the following command: Verify the Velero backup restore status ``` velero restore logs ``` This restores the vCluster workloads, configuration, and state in the tenant cluster namespace. Moving vCluster Moving a vCluster from one namespace to another can be challenging because some objects, such as cluster role bindings and persistent volumes, contain namespace references. Velero supports namespace mapping, which works in most cases, but use caution—this may not be compatible with all vCluster setups. ### Use Velero inside vCluster[​](#use-velero-inside-vcluster "Direct link to Use Velero inside vCluster") You can use Velero inside vCluster to protect your workloads and data. To use Velero for backups: 1. Enable the [hostpath-mapper](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/host-path-mapper.md) component in vCluster. 2. After enabling `hostpath-mapper`, install the Velero CLI (as described above), connect to your vCluster, and install Velero inside the tenant cluster: Velero inside the tenant cluster ``` velero install --provider --bucket --secret-file --plugins velero/velero-plugin-for-: --use-node-agent ``` note Replace `provider`, `bucket_name`, `secret-file`, and other placeholders with the correct values for your environment. Velero uses Kopia as the default file system backup tool. 3. After installation is complete, check the status of the Velero resources: Verify the status of the Velero resources ``` $ kubectl get all -n velero NAME READY STATUS RESTARTS AGE pod/node-agent-5szkb 1/1 Running 0 118s pod/velero-75c5479dfd-4x7sl 1/1 Running 0 118s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-agent 1 1 1 1 1 118s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/velero 1/1 1 1 119s NAME DESIRED CURRENT READY AGE replicaset.apps/velero-75c5479dfd 1 1 1 119s ``` 4. Create a backup using file system backup: Create a backup with file system backup ``` velero backup create test1 --default-volumes-to-fs-backup ``` 5. Wait for the backup to complete. Once complete, you should see the following output: Sample Velero backup output ``` $ velero backup describe test1 Name: test1 Namespace: velero Labels: velero.io/storage-location=default Annotations: velero.io/source-cluster-k8s-gitversion=v1.25.0+k3s1 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=25 Phase: Completed Errors: 0 Warnings: 0 Namespaces: Included: * Excluded: Resources: Included: * Excluded: Cluster-scoped: auto ... ``` --- # Rotate certificates note Since version 0.31, vCluster automatically rotates client and server leaf certificates before they expire. Manual rotation is only needed if you want to rotate certificates ahead of the automatic schedule, or if you need to rotate the CA certificate. vCluster client and server certificates (also called leaf certificates) remain valid for 1 year from the initial tenant cluster creation. By default, vCluster uses a self-signed Certificate Authority (CA) to sign these certificates. The CA certificate remains valid for 10 years. vCluster provides commands that allow you to check details about the certificates as well as rotate them. info Checking and rotating certificates is only supported for tenant clusters using the k8s distribution. important The tenant cluster Pod must be in a [running](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-state-running) state otherwise the cert commands below abort the operation. In case of certificate expiration during runtime of a tenant cluster it will still be in a running state (though, non-functional) and the rotation can still be performed. ## Check certificate information[​](#check-certificate-information "Direct link to Check certificate information") The `vcluster certs check` command provides the following information for each certificate of the given tenant cluster:
| Field | Description | | ---------- | --------------------------------------------------------------------------------- | | Filename | The name of the certificate file. | | Subject | The common name of the subject of the certificate. | | Issuer | The common name of the issuer of the certificate. | | Expires On | The date when the certificate expires. | | Status | The expiry status of the certificate (either `OK`, `EXPIRED` or `NOT YET VALID`). | The following command shows this information as a table. You can change the format by using `--output`:
Modify the following with your specific values to generate a copyable command: VCLUSTER NAMESPACEmy-vcluster VCLUSTER NAMEmy-vcluster ``` vcluster -n my-vcluster certs check my-vcluster ``` **Example output**: ``` FILENAME | SUBJECT | ISSUER | EXPIRES ON | STATUS -------------------------------+-------------------------------+----------------+------------------------+--------- apiserver-etcd-client.crt | kube-apiserver-etcd-client | etcd-ca | Jul 14, 2026 19:24 UTC | OK apiserver-kubelet-client.crt | kube-apiserver-kubelet-client | kubernetes | Jul 14, 2026 19:24 UTC | OK apiserver.crt | kube-apiserver | kubernetes | Jul 14, 2026 19:24 UTC | OK ca.crt | kubernetes | kubernetes | Jul 12, 2035 19:24 UTC | OK client-ca.crt | kubernetes | kubernetes | Jul 12, 2035 19:24 UTC | OK ca.crt | etcd-ca | etcd-ca | Jul 12, 2035 19:24 UTC | OK healthcheck-client.crt | kube-etcd-healthcheck-client | etcd-ca | Jul 14, 2026 19:24 UTC | OK peer.crt | vcluster | etcd-ca | Jul 14, 2026 19:24 UTC | OK server.crt | vcluster | etcd-ca | Jul 14, 2026 19:24 UTC | OK front-proxy-ca.crt | front-proxy-ca | front-proxy-ca | Jul 12, 2035 19:24 UTC | OK front-proxy-client.crt | front-proxy-client | front-proxy-ca | Jul 14, 2026 19:24 UTC | OK server-ca.crt | kubernetes | kubernetes | Jul 12, 2035 19:24 UTC | OK ``` ## Rotate client and server leaf certificates[​](#rotate-client-and-server-leaf-certificates "Direct link to Rotate client and server leaf certificates") The certificate rotation command performs the following operations: * **Creates PKI backup**: Backs up the existing Public Key Infrastructure (PKI) directory to preserve the current certificate state. * **Generates new certificates**: Creates new client and server leaf certificates signed by the existing Certificate Authority. * **Restarts tenant cluster**: Restarts the tenant cluster to load the new certificates. Control-plane components (API server, etcd, kubelet) require the restart because they cache certificates in memory during startup. Without a restart, these components continue using the expired or soon-to-expire certificates, which causes authentication failures. The restart does not affect running application workloads because the underlying cluster nodes and container runtime remain operational during the tenant cluster restart process.
Modify the following with your specific values to generate a copyable command: VCLUSTER NAMESPACEmy-vcluster VCLUSTER NAMEmy-vcluster ``` vcluster -n my-vcluster certs rotate my-vcluster ``` Client and server leaf certificate rotation is less intrusive compared to [CA rotation](#rotate-ca-certificate). This means that connecting to the tenant cluster with previously issued kubeconfigs continues to work. ## Changing the CA certificate[​](#changing-the-ca-certificate "Direct link to Changing the CA certificate") warning CA rotation replaces the Certificate Authority that validates all certificates in the cluster. This breaks existing client connections and requires regenerating all kubeconfigs. Use this command carefully, as it affects all cluster access. When working with tenant clusters with private nodes, there are [extra steps](#additional-steps-required-for-private-nodes) that need to happen before doing any changes to the CA certificate and after making the changes to the CA certificate. ### Rotate the CA Certificate[​](#rotate-ca-certificate "Direct link to Rotate the CA Certificate") The CA rotation command performs these operations: * **Creates PKI backup**: Backs up the existing certificate directory to enable recovery if needed. * **Generates new Certificate Authority**: Creates a new CA certificate and uses it to sign fresh certificates for all cluster components. * **Restarts tenant cluster**: Restarts the cluster to load the new certificates. If the rotation fails, you can restore the previous PKI from the automatically created backup. For more information, see the documentation on [restoring the previous CA from a backup](#restore-the-previous-ca-from-a-backup) for recovery.
Modify the following with your specific values to generate a copyable command: VCLUSTER NAMESPACEmy-vcluster VCLUSTER NAMEmy-vcluster ``` vcluster -n my-vcluster certs rotate-ca my-vcluster ``` As this replaces the trust anchor completely, everything configured to trust the old CA becomes invalidated (for instance previously issued kubeconfigs). ### Restore the previous CA from a backup[​](#restore-the-previous-ca-from-a-backup "Direct link to Restore the previous CA from a backup") The rotation creates a backup in the tenant cluster's data directory using the following pattern: ``` /data/pki.bak/ ``` warning Use caution when performing the following operations. Verify each command before execution. 1. To restore from the backup, access the tenant cluster syncer container: Modify the following with your specific values to generate a copyable command: VCLUSTER NAMESPACEmy-vcluster VCLUSTER NAMEmy-vcluster ``` kubectl -n my-vcluster exec -ti svc/my-vcluster -c syncer -- sh ``` 2. List the backups and choose the latest: ``` ls -lat /data/pki.bak/ ``` 3. Delete the PKI directory and copy the backup to that location: Modify the following with your specific values to generate a copyable command: BACKUP UNIX TIMESTAMP1752736900 ``` rm -rf /data/pki cp -R /data/pki.bak/1752736900 /data/pki ``` 4. Log out and perform a regular leaf [certificate rotation](#rotate-client-and-server-leaf-certificates): Modify the following with your specific values to generate a copyable command: VCLUSTER NAMESPACEmy-vcluster VCLUSTER NAMEmy-vcluster ``` vcluster -n my-vcluster certs rotate my-vcluster ``` ## Rotate certificates using vCluster Standalone[​](#rotate-certificates-using-vcluster-standalone "Direct link to Rotate certificates using vCluster Standalone") The general advices and warnings mentioned in sections [Rotate client and server leaf certificates](#rotate-client-and-server-leaf-certificates) and [Rotate the CA Certificate](#rotate-ca-certificate) still apply. Though, when running vCluster Standalone the actual steps to perform the rotation differ slightly. 1. Login to the machine where vCluster is currently running. 2. Stop currently running vCluster: ``` systemctl stop vcluster ``` 3. Rotate the certificates: Modify the following with your specific values to generate a copyable command: VCLUSTER\_NAMEstandalone ROTATE\_SUBCOMMANDrotate ``` VCLUSTER_NAME=standalone /var/lib/vcluster/bin/vcluster certs rotate --path=/var/lib/vcluster/pki --standalone ``` 4. Restart vCluster: ``` systemctl start vcluster ``` ## Additional steps required for private nodes[​](#additional-steps-required-for-private-nodes "Direct link to Additional steps required for private nodes") There are additional steps required for private nodes. Depending on how your private worker nodes were added will indicate what needs to be done. There are two types of private work nodes: * **Manually Provisioned Nodes** - These nodes are manually provisioned and joined to the vCluster. The user needs to provision and join these nodes. * **Automatically Provisioned Nodes** - These nodes are automatically provisioned and joined to the vCluster by the vCluster and vCluster Platform. ### Manually provisioned private nodes[​](#manually-provisioned-private-nodes "Direct link to Manually provisioned private nodes") Manually provisioned nodes that are managed outside of the vCluster require additional steps before and after rotating the CA certificate. #### Before changing the CA certificate[​](#before-changing-the-ca-certificate "Direct link to Before changing the CA certificate") important Node deletion needs to happen before any CA cert changes whether it's rotating or restoring the cert. Otherwise, the kubelet running on the node is not able to communicate with the apiserver running with the updated certificate. You must delete all previously joined manually provisioned private nodes. After the rotation, you must re-join the nodes again to the tenant cluster. Each step must be performed on each manually provisioned private node. 1. Connect to your tenant cluster: Modify the following with your specific values to generate a copyable command: VCLUSTER NAMESPACEmy-vcluster VCLUSTER NAMEmy-vcluster ``` vcluster -n my-vcluster connect my-vcluster ``` 2. Delete each manually provisioned private node: info `vcluster node delete` tries to evict any workload before deleting the node from the tenant cluster. If there's no other node for the workload to be re-scheduled to, the workload stops. If you forgot to delete a node before change the CA cert, you must append `--drain=false` to successfully delete the node. Modify the following with your specific values to generate a copyable command: NODE NAMEmy-node ``` vcluster node delete my-node ``` #### After changing the CA certificate[​](#after-changing-the-ca-certificate "Direct link to After changing the CA certificate") After a CA rotation or restoration, you need to re-create a token to join the tenant cluster and join each manually provisioned private node back to the tenant cluster. 1. Connect to your tenant cluster: Modify the following with your specific values to generate a copyable command: VCLUSTER NAMESPACEmy-vcluster VCLUSTER NAMEmy-vcluster ``` vcluster -n my-vcluster connect my-vcluster ``` 2. Create a new join token: ``` vcluster token create ``` 3. Copy the command and append the `--force-join` flag. Run the updated command on every private node. warning The `--force-join` flag leads to running containers being killed via the container runtime and being restarted by the kubelet. If you have a stateful application running that requires graceful termination, please ensure that you have performed the necessary steps beforehand (`vcluster node delete`, backups etc.). ``` curl -fsSLk "https://example.com:443/node/join?token=d34db33f" | sh -s -- --force-join ``` ### Automatically provisioned private nodes[​](#automatically-provisioned-private-nodes "Direct link to Automatically provisioned private nodes") Automatically provisioned nodes that are managed by the vCluster only require additional steps after rotating the CA certificate. #### After changing the CA certificate[​](#after-changing-the-ca-certificate-1 "Direct link to After changing the CA certificate") All logic of the nodes that need to be provisioned is stored in vCluster Platform. After a CA rotation or restoration, you need to delete all `NodeClaims` related to the tenant cluster, but those `NodeClaims` are in the Kubernetes cluster that has vCluster Platform deployed on. 1. Connect to the Kubernetes cluster that is hosting your vCluster Platform so you can run `kubectl` commands. 2. List current `NodeClaims` in vCluster Platform. ``` kubectl get nodeclaims.management.loft.sh -A NAMESPACE NAME STATUS VCLUSTER NODETYPE CREATED AT p-default vcluster-4zbg8 Available vcluster medium 2025-08-28T11:08:18Z p-default vcluster-5xtx3 Available vcluster-2 medium 2025-08-27T10:01:33Z ``` 3. Find the relevant `NodeClaims` for the tenant cluster. From the list of `NodeClaims`, identify the ones associated to your tenant cluster that had the CA certificate change. In this example, if you have rotated the CA certificate of the tenant cluster named `vcluster-2`, then the associated `NodeClaim` is `vcluster-5xtx3`. 4. Delete the `NodeClaim` of the tenant cluster that had a CA change. For the tenant cluster that you had changed the CA certificate, you'll need to delete each `NodeClaim`, which removes the `NodeClaim` and deletes the underlying `Node`. vCluster Platform recreates the `NodeClaim` and the new nodes re-join the tenant cluster using the new CA certificate. Example of deleting the NodeClaim for a single tenant cluster ``` kubectl -n p-default delete nodeclaims.management.loft.sh vcluster-5xtx3 nodeclaim.management.loft.sh "vcluster-5xtx3" deleted ``` --- # Manage custom resources Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** vCluster syncs custom resources between the control plane cluster and a tenant cluster so tenant workloads can use types that Kubernetes core doesn't define. Before you configure `sync.fromHost.customResources` or `sync.toHost.customResources`, decide on a direction and understand what vCluster does differently depending on how the control plane cluster serves the resource. ## Choose a sync direction[​](#choose-a-sync-direction "Direct link to Choose a sync direction") | You want | Direction | Ownership | | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Read-only copies of control-plane-managed resources available to tenant workloads, such as shared `ClusterStores` from external-secrets | [From the control plane cluster](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/custom-resources.md) | The control plane cluster owns the object. A tenant field edit stays local until the control plane cluster changes that same field, which then overwrites it. | | Tenant workloads to create and manage resources that a control plane cluster controller or extension API server acts on, such as an Istio waypoint Gateway | [To the control plane cluster](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/custom-resources.md) | The tenant cluster owns the object. vCluster mirrors it to the control plane cluster. | You can't sync the same resource type in both directions. From-host sync skips any resource already configured for to-host sync. ## CRD-backed vs aggregated API resources[​](#crd-backed-vs-aggregated-api-resources "Direct link to CRD-backed vs aggregated API resources") You don't declare which one you're syncing. vCluster resolves each configured resource against the control plane cluster on startup: * If a CustomResourceDefinition (CRD) backs it, vCluster copies that CRD. * If an extension [API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) serves it through [API aggregation](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/custom-resources.md#aggregated-api-resources) instead, vCluster generates a schemaless CRD for it. Which one applies changes what you get: | | CRD-backed | Aggregated API resource | | ----------------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Schema validation | Tenant API server validates against the copied schema | No schema in the tenant cluster. To-host `create`, `update`, and `patch` requests validate synchronously against the control plane cluster instead; dry runs and subresource requests don't | | Unknown fields | Pruned unless the copied schema allows unknown fields | Preserved, not validated | | To-host writes | Applied to the tenant object, synced to the control plane cluster asynchronously | Written through to the control plane cluster synchronously before the tenant request returns, with rollback on rejection | | Subresources | Whatever the CRD defines | Only the resource and `status`. `scale`, `logs`, `exec`, and similar aren't available in the tenant cluster | The synchronous write-through path means a tenant `create`, `update`, or `patch` on an aggregated resource gets the same extension API server validation it would get by writing directly to the control plane cluster, and a rejected write doesn't come back as a success. It isn't a blanket guarantee: dry runs and subresource requests skip write-through, deletion is asynchronous, and rollback is best-effort. See [Synchronous validation on the control plane cluster](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/custom-resources.md#synchronous-validation-on-the-control-plane-cluster) for the exact behavior and failure modes. ## Lifecycle[​](#lifecycle "Direct link to Lifecycle") **Enabling** a resource for the first time resolves it against the control plane cluster and creates the CRD in the tenant cluster if needed. This resolution only happens on startup. If the control plane cluster isn't serving the resource yet, the tenant cluster fails to start, and vCluster retries on the next restart. **Disabling** a resource, by removing its entry from `customResources` or setting `enabled: false`, takes effect on the next restart or rollout of the vCluster control plane, the same as enabling one. Once it does, the [syncer](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync) for that resource stops, but vCluster doesn't delete the CRD, its objects, or anything it created on the control plane cluster. What happens next depends on the resource type: a copied CRD-backed schema keeps giving the tenant API server resource-specific validation on its own. A generated aggregated-resource CRD never had one, so tenant writes only get Kubernetes' basic object and metadata validation. For a to-host aggregated resource, disabling also removes the write-through validation the extension API server was providing. Delete the CRD yourself if you want it gone. **Upgrading the synced version** of a resource differs by resource type. For a CRD-backed resource, vCluster adds the new version to the tenant CRD and switches the storage version, but any other version already on the CRD keeps whatever `served` setting it had. For a generated aggregated-resource CRD, vCluster explicitly stops serving previously stored versions while still retaining them. See [Versioned example](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/custom-resources.md#versioned-example) for the CRD-backed upgrade sequence. ## Troubleshoot write-through rollback failures[​](#troubleshoot-write-through-rollback-failures "Direct link to Troubleshoot write-through rollback failures") When a to-host write to an aggregated resource fails, whether the extension API server rejects it or the write-through path itself hits a transport, translation, or response-parsing error, vCluster rolls the tenant write back. Rollback is best-effort. A concurrent write can beat it, and it gives up after a few conflicting attempts. Watch the `vcluster_writethrough_rollback_failures_total` metric. An increase doesn't always mean the same failure, and the reason alone doesn't tell you which side is actually clean: * `virtual_delete`, `virtual_update`, `virtual_delete_conflict`, or `virtual_update_conflict`: the tenant-side rollback failed or gave up. A rejected write is likely still sitting in the tenant cluster. If the original write was rejected by the extension API server, the control plane cluster keeps rejecting it, so the syncer retries without ever succeeding. If the original failure was a transient transport error instead, the syncer can succeed once conditions clear. * `unidentified`: vCluster couldn't tell which tenant object to roll back. * `host_lookup`: vCluster couldn't tell whether an object on the control plane cluster belongs to a later, valid write, so it deliberately left the rejected tenant object in place rather than risk deleting a valid one. Treat this as an indeterminate state, not proof that anything was cleaned up. * `host_delete`: vCluster failed to remove a mirror object it created on the control plane cluster. This can fire alongside `virtual_delete` for the same rejected write, so it doesn't by itself mean the tenant side is clean. The metric carries only `resource` and `reason` labels, and its log lines don't include a matching `reason` field either. Filter the control plane's logs for `write-through:` entries for that resource around the time the metric increased, then match the message text: "rollback delete failed," "rollback update failed," or "gave up" point to a tenant-side failure; "cannot find" or "cannot be identified" is the `unidentified` case; "cannot tell whether the object reached the host" is `host_lookup`; "rollback host delete failed" is `host_delete`. To resolve it: 1. Find why the original write needed rolling back, since that determines whether there's anything to inspect on the control plane cluster. Search the control plane's logs for `write-through: rejecting write, rolling back` entries for that resource. Unlike the rollback-failure logs, these carry a `reason` field. * `host_rejected`: the extension API server actually rejected the object. Check the tenant cluster for the retained object, then check the control plane cluster's rejection reason. vCluster rewrites object and namespace names in the error, so match by resource and namespace rather than by the message text. Fix the object in the tenant cluster, or delete it if it shouldn't exist. * Any other reason (`response_error`, `translate_error`, `unparseable_response`, and so on): the failure was local or transient, not a validation rejection. There's no control plane cluster rejection to inspect. Check whether the syncer has since reconciled the retained object, since the underlying condition may have cleared on its own. 2. For `host_lookup` or `host_delete`, don't delete anything on the control plane cluster without confirming identity first. The control plane cluster object carries `vcluster.loft.sh/object-name`, `vcluster.loft.sh/object-namespace`, and `vcluster.loft.sh/object-uid` annotations identifying which tenant object it mirrors. If the tenant object still exists, all three must match its current name, namespace, and UID before you act on the control plane cluster object. If the tenant object is gone, you have no UID left to compare against, and neither the rollback-failure logs nor the rejection logs record one. Without another source that captured the original UID, you can't confirm identity, and shouldn't delete the object. The tenant cluster's [audit log](https://vcluster.com/docs/vcluster/manage/logging.md#audit-logs) only works as that source if its policy logs `create` for this resource at `RequestResponse`. The default policy in the linked setup logs most resources at `Metadata`, which omits the response body and the UID it would contain. There's no automatic retry of an abandoned rollback. ## Related[​](#related "Direct link to Related") * [Custom resources from the control plane cluster](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/custom-resources.md) — configuration reference for from-host sync * [Custom resources to the control plane cluster](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/custom-resources.md) — configuration reference for to-host sync, including write-through behavior * [Sync a namespaced custom resource from the control plane cluster](https://vcluster.com/docs/vcluster/learn-how-to/sync-custom-resource-from-host.md) — worked example * [Sync Calico network policies to the control plane cluster](https://vcluster.com/docs/vcluster/learn-how-to/sync-calico-network-policies.md) — worked example for scoping an opaque selector on to-host sync --- # Deploy configuration changes ## Deploy changes for control plane running as a container[​](#deploy-changes-for-control-plane-running-as-a-container "Direct link to Deploy changes for control plane running as a container") Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** You can always introduce or update your `vcluster.yaml` and deploy new configuration options. The vCluster control plane pod is going to be re-deployed. Limitations on changing configuration You cannot change distros once a tenant cluster is deployed, with one exception: starting with vCluster 0.25.0, migration from K3s to K8s distro is supported. For more details, see the [K3s to K8s migration guide](https://vcluster.com/docs/vcluster/next/manage/upgrade/distro-migration). * vCluster CLI * Helm * Terraform * Argo CD * Cluster API vCluster CLI Version To keep the same version of your vCluster, your CLI needs to be the same version as your virtual cluster. ``` vcluster create --upgrade -n -f vcluster.yaml ``` Replace: * `` with the name of the vCluster instance to update. * `` with the namespace where the vCluster instance is deployed. ``` helm upgrade --install vcluster \ --values vcluster.yaml \ --repo https://charts.loft.sh \ --namespace \ --repository-config='' ``` Replace: * `` with the name of the vCluster instance to update. * `` with the namespace where the vCluster instance is deployed. 1. The steps assume that you have the terraform file that you initially deployed your terraform resource from. Generate a new plan with your updated `vcluster.yaml`. ``` terraform plan ``` Verify that the provider can access your cluster and that the proposed changes are correct. 2. Apply your new changes. ``` terraform apply ``` 1) Commit and push your updated `vcluster.yaml` to your configured ArgoCD repository. 2) Synchronize your ArgoCD repository with your configured cluster. Learn more about [Cluster API Provider for vCluster](https://github.com/loft-sh/cluster-api-provider-vcluster). 1. Install the vCluster provider. ``` clusterctl init --infrastructure vcluster:v0.2.0 ``` 2. Export environment variables to be used by the cluster API provider to create an updated manifest. The manifest will be applied to your Kubernetes cluster, which will update your vCluster with the updated configuration options. ``` # Replace and with values of the vCluster that you are updating export CLUSTER_NAME= export CLUSTER_NAMESPACE= # Since the vcluster.yaml has changed, you'll need to re-export it to the updated vcluster.yaml export VCLUSTER_YAML=$(awk '{printf "%s\\n", $0}' vcluster.yaml) ``` Replace: * `` with the name of the vCluster instance to update. * `` with the namespace where the vCluster instance is deployed. 3. Regenerate the manifest and apply the updated manifest. ``` clusterctl generate cluster ${CLUSTER_NAME} \ --infrastructure vcluster \ --target-namespace ${CLUSTER_NAMESPACE} \ | kubectl apply -f - ``` Kubernetes Version The Kubernetes version for the vCluster is not set at the CAPI provider command. It is configured the `vcluster.yaml` file based on your Kubernetes distribution. 4. Wait for vCluster to be updated by watching for the vCluster custom resource to report a `ready` status. ``` kubectl wait --for=condition=ready vcluster -n $CLUSTER_NAMESPACE $CLUSTER_NAME --timeout=300s ``` ## Deploy changes for vCluster Standalone[​](#deploy-changes-for-vcluster-standalone "Direct link to Deploy changes for vCluster Standalone") Supported Configurations Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. Changing the `vcluster.yaml` configuration has to be done on all control plane nodes. 1. Modify the `vcluster.yaml`. The `vcluster.yaml` is located at: `/etc/vcluster/vcluster.yaml`. Edit and save your changes. 2. Restart vCluster systemd service to deploy your changes. Restart vCluster ``` systemctl restart vcluster.service ``` --- # Migrate from ingress-nginx to Gateway API Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Gateway API is the recommended path for new vCluster endpoint and tenant routing deployments. Existing ingress-based configurations continue to work in vCluster v0.35 and vCluster Platform v4.10, but ingress-nginx is deprecated upstream and should not be the default choice for new rollouts. Migration is explicit. vCluster does not automatically convert existing `Ingress` resources, Platform ingress access settings, or ingress-nginx annotations to Gateway API resources. ## Before you migrate[​](#before-you-migrate "Direct link to Before you migrate") Validate the Gateway API foundation on the Control Plane Cluster: * Gateway API CRDs are installed. * A supported Gateway API controller is installed and healthy. * At least one `GatewayClass` is visible. * A `Gateway` exists for the listener and protocol you need. * DNS points to the Gateway infrastructure. * Certificate ownership and TLS termination or passthrough behavior are clear. * Cross-namespace attachment rules and `ReferenceGrant` resources are planned. For tenant cluster API server exposure, verify that your controller supports `TLSRoute` and TLS passthrough. For workload HTTP routing and Gateway-backed auto sleep, verify `HTTPRoute` support. For Gateway-backed auto sleep wakeup, also verify request mirroring support. ## Recommended rollout[​](#recommended-rollout "Direct link to Recommended rollout") 1. Keep existing ingress configurations running. 2. Install Gateway API CRDs and the selected Gateway controller in a non-production Control Plane Cluster. 3. Create or select a `GatewayClass` and `Gateway` that match the intended production model. 4. Create a test tenant cluster with a Gateway-backed `TLSRoute` endpoint. 5. Enable Gateway API sync for selected test workloads and validate `HTTPRoute` behavior. 6. Test auto sleep and wakeup if the workload depends on Gateway-backed activity detection. 7. Migrate one tenant or workload group at a time. 8. Keep ingress fallback available until DNS, certificates, route status, and wakeup behavior are verified. ## What to check[​](#what-to-check "Direct link to What to check") Before moving production traffic, confirm: * Tenant cluster API access works through the Gateway-backed endpoint. * `TLSRoute` and `HTTPRoute` resources show accepted and programmed status conditions. * The selected controller supports the Gateway API resources and filters you use. * Gateway-backed auto sleep works only when the controller supports `HTTPRoute` request mirroring. * Tenants cannot attach routes to Gateways they should not use. * Shared-node tenants use approved shared Gateways or constrained tenant-created Gateways. * Private nodes, standalone clusters, or externally managed control plane clusters have a clear endpoint and Gateway ownership model. ## Compatibility caveats[​](#compatibility-caveats "Direct link to Compatibility caveats") Gateway API support varies by controller. A controller can accept `HTTPRoute` resources and serve traffic while still lacking request mirroring support for Gateway-backed auto sleep. In that case, Gateway endpoints can still be usable, but automatic wakeup from HTTP traffic is not available through that controller. If Gateway API CRDs are installed after vCluster Platform is already running, restart Platform so it discovers the new resource types and starts Gateway API sleep-mode controllers: Restart vCluster Platform ``` kubectl rollout restart deployment/loft -n vcluster-platform ``` If your Platform release or namespace name differs, adjust the deployment and namespace. ## Fallback strategy[​](#fallback-strategy "Direct link to Fallback strategy") During the deprecation window, you can keep existing ingress-based access while you validate Gateway API. Do not delete ingress resources, DNS records, or ingress-nginx controller configuration until Gateway status, endpoint reachability, certificate behavior, and sleep/wakeup behavior are all confirmed. For new deployments, prefer [Gateway API](https://vcluster.com/docs/vcluster/key-features/gateway-api.md). For the vCluster API server endpoint example, see [Access and expose vCluster](https://vcluster.com/docs/vcluster/manage/accessing-vcluster.md#expose-vcluster). --- # Logging vCluster produces three distinct types of logs. Each type has a different source, a different audience, and different tooling requirements. | Type | Source | Typical access | | ----------------------------------------- | ------------------------------ | ---------------------- | | [Control plane logs](#control-plane-logs) | vCluster pod `stdout` | `kubectl logs` | | [Audit logs](#audit-logs) | Tenant cluster API server | `kubectl logs` or file | | [Workload logs](#workload-logs) | Apps inside the tenant cluster | Log collector | ## Control plane logs[​](#control-plane-logs "Direct link to Control plane logs") The vCluster pod writes operational logs to `stdout`. These cover syncer activity, API request handling, controller reconciliation, and runtime errors. Check these logs first when diagnosing unexpected vCluster behavior. ``` kubectl logs -n -f ``` Example output ``` 2025-06-16 04:43:25 INFO license loader/inject.go:52 initializing license... {"component": "vcluster"} 2025-06-16 04:43:25 INFO license loader/inject.go:59 detected license type {"component": "vcluster", "licenseType": "Online"} 2025-06-16 04:43:29 INFO online/license.go:141 Enabled features: {"component": "vcluster"} 2025-06-16 04:43:29 INFO syncer/syncer.go:84 starting syncer {"component": "vcluster"} ``` By default, logs use a human-readable console format at `info` level. To change the output format or increase verbosity: * [Log encoding](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/logging.md) — switch between console and JSON format * [Enable debug logging](https://vcluster.com/docs/vcluster/learn-how-to/control-plane/container/enable-debug-logging.md) — increase verbosity for troubleshooting ## Audit logs[​](#audit-logs "Direct link to Audit logs") The tenant cluster's Kubernetes API server can record an audit trail of every request it receives. This includes who made the request, what resource it targeted, and what action was taken. These are distinct from [platform audit logs](https://vcluster.com/docs/platform/configure/platform-configs/audit.md). Platform audit logs capture requests to the platform gateway. Tenant cluster audit logs capture requests to the tenant cluster's own Kubernetes API server. To enable audit logging, pass the `--audit-policy-file` flag as an API server extra argument. For a complete setup example with a ConfigMap and volume mount, see [Control plane configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/control-plane.md#32-logging). Each audit event is a single-line JSON object: Example audit event ``` {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a3d3e5f0-1b2c-4d5e-9f0a-1b2c3d4e5f6a","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/secrets","verb":"get","user":{"username":"admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["10.0.0.1"],"objectRef":{"resource":"secrets","namespace":"default","name":"db-credentials","apiVersion":"v1"},"responseStatus":{"code":200},"requestReceivedTimestamp":"2025-06-16T09:20:56.123456Z","stageTimestamp":"2025-06-16T09:20:56.145678Z","annotations":{"authorization.k8s.io/decision":"allow"}} ``` By default, audit events write to `stderr`. To route them to `stdout` for collection with `kubectl logs`, add `--audit-log-path` as a second extra argument: vcluster.yaml ``` controlPlane: distro: k8s: apiServer: extraArgs: - --audit-policy-file=/etc/kubernetes/audit-policy.yaml - --audit-log-path=/dev/stdout ``` ## Workload logs[​](#workload-logs "Direct link to Workload logs") Apps running inside the tenant cluster write logs to `stdout` and `stderr`, the same as in any Kubernetes cluster. You can access them directly through the tenant cluster's API server: ``` kubectl logs -n ``` Example output ``` 2025-06-16T09:21:03Z INFO Server listening on :8080 2025-06-16T09:21:04Z INFO GET /health 200 OK (2ms) 2025-06-16T09:21:10Z ERROR Failed to connect to database: connection refused ``` For centralized collection across multiple tenant clusters, standard log collectors such as Fluentd, Promtail, and the ELK stack need extra configuration. When vCluster syncs pods to the control plane cluster, it rewrites pod names. A pod named `web-6d4f9b` inside the tenant cluster appears as `web-6d4f9b-x-default-x-my-vcluster` on the control plane cluster. Log collectors reading from the node filesystem see only the rewritten name and cannot correlate entries back to the originating workload. The HostPath Mapper solves this by creating symlinks on each node that map physical pod paths back to their virtual names. * vCluster OSS * vCluster Platform Use the [Central HostPath Mapper](https://vcluster.com/docs/platform/maintenance/logging/central-hostpath-mapper.md). It installs a single DaemonSet on the control plane cluster that handles path remapping for all tenant clusters. Two options are available. **Option 1: Per-cluster HostPath Mapper** First, enable the HostPath Mapper in `vcluster.yaml`: vcluster.yaml ``` controlPlane: hostPathMapper: enabled: true ``` Then install the HostPath Mapper DaemonSet in the same namespace as the vCluster: ``` helm install vcluster-hpm vcluster-hpm \ --repo https://charts.loft.sh \ -n \ --set VclusterReleaseName= ``` See [Host path mapper](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/host-path-mapper.md) for version compatibility notes and full configuration options. **Option 2: Collect logs from inside the tenant cluster** Deploy your log collector inside the tenant cluster rather than on the control plane cluster node. Collectors running inside the tenant cluster read logs through the tenant cluster's API server and see virtual pod names directly. No path remapping is needed. --- # Migrate etcd backing store This guide explains how to migrate your vCluster from [deployed (external) etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md) to [embedded etcd](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/embedded.md) without data loss. If you're currently using deployed [etcd](https://etcd.io/) and want to simplify your deployment, reduce resource consumption, or streamline operations, migrating to embedded etcd is a straightforward process that preserves all your data. Scope of this guide This guide covers migration between two etcd deployment modes using the `migrateFromDeployedEtcd` flag. vCluster doesn't support changing the backing store type by editing configuration in place, such as etcd to database or database to etcd. The one other supported migration path is restoring an embedded etcd snapshot onto a tenant cluster configured with an embedded database, external database, or deployed etcd backing store. See [Migrate from embedded etcd to a different backing store](https://vcluster.com/docs/vcluster/manage/backup-restore/restore.md#migrate-from-embedded-etcd-to-a-different-backing-store). ## Overview[​](#overview "Direct link to Overview") The migration from deployed to embedded etcd is a seamless process that: * Preserves all your vCluster data and workloads * Requires minimal downtime (only during the upgrade process) * Automatically handles the data transfer between etcd instances * Cleans up the deployed etcd resources after successful migration Critical requirement The `migrateFromDeployedEtcd` flag is essential for preserving data. Without this flag, your vCluster state will be wiped during the transition. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Run the command `kubectl auth can-i create clusterrole -A` to verify that your current kube-context has administrative privileges. info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm`: Helm v3.10 is required for deploying the platform. Refer to the [Helm Installation Guide](https://Helm.sh/docs/intro/install/) if you need to install it. * `kubectl`: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. - **vCluster CLI** - Required for migration commands * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` - **vCluster version 0.24.2 or later** - Earlier versions don't support this migration - **Existing vCluster with deployed etcd** - This guide assumes you have a vCluster currently using deployed (external) etcd - **Back up your etcd data** - `vcluster snapshot create` isn't supported for deployed etcd, so back it up with its own tooling, such as `etcdctl snapshot save`, before migrating (see [Back up and restore](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md#back-up-and-restore)) - **Storage requirements**: * Storage class available for the embedded etcd PVC (default: 5Gi) * Sufficient storage capacity for your etcd data - **RBAC permissions** - You need permissions to: * Modify StatefulSets and Deployments in the vCluster namespace * Create and delete PVCs * View pod logs for verification ## Migration process[​](#migration-process "Direct link to Migration process") 1. **Verify current backing store** Check that you're using deployed etcd: Modify the following with your specific values to generate a copyable command: NAMESPACEvcluster-my-team VCLUSTER NAMEmy-vcluster ``` kubectl get statefulset -n vcluster-my-team my-vcluster-etcd ``` You should see the deployed etcd StatefulSet running. 2. **Create migration configuration** Create a new values file that enables embedded etcd with the migration flag. Since embedded etcd runs inside the vCluster StatefulSet pod, storage is configured under `controlPlane.statefulSet.persistence` (not under the `backingStore` section like deployed etcd). The `migrateFromDeployedEtcd: true` flag tells vCluster to copy data from the deployed etcd before switching to embedded mode. During migration, both etcd instances temporarily coexist, with the deployed etcd remaining available as a fallback. migrate-to-embedded.yaml ``` controlPlane: backingStore: etcd: embedded: enabled: true migrateFromDeployedEtcd: true # Optional: Configure embedded etcd settings extraArgs: - '--auto-compaction-mode=periodic' - '--auto-compaction-retention=30m' - '--quota-backend-bytes=8589934592' statefulSet: persistence: volumeClaim: size: 10Gi # storageClassName: "your-storage-class" # Optional ``` Admission controllers If your cluster uses admission controllers (such as Kyverno or OPA Gatekeeper), ensure they allow StatefulSet spec modifications and volume claim changes. You may need to add temporary exceptions for the vCluster namespace during migration. 3. **Apply the migration configuration** Upgrade your vCluster with the migration configuration. Note that you should not use `--reuse-values` for the migration, as it will cause a conflict between the deployed and embedded etcd configurations. Instead, include all necessary values in your migration configuration file. Modify the following with your specific values to generate a copyable command: VCLUSTER NAMEmy-vcluster NAMESPACEvcluster-my-team VCLUSTER VERSION0.27.0 ``` helm upgrade my-vcluster vcluster \ --repo https://charts.loft.sh \ --namespace vcluster-my-team \ --version 0.27.0 \ -f migrate-to-embedded.yaml \ --wait ``` Migration duration The migration typically completes within 1-5 minutes, depending on: * The size of your etcd database * Network latency between pods * Storage performance 4. **Verify successful migration** Check the vCluster logs for the success message: Modify the following with your specific values to generate a copyable command: NAMESPACEvcluster-my-team VCLUSTER PODmy-vcluster-0 ``` kubectl logs -n vcluster-my-team my-vcluster-0 | grep -i "migrated" ``` You should see: `Successfully migrated etcd database to embedded etcd` 5. **Remove deployed etcd resources** After confirming successful migration, remove the migration flag to clean up deployed etcd: final-embedded.yaml ``` controlPlane: backingStore: etcd: embedded: enabled: true # Optional: Configure embedded etcd settings extraArgs: - '--auto-compaction-mode=periodic' - '--auto-compaction-retention=30m' - '--quota-backend-bytes=8589934592' statefulSet: persistence: volumeClaim: size: 10Gi # storageClassName: "your-storage-class" # Optional ``` Apply the final configuration: Modify the following with your specific values to generate a copyable command: VCLUSTER NAMEmy-vcluster NAMESPACEvcluster-my-team VCLUSTER VERSION0.27.0 ``` helm upgrade my-vcluster vcluster \ --repo https://charts.loft.sh \ --namespace vcluster-my-team \ --version 0.27.0 \ -f final-embedded.yaml \ --wait ``` This removes: * The deployed etcd StatefulSet * Associated etcd Services * The deployed etcd pods (they are terminated gracefully) About PVCs The deployed etcd PVCs are NOT automatically deleted to prevent accidental data loss. You can manually delete them after verifying the migration: Modify the following with your specific values to generate a copyable command: NAMESPACEvcluster-my-team VCLUSTER NAMEmy-vcluster ``` kubectl delete pvc -n vcluster-my-team data-my-vcluster-etcd-0 ``` ## Pod deletion behavior[​](#pod-deletion-behavior "Direct link to Pod deletion behavior") During the migration process, pod management follows this sequence: 1. **During initial upgrade** (with `migrateFromDeployedEtcd: true`): * vCluster deployment is replaced with a StatefulSet * Embedded etcd starts within the vCluster pod * Deployed etcd pods remain running for data migration * Data is copied from deployed to embedded etcd * No vCluster workload pods are affected 2. **After migration completion**: * The embedded etcd becomes the primary backing store * Deployed etcd pods continue running but are no longer used * vCluster workloads continue uninterrupted 3. **After removing migration flag**: * Deployed etcd StatefulSet is deleted * Deployed etcd pods are terminated gracefully * PVCs remain for manual cleanup * vCluster workloads are not affected ## Storage considerations[​](#storage-considerations "Direct link to Storage considerations") ### Storage sizing[​](#storage-sizing "Direct link to Storage sizing") Embedded etcd creates a new PVC for storage. The default size is 5Gi, but you can customize it: ``` controlPlane: statefulSet: persistence: volumeClaim: size: 10Gi ``` ### Storage migration[​](#storage-migration "Direct link to Storage migration") The data migration process: * Creates a new PVC for embedded etcd * Copies all data from deployed etcd to embedded etcd * Verifies data integrity before switching over * Maintains the deployed etcd as a fallback until explicitly removed ## Certificate and path differences[​](#certificate-and-path-differences "Direct link to Certificate and path differences") When migrating, be aware of these differences between deployed and embedded etcd: | Aspect | Deployed etcd | Embedded etcd | | -------------------- | ---------------------- | ---------------------------- | | Certificate location | `/run/config/pki/` | `/data/pki/etcd/` | | Data directory | Separate PVC | `/data` in vCluster PVC | | Service endpoint | `-etcd:2379` | `127.0.0.1:2379` (localhost) | | Management | Separate StatefulSet | Part of vCluster pod | ## Resolve common issues[​](#resolve-common-issues "Direct link to Resolve common issues") ### Migration doesn't start[​](#migration-doesnt-start "Direct link to Migration doesn't start") If you don't see the migration message in logs: 1. Verify the deployed etcd is accessible: Modify the following with your specific values to generate a copyable command: NAMESPACEvcluster-my-team VCLUSTER PODmy-vcluster-0 VCLUSTER NAMEmy-vcluster ``` kubectl exec -n vcluster-my-team my-vcluster-0 -- curl -k https://my-vcluster-etcd:2379/health ``` 2. Check for Helm upgrade issues: Modify the following with your specific values to generate a copyable command: VCLUSTER NAMEmy-vcluster NAMESPACEvcluster-my-team ``` helm status my-vcluster -n vcluster-my-team ``` 3. Ensure the configuration is correct: Modify the following with your specific values to generate a copyable command: VCLUSTER NAMEmy-vcluster NAMESPACEvcluster-my-team ``` helm get values my-vcluster -n vcluster-my-team | grep -A5 etcd ``` ### Data appears to be missing[​](#data-appears-to-be-missing "Direct link to Data appears to be missing") If resources seem missing after migration: 1. Verify you used `migrateFromDeployedEtcd: true` 2. Check the migration completed successfully in logs 3. Ensure you're connected to the correct vCluster context 4. Review the vCluster pod logs for any errors ### StatefulSet immutability errors[​](#statefulset-immutability-errors "Direct link to StatefulSet immutability errors") During migration, you might encounter errors about forbidden StatefulSet spec updates: ``` error when patching: StatefulSet.apps "vcluster-name" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden ``` This happens because: * The migration converts a Deployment to a StatefulSet (for embedded etcd) * Kubernetes StatefulSets have strict immutability rules for certain fields * Some fields like `volumeClaimTemplates` cannot be modified after creation **Solution:** 1. Delete the StatefulSet with `--cascade=orphan` to keep pods running: Modify the following with your specific values to generate a copyable command: VCLUSTER NAMEmy-vcluster NAMESPACEvcluster-my-team ``` kubectl delete statefulset my-vcluster -n vcluster-my-team --cascade=orphan ``` 2. Then retry the Helm upgrade - it will recreate the StatefulSet with correct configuration Why cascade=orphan? Using `--cascade=orphan` removes the StatefulSet object but leaves pods running, minimizing downtime during the migration process. ## Next steps[​](#next-steps "Direct link to Next steps") After successfully migrating to embedded etcd, you can further customize your configuration: * **[Configure embedded etcd settings](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/embedded.md)** - Tune performance, set resource limits, and configure advanced options * **[Review deployed etcd configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store/etcd/deploy.md)** - Understand the configuration you migrated from ## Related topics[​](#related-topics "Direct link to Related topics") * [Backup and restore](https://vcluster.com/docs/vcluster/manage/backup-restore/backup.md) - Create snapshots before and after migration * [High availability setup](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/high-availability.md) - Scale your embedded etcd for HA --- # Operate the resource proxy Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. vCluster Platform required This feature requires [vCluster Platform](https://vcluster.com/docs/platform/.md). Both the client and target tenant clusters must be managed as [VirtualClusterInstance](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance/.md) within the platform. The resource proxy lets a client tenant cluster transparently forward custom resource requests to a target tenant cluster, which stores the objects and can run controllers against them. For how it works and how to set it up, see [How the resource proxy works](https://vcluster.com/docs/vcluster/understand/resource-proxy.md) and [Set up the resource proxy](https://vcluster.com/docs/vcluster/learn-how-to/resource-proxy-setup.md). Once it's running, there are two ongoing operational concerns. Keep proxy traffic off the DERP relay, and monitor the proxy so degradation surfaces before callers notice. Keep the client, the target, and the platform on current versions. Peer staleness fixes span both vCluster and the platform's stale peer cleanup, and running a new client against an old platform keeps the degradation risk alive. ## Prefer direct connections over the DERP relay[​](#direct-connections "Direct link to Prefer direct connections over the DERP relay") Proxy traffic between tenant clusters flows over the platform network. Peers first try a direct connection and fall back to relaying through the vCluster Platform DERP server when no direct path can be established. The relay keeps traffic flowing, but it doesn't carry it well. Relayed traffic adds latency and routes all proxy bandwidth through the vCluster Platform deployment. That makes the platform a throughput bottleneck and a shared failure domain for otherwise independent tenant clusters. Adding a connected cluster as a [self-hosted DERP relay](https://vcluster.com/docs/platform/administer/clusters/advanced/derp-relay.md) shortens the relayed path, but a direct path is still better. To keep connections direct: * Open the paths that WireGuard hole punching needs. Allow outbound UDP to the peer endpoints advertised by the platform, outbound UDP to the configured Session Traversal Utilities for NAT (STUN) endpoints on their configured ports, and outbound TCP to ports `80` and `443`. The embedded `tsnet` server doesn't configure a fixed UDP source port. A peer endpoint can be a pod, node, load balancer, or public network address after Network Address Translation (NAT), so don't restrict both UDP rules to the peer pod addresses. Scope the peer rule to the endpoint CIDRs your environment advertises when possible, and scope the STUN rule separately. Environments that only allow TCP `443` egress force everything through the relay. * Review the NAT topology. Symmetric NAT between the clusters defeats hole punching. Clusters in the same VPC or in peered VPCs typically connect directly without special work. * Check NetworkPolicies and security groups applied to the vCluster control plane pods. They must not block the direct path that the platform negotiates. ### NetworkPolicy exceptions for a direct path[​](#direct-connections-networkpolicy "Direct link to NetworkPolicy exceptions for a direct path") The NetworkPolicies that vCluster generates for [`policies.networkPolicy.enabled: true`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md) allow control plane egress on TCP `443`, `8443`, and `6443` to any destination. Separate rules allow egress to the control plane cluster's DNS pods, the tenant cluster's own pods, and the platform pods. The generated policies don't allow the UDP path to another tenant cluster's control plane peers that direct WireGuard connectivity requires. Their ingress rules also don't allow that peer UDP traffic. With the generated policies alone, proxied requests therefore fall back to the DERP relay. For a client and a target on the same control plane cluster, add a rule on each end. Both are required, because one alone still leaves you on the relay. vcluster.yaml - client tenant cluster ``` policies: networkPolicy: enabled: true controlPlane: egress: # Let the resource proxy dial the target's control plane pods directly # instead of falling back to the DERP relay. - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: podSelector: matchLabels: app: vcluster release: ports: - protocol: UDP ``` vcluster.yaml - target tenant cluster ``` policies: networkPolicy: enabled: true controlPlane: ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: podSelector: matchLabels: app: vcluster release: ports: - protocol: UDP ``` The `app: vcluster` label narrows the pod selector to control plane pods, not just anything sharing the release. The `ports` block restricts the rule to UDP without pinning a port number. The embedded `tsnet` server never fixes a listen port, so it binds a different OS-assigned UDP port each session, and pinning one would break on restart. For tenant clusters on different control plane clusters, replace the selectors with an `ipBlock` that covers the endpoint addresses the platform advertises for the peer. Add a separate egress rule for the configured STUN endpoints and ports, and open the firewall paths listed above. ### Confirm which path you are on[​](#confirm-connection-path "Direct link to Confirm which path you are on") No proxy metric distinguishes a direct path from a relayed one. Three indirect signals: * Compare the p99 of `vcluster_proxy_request_duration_seconds` on the client against the same request issued directly against the target. A relayed path shows a consistent latency floor that a direct path doesn't. * Watch `vcluster_proxy_peer_dial_duration_seconds`. Relayed traffic adds a hop, so dial latency on a relayed path is typically higher than on a direct path between the same two clusters. A step change here with no corresponding change on the target is consistent with a fallback to the relay. * Enable `LOFT_LOG_TSNET_CONTROL_PLANE` on the platform for verbose TSNet logging, which covers DERP relay and mesh traffic on the same toggle. See [Enable verbose agent and platform logs](https://vcluster.com/docs/platform/troubleshoot/connected-cluster-connectivity.md#enable-verbose-agent-and-platform-logs). ## Monitor the proxy[​](#monitoring "Direct link to Monitor the proxy") The proxy exposes Prometheus metrics on the client tenant cluster. Scrape them, alert on the starting points below, pair them with a [synthetic probe](#synthetic-probe), and [verify that all three work](#verify-monitoring) before you depend on them during an incident. ### Scrape proxy metrics[​](#scrape-metrics "Direct link to Scrape proxy metrics") The client tenant cluster exposes proxy metrics at the `/metrics/syncer` path of its [API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/), using the same delegated authentication as the other metrics paths. The caller needs `get` on the non-resource URL `/metrics/syncer` inside the tenant cluster. Check proxy metrics manually ``` kubectl get --raw /metrics/syncer | grep vcluster_proxy ``` Run this against the client tenant cluster API. RBAC for a metrics scraper - apply inside the client tenant cluster ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: metrics-syncer-reader rules: - nonResourceURLs: ["/metrics/syncer"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: metrics-syncer-reader roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: metrics-syncer-reader subjects: - kind: ServiceAccount name: namespace: ``` With the Prometheus Operator, setting `controlPlane.serviceMonitor.enabled: true` in the client's `vcluster.yaml` adds a [ServiceMonitor](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/deployment/service-monitor.md) that scrapes `/metrics/syncer` alongside the other control plane endpoints. The ServiceMonitor authenticates with the TLS client certificate from the `vc-` secret, so it needs no extra RBAC. The ClusterRole above is only for scrapers you wire up yourself. ### Metrics reference[​](#metrics-reference "Direct link to Metrics reference") Every series is labeled `target="/"`, and the tables list only additional labels. No series carries per-peer-host labels, because peer hostnames churn with target restarts. Counter series only appear after their first event, so don't alert on counter absence. A missing counter means the event has never happened since the pod started. Most series are proxy-side equivalents of standard Kubernetes API server metrics, so existing `apiserver_*` dashboards and alert patterns translate directly. #### Request metrics[​](#request-metrics "Direct link to Request metrics") ##### vcluster\_proxy\_request\_total[​](#vcluster-proxy-request-total "Direct link to vcluster_proxy_request_total") **Type:** Counter **Labels:** `group_version`, `verb`, `code`, `reason` Traffic and error rate. The `reason` label attributes recognized proxy failure paths, so most 503 spikes are attributable without log access. See [Outcome reasons](#outcome-reasons). ##### vcluster\_proxy\_request\_duration\_seconds[​](#vcluster-proxy-request-duration-seconds "Direct link to vcluster_proxy_request_duration_seconds") **Type:** Histogram **Labels:** `verb` Latency distribution for non-watch requests, with buckets up to the 60-second request deadline. Watches are excluded, because their lifetime would swamp the distribution. Fast 503s with a peer-availability or dial `reason` point at proxy-path degradation; a prompt target-side 5xx can also produce a fast 503. Creeping p99 points at a slow target or a relayed network path. ##### vcluster\_proxy\_request\_terminations\_total[​](#vcluster-proxy-request-terminations-total "Direct link to vcluster_proxy_request_terminations_total") **Type:** Counter **Labels:** `reason` Requests the proxy itself terminated, as distinct from target-side errors: `deadline` (the 60-second cap expired) or `upgrade_rejected` (a rejected connection upgrade). ##### vcluster\_proxy\_inflight\_requests[​](#vcluster-proxy-inflight-requests "Direct link to vcluster_proxy_inflight_requests") **Type:** Gauge Non-watch requests currently being served, for saturation tracking. Watch streams are counted by `vcluster_proxy_active_watches` instead. ##### vcluster\_proxy\_discovery\_fallbacks\_total[​](#vcluster-proxy-discovery-fallbacks-total "Direct link to vcluster_proxy_discovery_fallbacks_total") **Type:** Counter **Labels:** `group_version`, `reason` Discovery requests answered from the [local fallback document](https://vcluster.com/docs/vcluster/understand/resource-proxy.md#degraded-behavior) instead of the target. Callers see a `200`, so this is the dedicated counter for degraded discovery. The matching `vcluster_proxy_request_total` series also carries the fallback reason. See [Discovery lists resources with only a get verb, or an empty list](https://vcluster.com/docs/vcluster/troubleshoot/resource-proxy.md#discovery-fallback). #### Outcome reasons[​](#outcome-reasons "Direct link to Outcome reasons") The `reason` label reuses a shared vocabulary across `vcluster_proxy_request_total`, `vcluster_proxy_request_terminations_total`, `vcluster_proxy_peer_dial_failures_total`, and `vcluster_proxy_discovery_fallbacks_total`, but related series don't always carry the same reason. A request failed by a dial carries that dial's reason. A discovery fallback records the peer-availability reason when the initial check fails, or `upstream_error` when the proxied discovery attempt returns 401, 404, or 5xx. Because the first request reason wins, `request_total` preserves an earlier dial reason even when `discovery_fallbacks_total` records `upstream_error` for the same request. | Reason | Meaning | Also on | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | | `upstream` | Default or unattributed outcome. It normally means the target produced the response, but proxy-local failures without a more specific reason also use this value. | | | `no_peer` | No peer host was available: no peer list has arrived since startup, or the list is empty. | `peer_dial_failures_total`, `discovery_fallbacks_total` | | `stale_peers` | The peer watch subscription has been down longer than the 60-second staleness tolerance, so the cached peer list is no longer trusted. | `peer_dial_failures_total`, `discovery_fallbacks_total` | | `all_quarantined` | Every peer host is quarantined, or the only candidate is mid-trial. | `peer_dial_failures_total`, `discovery_fallbacks_total` | | `timeout` | The 5-second peer dial timeout expired. | `peer_dial_failures_total` | | `deadline` | A deadline the proxy set expired, usually the 60-second non-watch cap. | `request_terminations_total`, `peer_dial_failures_total` | | `canceled` | The caller's own context ended, for example the client disconnected. Not a proxy fault. | `peer_dial_failures_total` | | `peer_departed` | The peer left the peer list mid-dial. | `peer_dial_failures_total` | | `ts_server_unavailable` | The local platform network server or the license failed before any peer was contacted. | `peer_dial_failures_total` | | `error` | Any other dial error. | `peer_dial_failures_total` | | `ownership_masked` | `accessResources: owned` returned 404 because an object lacked the client's ownership labels. It can also mean the preliminary ownership GET returned a non-5xx status other than 200 or 404, such as 401 or an RBAC 403. Usually tenant isolation working as intended; if unexpected, check target `get` permission. See [Existing target resources return 404](https://vcluster.com/docs/vcluster/troubleshoot/resource-proxy.md#resources-not-appearing). | | | `ownership_check_failed` | The proxy couldn't complete the ownership check. A named GET whose response can't be decoded is masked as 404; a named PUT, PATCH, or DELETE whose preliminary GET returns 5xx fails with a retryable 503. Unlike `ownership_masked`, this indicates an operational failure during the check, not isolation working. | | | `namespace_missing` | A namespaced POST, PUT, or PATCH targeted a namespace that doesn't exist in the client tenant cluster. | | | `upgrade_rejected` | A connection upgrade verb was rejected. See [Connection upgrades return 501](https://vcluster.com/docs/vcluster/troubleshoot/resource-proxy.md#connection-upgrades). | `request_terminations_total` | | `upstream_error` | A proxied discovery attempt returned 401, 404, or 5xx after the initial peer check, so the fallback document was served. The response can come from the target or from a transport failure converted to 5xx by the proxy path. | `discovery_fallbacks_total` | #### Peer health metrics[​](#peer-health-metrics "Direct link to Peer health metrics") ##### vcluster\_proxy\_peers[​](#vcluster-proxy-peers "Direct link to vcluster_proxy_peers") **Type:** Gauge **Labels:** `group_version`, `state` Current peer count per state: `healthy`, `quarantined`, or `half_open`. Computed at scrape time, so expired quarantines are already reflected. The proxy emits all three state series on every scrape, including a zero-valued series when a state has no peers. With no healthy peers, the target is degraded: requests fail unless a `half_open` peer succeeds on its single trial dial. Unlike the transition counters, this also catches a pod that starts already degraded. ##### vcluster\_proxy\_peer\_cache\_synced[​](#vcluster-proxy-peer-cache-synced "Direct link to vcluster_proxy_peer_cache_synced") **Type:** Gauge **Labels:** `group_version` Whether the proxy holds a current, trusted peer snapshot (`1`). A fresh empty snapshot still reports `1`, so this metric doesn't guarantee that an endpoint is available; check `vcluster_proxy_peers{state="healthy"}` as well. At `0`, resource requests to that target fail fast with 503, while discovery uses the local fallback. This covers both a peer list that has never arrived and one that aged out after the subscription dropped. ##### vcluster\_proxy\_peer\_list\_age\_seconds[​](#vcluster-proxy-peer-list-age-seconds "Direct link to vcluster_proxy_peer_list_age_seconds") **Type:** Gauge **Labels:** `group_version` Seconds since the last peer list was published. Absent before the first publication. Publication is event-driven, so a high value on its own doesn't mean anything is wrong, and it isn't what triggers the staleness cutoff. See the caveat under [Alert starting points](#alerts). ##### vcluster\_proxy\_peer\_quarantines\_total[​](#vcluster-proxy-peer-quarantines-total "Direct link to vcluster_proxy_peer_quarantines_total") **Type:** Counter A target endpoint was quarantined after 3 consecutive dial failures. A steady increase over tens of minutes means a dead or stale endpoint is re-quarantined every cycle, or the target is flapping. ##### vcluster\_proxy\_target\_all\_peers\_quarantined\_total[​](#vcluster-proxy-target-all-peers-quarantined-total "Direct link to vcluster_proxy_target_all_peers_quarantined_total") **Type:** Counter A quarantine transition left no endpoint selectable for new requests. Resource requests fail fast with 503 while that state persists; discovery uses the local fallback. ##### vcluster\_proxy\_peer\_recoveries\_total[​](#vcluster-proxy-peer-recoveries-total "Direct link to vcluster_proxy_peer_recoveries_total") **Type:** Counter A previously failing or quarantined endpoint recovered on a successful dial. A replacement endpoint from a new peer list starts healthy and doesn't increment this counter. ##### vcluster\_proxy\_peer\_dial\_failures\_total[​](#vcluster-proxy-peer-dial-failures-total "Direct link to vcluster_proxy_peer_dial_failures_total") **Type:** Counter **Labels:** `reason` Every failed dial, not only the third consecutive failure that triggers quarantine. Also counts rejections that never reach a peer (`no_peer`, `stale_peers`, `all_quarantined`, `peer_departed`), which have no matching duration observation. See [Outcome reasons](#outcome-reasons). ##### vcluster\_proxy\_peer\_dial\_duration\_seconds[​](#vcluster-proxy-peer-dial-duration-seconds "Direct link to vcluster_proxy_peer_dial_duration_seconds") **Type:** Histogram Latency of dials that were actually attempted, whether successful or not, with buckets up to the 5-second dial timeout. Creep here is the earliest signal of network path degradation. #### Watch metrics[​](#watch-metrics "Direct link to Watch metrics") ##### vcluster\_proxy\_active\_watches[​](#vcluster-proxy-active-watches "Direct link to vcluster_proxy_active_watches") **Type:** Gauge Open watch streams. Watches are long-running and bypass the 60-second request deadline. ##### vcluster\_proxy\_watch\_events\_total[​](#vcluster-proxy-watch-events-total "Direct link to vcluster_proxy_watch_events_total") **Type:** Counter Events flowing per target for uncompressed JSON watch streams and CBOR sequence watch streams. Compressed watch responses and other content types aren't counted. For supported formats, an open watch whose counter stops increasing can indicate a stalled stream, but an unchanged counter doesn't prove that every watch is silent. ### Alert starting points[​](#alerts "Direct link to Alert starting points") On the per-group/version `== 0` gauge rules, don't aggregate away `group_version` or the scrape target's own pod label. Each group and version keeps its own peer cache, and in an HA deployment so does each replica. A `sum` therefore lets one healthy series hide a degraded one. The target-wide `ResourceProxyPeerChurn` rule is the exception. It uses `max without (group_version, state)` because it intentionally requires every group/version proxy for that target to have zero healthy peers. Aggregating a `rate` and testing `> 0` is safe, because a non-negative sum can't mask a positive member. Prometheus alert rules ``` # A target has had no fully healthy peers for two minutes. Requests are failing # or are limited to single trial dials against half-open peers. - alert: ResourceProxyNoHealthyPeers expr: vcluster_proxy_peers{state="healthy"} == 0 for: 2m labels: {severity: critical} annotations: summary: 'Resource proxy target {{ $labels.target }} ({{ $labels.group_version }}): no fully healthy peers' # No current, trusted peer snapshot: resource requests to this target fail fast # with 503, while discovery uses the local fallback. Covers both "no peer list # has ever arrived" (platform connectivity or license) and "the peer watch # subscription dropped and the cached list aged out past the 60-second # tolerance". - alert: ResourceProxyPeerCacheUnsynced expr: vcluster_proxy_peer_cache_synced == 0 for: 2m labels: {severity: critical} annotations: summary: 'Resource proxy target {{ $labels.target }} ({{ $labels.group_version }}): no usable peer snapshot, requests fail with 503' # A quarantine transition left no peer selectable within the window. # Warning, not critical: a target rolling restart quarantines every old endpoint # and self-heals within seconds. ResourceProxyNoHealthyPeers above is the paging # rule for resource requests that are still failing. - alert: ResourceProxyTargetUnavailable expr: increase(vcluster_proxy_target_all_peers_quarantined_total[10m]) > 0 labels: {severity: warning} annotations: summary: 'Resource proxy target {{ $labels.target }}: quarantine left no selectable peer' # Sustained quarantine churn and no currently healthy peer across any of the # target's group/version proxies: stale peers or a flapping target. Preserve # scrape labels while aggregating the labels that exist only on the peer gauge. - alert: ResourceProxyPeerChurn expr: > increase(vcluster_proxy_peer_quarantines_total[30m]) > 5 and max without (group_version, state) (vcluster_proxy_peers{state="healthy"}) == 0 labels: {severity: warning} annotations: summary: 'Resource proxy target {{ $labels.target }}: peers repeatedly quarantined without recovery' # Discovery is being served from the local fallback: callers see a 200 and a # healthy APIService while real requests fail. Any sustained rate is degraded. - alert: ResourceProxyDiscoveryDegraded expr: sum by (target, group_version) (rate(vcluster_proxy_discovery_fallbacks_total[5m])) > 0 for: 5m labels: {severity: warning} annotations: summary: 'Resource proxy target {{ $labels.target }} ({{ $labels.group_version }}): serving fallback discovery, real requests are likely failing' # Watches are open but no events are flowing: a half-dead stream. # Tune the window to how often your proxied resources actually change, or drop # this rule if they are normally idle for an hour at a time. # Use this rule only when clients use uncompressed JSON or CBOR sequence watch # streams. Other response formats aren't counted and can cause false alerts. - alert: ResourceProxyWatchesSilent expr: > vcluster_proxy_active_watches > 0 unless increase(vcluster_proxy_watch_events_total[1h]) > 0 for: 30m labels: {severity: warning} annotations: summary: 'Resource proxy target {{ $labels.target }}: watches open but no events for an hour' ``` Peer list age isn't a leading indicator It is tempting to alert on `vcluster_proxy_peer_list_age_seconds` crossing the 60-second staleness tolerance, but the two aren't measured from the same instant. The gauge measures time since the last peer list publication, which is event-driven. On a cluster with no peer churn, nothing publishes, and the age grows with the proxy perfectly healthy. The 60-second tolerance is measured from the moment the peer watch subscription drops, and a successful publication resets that clock. Alert on `vcluster_proxy_peer_cache_synced == 0` for the condition that actually fails requests, and treat a sustained high age as a hint to check platform connectivity. Complementary signals worth having: * Caller-observed errors from the client tenant cluster's own API server metrics: `apiserver_request_total{group="",code=~"5.."}`. * APIService availability inside the client tenant cluster: `kube_apiservice_status_condition` if you run kube-state-metrics there, or a scheduled `kubectl get apiservice`. * A [synthetic probe](#synthetic-probe), because counters only move when traffic flows. ### Synthetic probe[​](#synthetic-probe "Direct link to Synthetic probe") Metrics report the proxy's own view. A synthetic request reports what callers experience, including the parts before the proxy, such as the APIService wiring and the aggregation layer. Run a probe like this on a schedule against the client tenant cluster: ConfigMap for the proxy probe - apply on the control plane cluster ``` apiVersion: v1 kind: ConfigMap metadata: name: proxy-probe namespace: vcluster- data: proxy-probe.sh: | #!/bin/sh # Probe one proxied group/version end to end. Exits non-zero on failure. set -eu GROUP_VERSION="example.com/v1" # API group and version, adjust RESOURCE="myresources" # plural resource name, adjust NAMESPACE="default" # adjust # --request-timeout keeps a degraded path from hanging the probe for 60s. kubectl get --raw "/apis/${GROUP_VERSION}/namespaces/${NAMESPACE}/${RESOURCE}?limit=1" \ --request-timeout=15s > /dev/null ``` Do not alert on a single failed run. A 503 during a target rolling restart is expected and self-heals. Schedule the probe instead and alert on the age of its last success, which covers repeated failures without needing to count them across runs. Run the CronJob on the control plane cluster, in the client tenant cluster's namespace, so that the control plane cluster's kube-state-metrics observes it. The default `vc-` kubeconfig points to `https://localhost:8443`, which is only reachable from the vCluster control plane pod. Export an [additional kubeconfig secret](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/export-kube-config.md#export-additional-secrets) that uses the tenant cluster Service instead: vcluster.yaml ``` exportKubeConfig: additionalSecrets: - name: vc--proxy-probe server: "https://.vcluster-.svc:443" ``` The generated secret holds the tenant cluster kubeconfig under the key `config`. If the release name or namespace differs from the example, update both the Service address and the CronJob placeholders. CronJob for the probe - apply on the control plane cluster ``` apiVersion: batch/v1 kind: CronJob metadata: name: proxy-probe namespace: vcluster- spec: schedule: "*/5 * * * *" concurrencyPolicy: Forbid jobTemplate: spec: backoffLimit: 0 template: spec: restartPolicy: Never containers: - name: probe image: registry.k8s.io/kubectl:v1.34.0 # match your cluster's minor version command: ["/bin/sh", "/scripts/proxy-probe.sh"] env: - name: KUBECONFIG value: /kubeconfig/config volumeMounts: - name: kubeconfig mountPath: /kubeconfig readOnly: true - name: script mountPath: /scripts volumes: - name: kubeconfig secret: secretName: vc--proxy-probe - name: script configMap: name: proxy-probe defaultMode: 0755 ``` To deploy the ConfigMap and CronJob without applying them by hand, add both manifests to [`experimental.deploy.host.manifests`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/deploy.md#apply-manifests-on-initialization-in-host) in the client's `vcluster.yaml`. vCluster then applies them into the tenant cluster's namespace on the control plane cluster whenever it initializes. Prometheus alert rule ``` # No successful probe run in three schedule intervals. - alert: ResourceProxyProbeFailing expr: > time() - kube_cronjob_status_last_successful_time{namespace="vcluster-",cronjob="proxy-probe"} > 900 or ( time() - kube_cronjob_created{namespace="vcluster-",cronjob="proxy-probe"} > 900 unless on(namespace, cronjob) kube_cronjob_status_last_successful_time{namespace="vcluster-",cronjob="proxy-probe"} ) labels: {severity: critical} annotations: summary: 'Resource proxy probe has not succeeded for {{ $value }}s' ``` If you use log-based alerts instead, match on recurring 503 request failures and on `context deadline exceeded` errors mentioning `.vcluster.invalid:443` hostnames in the client's vCluster control plane logs. ### Verify the monitoring works[​](#verify-monitoring "Direct link to Verify the monitoring works") All three pieces can silently no-op. A ServiceMonitor can be created without ever scraping the target, alert rules can fail to load, and a probe can be written but never scheduled. Check each one before you need it. Confirm the metrics exist on the client ``` kubectl get --raw /metrics/syncer | grep vcluster_proxy_peers ``` * **Scraping works.** Query `up{job=~".*vcluster.*"}` in Prometheus and confirm the client tenant cluster's control plane target is present and `1`. An absent target is indistinguishable from a healthy proxy in every alert above. * **Rules load.** Run `promtool check rules .yaml`, then confirm the rules are loaded in the Prometheus UI under Status, Rules. * **The probe runs.** Confirm the CronJob has a recent successful run with `kubectl get cronjob proxy-probe -n vcluster-`. Confirm `kube_cronjob_created` exists in Prometheus. The alert uses that series to detect a CronJob that has never succeeded; after the first success, it uses `kube_cronjob_status_last_successful_time`. If both series are absent, kube-state-metrics isn't exposing the CronJob and the alert can't evaluate it. --- # Pause and resume a tenant cluster Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Pause a tenant cluster when you want to stop it on demand and free its compute resources. Resume it when you need it again. Manual pause doesn't start from an inactivity timeout or schedule. For automatic behavior, see [Configure auto sleep](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md). ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Run the command `kubectl auth can-i create clusterrole -A` to verify that your current kube-context has administrative privileges. info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm`: Helm v3.10 is required for deploying the platform. Refer to the [Helm Installation Guide](https://Helm.sh/docs/intro/install/) if you need to install it. * `kubectl`: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. - `vcluster` CLI * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` ## Pause a tenant cluster[​](#pause-a-tenant-cluster "Direct link to Pause a tenant cluster") Run the following command against the Kubernetes context that contains the tenant cluster: Pause a tenant cluster ``` vcluster pause --namespace ``` The command uses the driver in the `vcluster` CLI configuration. The default is `helm`. Select a different driver with `--driver`; supported values are `helm`, `platform`, and `docker`. For a platform-managed tenant cluster, you can identify it by project instead: Pause a platform-managed tenant cluster ``` vcluster pause --driver platform --project ``` With the Platform driver, add `--prevent-wakeup ` to prevent automatic wakeup for a period. Use `--prevent-wakeup 0` to require an explicit wakeup through the CLI or Platform UI. This flag doesn't apply to the Helm or Docker drivers. Without this flag, Kubernetes API requests routed through the platform and supported HTTP traffic can wake the tenant cluster. CLI reference See the [`vcluster pause` CLI reference](https://vcluster.com/docs/vcluster/cli/vcluster_pause) for all flags. ### What pause changes[​](#what-pause-changes "Direct link to What pause changes") The effect of pause depends on the deployment driver: * The Helm driver scales down the tenant cluster control plane and deletes its synced physical workload Pods from the control plane cluster. * The Platform driver puts the tenant cluster instance to sleep. The platform can scale down both its control plane and physical workloads. * The Docker driver stops the tenant cluster control plane, node, and load balancer containers. The tenant cluster's Kubernetes API objects remain, and PersistentVolumeClaims, Services, and other non-workload resources aren't deleted by pause. With the Helm and Platform drivers, resume recreates physical workload Pods the same way as [platform-managed auto sleep](https://vcluster.com/docs/vcluster/key-features/sleep.md#what-happens-during-platform-managed-auto-sleep), including Pods without a Kubernetes workload controller. ## Resume a tenant cluster[​](#resume-a-tenant-cluster "Direct link to Resume a tenant cluster") Resume a paused tenant cluster with the same name and deployment context: Resume a tenant cluster ``` vcluster resume --namespace ``` For a platform-managed tenant cluster identified by project, run: Resume a platform-managed tenant cluster ``` vcluster resume --driver platform --project ``` With the Helm or Platform driver, you can also connect to a paused tenant cluster. Unless wakeup is currently prevented, the connection resumes the tenant cluster automatically. For the Helm driver, run: Connect and resume ``` vcluster connect --namespace ``` For the Platform driver, run: Connect to and resume a platform-managed tenant cluster ``` vcluster connect --driver platform --project ``` The Docker driver requires `vcluster resume` before `vcluster connect`. CLI reference See the [`vcluster resume` CLI reference](https://vcluster.com/docs/vcluster/cli/vcluster_resume) for all flags. ## Manual pause compared with auto sleep[​](#manual-pause-compared-with-auto-sleep "Direct link to Manual pause compared with auto sleep") | Capability | Manual pause | Auto sleep | | ------------------------------ | -------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | Start condition | CLI command | Inactivity timeout, schedule, or manual action | | Automatic wakeup from activity | Helm: CLI connection. Platform: platform-proxied API, supported HTTP traffic, or CLI connection. Docker: none. | Inactivity sleep: Kubernetes API or supported HTTP traffic. Platform-scheduled sleep waits for its wakeup schedule or a manual wakeup. | | Scheduled sleep and wakeup | No | Yes | | Auto-delete after inactivity | Not configured by pause; available separately for platform-managed tenant clusters. | Not configured by auto sleep; available separately for platform-managed tenant clusters. | See [Sleep tenant clusters](https://vcluster.com/docs/vcluster/key-features/sleep.md) for an overview of the available sleep methods. --- # Migrate from K3s to Kubernetes Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Starting in vCluster 0.25.0, you can migrate tenant clusters from the K3s distribution to the standard Kubernetes (K8s) distribution. This migration gives you access to the full Kubernetes feature set while preserving your existing workloads and configuration. K3s removal in v0.33 vCluster v0.33 removes support for the K3s distribution entirely. If your tenant clusters currently use K3s, migrate them to K8s on your current vCluster version before upgrading to v0.33. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Run the command `kubectl auth can-i create clusterrole -A` to verify that your current kube-context has administrative privileges. info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm`: Helm v3.10 is required for deploying the platform. Refer to the [Helm Installation Guide](https://Helm.sh/docs/intro/install/) if you need to install it. * `kubectl`: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. Additionally, you need: * `vCluster` CLI version 0.25.0 or higher * An existing tenant cluster using K3s distro ## Migration process[​](#migration-process "Direct link to Migration process") The migration is triggered automatically when changing the distribution from K3s to Kubernetes (K8s) during an upgrade. During migration process `vCluster` performs these actions: * Detects that you are switching from K3s to K8s * Copies all relevant certificates and configuration from K3s paths to K8s paths * Preserves database content (for both SQLite and embedded etcd backing stores) * Adds an annotation to prevent accidental re-migration * Launches the new control plane with K8s distro All your existing workloads, services, and resources remain available after migration without interruption. Migration limitations The automated migration process only supports moving from the K3s distro to the K8s distro. Migration between other distro types is not supported. This is a one-way process—migrating back from the K8s distro to K3s is not possible. ## Migrate a tenant cluster[​](#migrate-a-virtual-cluster "Direct link to Migrate a tenant cluster") ### Use a vcluster.yaml file[​](#use-vcluster-yaml "Direct link to Use a vcluster.yaml file") If you're managing your tenant cluster with a vcluster.yaml file, modify it to use K8s distro: Original configuration: Original vcluster.yaml with K3s ``` controlPlane: distro: k3s: enabled: true ``` New configuration: Updated vcluster.yaml with K8s ``` controlPlane: distro: k8s: enabled: true ``` Then apply the changes: Apply the updated configuration ``` vcluster create my-vcluster --values vcluster.yaml --upgrade ``` Migration considerations * Migration is one-way: you cannot migrate back from K8s to K3s * The K3s-specific resources and directories are preserved after migration for safety * Migration supports all backing stores. For external databases (such as PostgreSQL or MySQL), certificates are rotated and the control plane pod restarts during migration * High-availability configurations are fully supported ## After migration[​](#after-migration "Direct link to After migration") After a successful migration, the control plane pod uses the Kubernetes distro instead of K3s. You can verify this by checking the container image of the control plane pod, which now shows a Kubernetes image. ## Troubleshoot migration issues[​](#troubleshoot-migration-issues "Direct link to Troubleshoot migration issues") If you encounter issues during migration, check the control plane pod logs for error messages: Check control plane logs ``` kubectl logs -n ``` Look for log entries that include "Migrating K3s distro certificates to K8s" to trace the migration process. --- # Lifecycle policy The following defines vCluster's lifecycle policy for releases and maintenance. vCluster follows a regular release cadence and adheres to [semantic versioning](https://semver.org/). We recommend that users run the latest stable release to stay up to date with new features and security patches. Due to rapid development, each minor version receives active support for 3 months after its release. Active support ends on the End of Support (EOS) date. For 3 months after the EOS date, the version receives only critical security updates. After the End of Life (EOL) date, the version receives no updates. During active support, updates from the active development branch can be backported upon request. Long Term Support (LTS) releases have longer support windows. See [LTS releases](#lts-releases). Versions not on the following list are no longer in any support window. ## vCluster releases[​](#vcluster-releases "Direct link to vCluster releases") | Release | Release Date | End of Support (EOS) Date | End of Life (EOL) Date | Default Kubernetes Version | | :--------------------------------------: | ------------ | ------------------------- | ---------------------- | -------------------------- | | v0.37 (LTS) | 2026 Sep 08 | 2027 Oct 15 | 2028 Apr 15 | v1.36 (k8s) | | v0.36 | 2026 Jul 22 | 2026 Oct 22 | 2027 Jan 22 | v1.36 (k8s) | | v0.35 | 2026 Jun 16 | 2026 Sep 16 | 2026 Dec 16 | v1.36 (k8s) | | *Versions below are no longer supported* | | | | | | v0.34 | 2026 Apr 29 | 2026 Jul 29 | 2026 Oct 29 | v1.35 (k8s) | | v0.33 | 2026 Mar 13 | 2026 Jun 13 | 2026 Sep 13 | v1.35 (k8s) | | v0.32 | 2026 Feb 18 | 2026 May 18 | 2026 Aug 18 | v1.35 (k8s) | | v0.31 | 2026 Jan 29 | 2026 Apr 29 | 2026 Jul 29 | v1.34 (k8s) | | v0.30 | 2025 Oct 28 | 2026 Jan 28 | 2026 Apr 28 | v1.34 (k8s) | | v0.29 | 2025 Oct 01 | 2026 Jan 01 | 2026 Apr 01 | v1.33 (k8s) | | v0.28 | 2025 Sep 08 | 2025 Dec 08 | 2026 Mar 08 | v1.33 (k8s) | | v0.27 | 2025 Aug 12 | 2025 Nov 12 | 2026 Feb 12 | v1.33 (k8s) | | v0.26 | 2025 Jun 26 | 2025 Sep 26 | 2025 Dec 26 | v1.32 (k8s) | | v0.25 | 2025 May 15 | 2025 Aug 15 | 2025 Nov 15 | v1.32 (k8s) | | v0.24 | 2025 Mar 18 | 2025 Jun 16 | 2025 Sep 14 | v1.32 (k8s) | | v0.23 | 2025 Feb 26 | 2025 May 25 | 2025 Aug 26 | v1.32 (k8s) | | v0.22 | 2024 Dec 18 | 2025 Mar 18 | 2025 Jun 18 | v1.31 (k8s) | | v0.21 | 2024 Nov 11 | 2025 Feb 11 | 2025 May 11 | v1.31 (k8s) | | v0.20 | 2024 Aug 14 | 2024 Nov 14 | 2025 Feb 14 | v1.30 (k8s) | | v0.19 | 2024 Feb 12 | 2025 Apr 01\* | 2025 Jul 01 | v1.29 (k3s) | \* **Extended support**: Due to the number of breaking changes from v0.19 to v0.20, the active support period for v0.19 has been extended. Review the [migration guide](https://vcluster.com/docs/vcluster/reference/migrations/0-20-migration.md) to upgrade your v0.19 clusters to v0.20. The default Kubernetes version reflects the distro and version pinned at the initial vX.Y.0 release of each vCluster version, and does not account for changes made in later patch releases. vCluster's default distro changed from k3s to k8s in v0.20. ## LTS releases[​](#lts-releases "Direct link to LTS releases") vCluster v0.37 and vCluster Platform v4.12 are designated LTS releases. LTS support covers the two releases as a pair, because the platform and the clusters it manages are built and tested together. To stay on LTS, run vCluster v0.37 for your clusters on vCluster Platform v4.12. | Phase | Standard release | LTS release | | --------------------------- | ---------------- | ----------- | | Active support | 3 months | 12 months | | End of Support (EOS) period | 3 months | 6 months | During active support, you can request backports of fixes from the active development branch, including support for newer Kubernetes versions. During the EOS period, LTS releases receive only critical security updates. If you need support for a newer Kubernetes version during this period, contact vCluster Labs support. The LTS dates for v0.37 apply only when vCluster v0.37 runs with vCluster Platform v4.12. Without the platform, vCluster v0.37 follows the standard lifecycle, with EOS on 2026 Dec 08 and EOL on 2027 Mar 08. Upgrading either product past its LTS release takes you off LTS. See the [Platform lifecycle policy](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/lifecycle-policy.md) for Platform dates. Support terminology * **End of Support (EOS)**: The period when active support for a version has ended and only critical security updates are provided. This period begins on the EOS date and lasts 3 months, or 6 months for LTS releases. * **End of Life (EOL)**: The period when all support, including security updates, has ended for a version. This period begins on the EOL date and continues indefinitely. ## Kubernetes compatibility matrix[​](#kubernetes-compatibility-matrix "Direct link to Kubernetes compatibility matrix") The matrix below shows compatibility between control plane clusters and tenant clusters running vanilla Kubernetes. It specifies which versions of vanilla Kubernetes on the control plane cluster support which versions on the tenant cluster. When a new Kubernetes version is released, conformance tests are conducted on the latest vCluster version within 30 days. Kubernetes versions that are no longer supported upstream can still be used but are not guaranteed to work. | Control plane cluster Kubernetes version | vCluster Kubernetes version | | | | ---------------------------------------- | --------------------------- | ----- | ----- | | | v1.36 | v1.35 | v1.34 | | **v1.36** | ✅ | 🆗 | 🆗 | | **v1.35** | 🆗 | ✅ | 🆗 | | **v1.34** | 🆗 | 🆗 | ✅ | * ✅ **Tested and verified** – Officially tested and supported. * 🆗 **Likely compatible** – Expected to work, but not officially tested. * ⚠️ **Known issues** – Compatible with documented limitations. ## Feature Versioning[​](#feature-versioning "Direct link to Feature Versioning") While most features we release are fully supported, some are introduced at an earlier stage of development. These stages generally align with [Kubernetes conventions](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-stages), with the exception of [experimental features](https://www.vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/) which may be removed at any time. * **Alpha**: Newly introduced features which may include breaking changes in future releases. * **Beta**: Features that are more stable and considered safe, but not recommended for business critical applications. --- # Upgrade vCluster At any time, you can upgrade the version of your vCluster. During the upgrade, you can also apply configuration option changes. Upgrades should be done one minor version at a time. While larger jumps may be possible, we do not actively test or support this path. The following list represents known upgrade path concerns, but does not include all breaking changes: * In [v0.29](https://loft.featurebase.app/changelog/vcluster-v029-vcluster-standalone) etcd was upgraded to v3.6. The upstream etcd docs state that before upgrading to v3.6, you must already be on v3.5.20 or higher, otherwise failures may occur. This version of etcd was introduced in vCluster v0.24.2. Ensure that you have upgraded to at minimum that version, before upgrading to v0.29. * In [v0.20](https://loft.featurebase.app/changelog/vcluster-v020-ga), `vcluster.yaml` was introduced which replaced `values.yaml`, and a migration was required. See the [following page](https://vcluster.com/docs/vcluster/reference/migrations/0-20-migration.md) for a guide on that path. ## Update `vcluster.yaml`[​](#update-vclusteryaml "Direct link to update-vclusteryaml") Optionally, you can update your `vcluster.yaml` to change your configuration options during a vCluster version upgrade. Limitations on changing configuration You cannot change distros or backing store once a tenant cluster is deployed, with one exception: starting with vCluster 0.25.0, migration from K3s to K8s distro is supported. For more details, see the [K3s to K8s Migration](https://vcluster.com/docs/vcluster/manage/upgrade/distro-migration.md). ## Upgrade vCluster version[​](#upgrade-vcluster-version "Direct link to Upgrade vCluster version") These steps assume that you have a `vcluster.yaml` file. Replace the variables in the code blocks with your values for: * Name of the tenant cluster * Namespace of where the tenant cluster is deployed * Version to upgrade to Modify the following with your specific values to replace on the whole page and generate copyable commands: VCLUSTER\_NAMEmyvcluster VCLUSTER\_NAMESPACEvcluster VCLUSTER\_VERSION0.31.0 * vCluster CLI * Helm * Terraform * Argo CD * Cluster API Upgrade your CLI to the version of vCluster that you want to upgrade your virtual cluster to. ``` vcluster upgrade --version 0.31.0 vcluster create --upgrade myvcluster -n vcluster -f vcluster.yaml ``` ``` helm upgrade --install myvcluster vcluster \ --values vcluster.yaml \ --repo https://charts.loft.sh \ --namespace vcluster \ --repository-config='' \ --version 0.31.0 ``` 1. Generate a new plan. ``` terraform plan ``` Verify that the provider can access your cluster and that the proposed changes are correct. 2. Upgrade your vCluster. ``` terraform apply ``` 1) Update ArgoCD `Application` file to update the version that you want to upgrade to. ``` apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: myvcluster namespace: argocd spec: project: default source: chart: vcluster repoURL: https://charts.loft.sh targetRevision: 0.31.0 helm: releaseName: myvcluster valueFiles: - vcluster.yaml destination: server: https://kubernetes.default.svc namespace:vcluster ``` 2) Commit and push this updated file to your configured ArgoCD repository. (Optional) Commit and push an updated `vcluster.yaml` file to your configured ArgoCD repository. 3) Synchronize your ArgoCD repository with your configured cluster. Learn more about [Cluster API Provider for vCluster](https://github.com/loft-sh/cluster-api-provider-vcluster). 1. Install the vCluster provider. ``` clusterctl init --infrastructure vcluster:v0.2.0 ``` 2. Export environment variables to be used by the cluster API provider to create an updated manifest. The manifest will be applied to your Kubernetes cluster, which will update your vCluster with the updated configuration options. ``` export VCLUSTER_YAML=$(awk '{printf "%s\\n", $0}' vcluster.yaml) ``` 3. Regenerate the manifest and apply the updated manifest. ``` clusterctl generate cluster myvcluster \ --infrastructure vcluster \ --target-namespace vcluster \ | kubectl apply -f - ``` Kubernetes Version The Kubernetes version for the vCluster is not set at the CAPI provider command. It is configured the `vcluster.yaml` file based on your Kubernetes distribution. 4. Wait for vCluster to be updated by watching for the vCluster custom resource to report a `ready` status. ``` kubectl wait --for=condition=ready vcluster -n vcluster myvcluster --timeout=300s ``` --- # AI Cloud: Managed Kubernetes Service Run a managed Kubernetes service on your GPU infrastructure. Each customer gets an isolated tenant cluster with dedicated GPU nodes. Your product is what customers interact with. Platform is the operations layer your team runs behind it. **Customer access model:** External paying customers, no direct Platform access. **Worker-node model:** Private nodes per customer. See [Choose a worker node model](https://vcluster.com/docs/vcluster/production-guide/choose-worker-node-model.md). **Typical stack:** Standalone (HA) as the control plane cluster. Private nodes per customer cluster. vMetal for bare metal GPU lifecycle. vNode for workload runtime isolation. ![Enterprise AI platform architecture: central control plane that creates tenant clusters for customers](/docs/img/production-paths/multi-tenant.png) Central control plane managing tenant clusters for customers **What makes this path different:** Customers never access Platform directly. They interact with your product. Platform RBAC locks direct access to your platform engineering team. This architecture also maps to the cluster-level isolation criteria that AI cloud buyers evaluate in frameworks like [ClusterMAX](https://www.vcluster.com/clustermax). If your customers buy managed model endpoints instead of Kubernetes clusters, start with [Inference Provider: Managed Model Serving](https://vcluster.com/docs/vcluster/production-guide/inference-provider.md). That path uses the same GPU infrastructure building blocks, but the customer-facing product boundary is an inference API and endpoint lifecycle instead of a tenant Kubernetes API. ## Day 0: Design decisions[​](#day-0-design-decisions "Direct link to Day 0: Design decisions") | Decision | Read next | Outcome | | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Choose the control plane deployment model | [Standalone deployment](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md), [Architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md) | Control planes as pods on an existing Kubernetes cluster, or Standalone on dedicated CPU nodes. Standalone is the common choice when no prior Kubernetes substrate exists. | | Plan bare metal GPU provisioning | [vMetal docs](https://www.vmetal.ai/docs/), [Metal3 node provider](https://vcluster.com/docs/platform/administer/node-providers/metal3.md), [bare metal overview](https://vcluster.com/docs/platform/administer/bare-metal/overview.md) | Decide whether vMetal manages the full machine lifecycle (PXE, OS imaging, BMC, reclaim) or nodes are joined manually or using another provisioner. | | Define per-customer node isolation | [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md), [node requirements](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/node-requirements.md) | Each customer's tenant cluster gets its own dedicated GPU node pool with a separate CNI/CSI, eliminating interference between customers. | | Plan network isolation | [VPN](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/vpn.md), [Netris integration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/bare-metal.md) | Tenant clusters connect to their private nodes over an encrypted VPN tunnel. Netris integration adds switch-level VLAN/VXLAN isolation per tenant. | | Define the default network policy posture | [Network policy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md) | Set a default-deny east-west posture per tenant template and confirm the control plane cluster's CNI enforces it. Some CNIs accept NetworkPolicy objects without enforcing them. | | Choose runtime isolation model | [vNode docs](https://www.vnode.com/docs/), [Virtual Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#virtual-nodes) | vNode provides kernel-level container isolation without VM overhead. Recommended when customers run privileged workloads, dynamic code execution, or need GPU access using CDI. | | Choose the GPU scheduling model | [DeviceClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/device-classes.md), [DRA ResourceClaims](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaims.md) | Device-plugin-style `DeviceClass` sync covers simple, whole-GPU allocation. DRA `ResourceClaim` sync (Pro) supports fine-grained and shared GPU allocation. | | Plan tenant storage | [Storage classes from host](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/storage-classes.md), [CSI storage capacities](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-storage-capacities.md) | Decide which storage classes each tenant tier can request (etcd/control-plane storage vs. model weights and checkpoints), and whether to sync host `CSIStorageCapacity` for scheduler-aware placement. | | Define cluster templates and AI stacks | [Templates](https://vcluster.com/docs/platform/administer/templates/overview.md), [Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md), [create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md), [Certified Stacks](https://vcluster.com/docs/platform/integrations/certified-stacks/.md) | Standardize each customer cluster and its AI software. Platform 4.12 includes Certified StackTemplates for dedicated and central NVIDIA Run:ai deployment models; create your own StackTemplates for other schedulers or developer environments. | | Decide whether to sell inference endpoints | [Inference Provider](https://vcluster.com/docs/vcluster/production-guide/inference-provider.md), [Inference serving stacks](https://vcluster.com/docs/vcluster/integrations/model-serving.md) | If your product abstracts Kubernetes away and exposes model endpoints, use the inference provider path for endpoint templates, serving stack deployment, routing, and autoscaling. | | Plan the customer-facing provisioning API | [Product control plane on the Platform API](https://vcluster.com/docs/platform/how-to/product-control-plane.md), [Projects](https://vcluster.com/docs/platform/administer/projects/create.md), [Quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md) | Your product API calls Platform to provision tenant clusters. Define the project structure, quota model, and automation hooks that back your customer-facing workflows. | | Plan durability | [Backing store](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store.md), [container control plane HA](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/high-availability.md), [Standalone HA](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md), [Platform HA](https://vcluster.com/docs/platform/maintenance/reliability-scaling/high-availability.md) | Choose the data store and replica model for Platform and per-customer control planes. | | Plan security hardening and audit logging | [Security baseline](https://vcluster.com/docs/vcluster/security/.md), [CIS hardening guide](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/overview.md), [Platform audit logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md) | Apply a CIS benchmark baseline to the control plane cluster and decide audit log retention and SIEM integration before onboarding tenants. | ## Day 1: Stand up the first production customer cluster[​](#day-1-stand-up-the-first-production-customer-cluster "Direct link to Day 1: Stand up the first production customer cluster") note Steps 3 and 4 configure Platform for your platform engineering team, not for your customers. Customers provision clusters through your product. Platform access should be restricted to your ops team. 1. Install [vCluster Platform](https://vcluster.com/docs/platform/install/quick-start-guide.md). If building from bare metal, deploy [vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) first, then move to [Standalone HA](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md) before production traffic. 2. Configure [backing store](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store.md) and [Platform HA](https://vcluster.com/docs/platform/maintenance/reliability-scaling/high-availability.md). 3. Configure [SSO](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md) and [permissions](https://vcluster.com/docs/platform/administer/users-permissions/overview.md) for your platform engineering team. 4. Create [projects](https://vcluster.com/docs/platform/administer/projects/create.md), [templates](https://vcluster.com/docs/platform/administer/templates/overview.md), [quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md), and [Auto Nodes](https://vcluster.com/docs/platform/administer/node-providers/overview.md) to back your customer provisioning workflows. 5. Set up [vMetal](https://www.vmetal.ai/docs/) and the [Metal3 node provider](https://vcluster.com/docs/platform/administer/node-providers/metal3.md): register BMC credentials, configure PXE networking, define OS images, and verify bare metal hosts reach `available`. 6. Configure per-customer network isolation with [VPN](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/vpn.md) and, if using Netris, the [Netris integration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/environment/bare-metal.md). Apply a default-deny [network policy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md) per customer template and confirm the control plane cluster's CNI enforces it. 7. Configure per-tier [storage classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/storage-classes.md) and, if the scheduler needs capacity-aware placement, sync [CSI storage capacities](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-storage-capacities.md). 8. Configure GPU scheduling: sync [DeviceClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/device-classes.md) for device-plugin allocation, or enable [DRA ResourceClaim sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaims.md) for fine-grained allocation. 9. Apply the [CIS hardening baseline](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/overview.md) to the control plane cluster and enable [Platform audit logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md). 10. Install [vNode](https://www.vnode.com/docs/) on eligible GPU nodes. Configure `sync.toHost.pods.runtimeClassName: vnode` in the cluster template. 11. If you use NVIDIA Run:ai, choose a supported deployment model and install the [Run:ai Certified Stacks](https://vcluster.com/docs/platform/integrations/certified-stacks/runai.md). Otherwise, [create a StackTemplate](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) for the applications your AI environment requires and [add it to the tenant cluster configuration](https://vcluster.com/docs/platform/use-platform/apps/use-stacks.md#choose-how-to-install-a-stack). 12. Validate tenant isolation from inside the tenant cluster: confirm the customer cannot see the control plane cluster, other tenants, or platform internals. 13. Wire your product API to Platform's provisioning endpoints and test the end-to-end customer onboarding flow. ## Day 2: Operate[​](#day-2-operate "Direct link to Day 2: Operate") | Operation | Read next | | ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Manage bare metal capacity and machine lifecycle | [Bare metal overview](https://vcluster.com/docs/platform/administer/bare-metal/overview.md), [Metal3 node provider](https://vcluster.com/docs/platform/administer/node-providers/metal3.md), [vMetal docs](https://www.vmetal.ai/docs/) | | Monitor platform and tenant workloads | [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md), [other architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md) | | Operate managed inference endpoints | [Inference Provider](https://vcluster.com/docs/vcluster/production-guide/inference-provider.md), [Inference serving stacks](https://vcluster.com/docs/vcluster/integrations/model-serving.md), [GPU and inference autoscaling](https://vcluster.com/docs/vcluster/integrations/gpu-hpa-dcgm.md) | | Upgrade Platform and tenant clusters | [Upgrade vCluster](https://vcluster.com/docs/vcluster/manage/upgrade/upgrade-version.md), [Upgrade Platform](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade.md) | | Back up and restore tenant clusters and Platform | [Snapshots](https://vcluster.com/docs/vcluster/manage/backup-restore/backup.md), [Restore](https://vcluster.com/docs/vcluster/manage/backup-restore/restore.md), [Platform backup](https://vcluster.com/docs/platform/maintenance/backup-restore/backup-restore-platform.md) | | Manage vNode compatibility during upgrades | [vNode limitations](https://www.vnode.com/docs/limitations/), [vNode configuration](https://www.vnode.com/docs/configuration/) | | Scale the control plane cluster | [Platform HA](https://vcluster.com/docs/platform/maintenance/reliability-scaling/high-availability.md), [multi-region Platform](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md) | | Track GPU utilization and faults for metering and troubleshooting | [GPU observability templates](https://vcluster.com/docs/platform/maintenance/observability/gpu-observability-templates.md), [NVSentinel GPU observability](https://vcluster.com/docs/platform/maintenance/observability/nvsentinel-gpu-observability.md) | | Review audit logs and security posture | [Platform audit logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md), [security baseline](https://vcluster.com/docs/vcluster/security/.md) | --- # Choose a worker node model Shared and private nodes aren't interchangeable settings you tune for cost or performance. They're a security boundary decision, and vCluster locks it at deployment time. Make this decision before you pick a [production path](https://vcluster.com/docs/vcluster/production-guide/overview.md) or deploy a worker node model, not after. **Private nodes** dedicate worker nodes to a single tenant cluster, so no other tenant's workloads share that node or its kernel. **Shared nodes** run multiple tenant clusters' workloads on the same worker nodes, isolated from each other at the control plane, API, and namespace level, but not at the kernel. See [Architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md#worker-nodes) for the full mechanics. Provisioning through a vCluster Platform template? This decision may already be made for you. Templates fix `privateNodes.enabled` in the underlying `vcluster.yaml`. You'll see the resolved config before confirming creation, but you can only change it if the template allows customization. Check with whoever owns the template, or see [Create templates](https://vcluster.com/docs/platform/administer/templates/create-templates.md) if you administer them yourself. This page covers customer-facing and other production offerings where tenants aren't your own trusted staff. "Trusted" here means an explicit, enforced trust domain, not just familiarity. If your tenants are internal and meet that bar, skip to [trusted internal tenants](#trusted-internal-tenants-are-a-separate-model) for the full criteria. ## Decide with three questions[​](#decide-with-three-questions "Direct link to Decide with three questions") For a customer-facing offering, use private nodes if the answer to **any** of these is yes. 1. Are you building Kubernetes-as-a-Service, Slurm-as-a-Service, or otherwise delivering a cluster or scheduler environment to the customer? 2. Does the customer get direct access to that cluster or scheduler, whether Kubernetes, Slurm, or an inference control plane, beyond narrowly scoped, read-only visibility such as viewing logs or status for their own jobs or pods? 3. Can the customer run custom workloads, whether by supplying images, code, or model artifacts directly, or by defining jobs, pods, or other workload resources through a higher-level API? Shared nodes may be suitable only when all three answers are no. You own the workload definitions and the cluster or scheduler's configuration and operations, and the customer only consumes your application or API. Treat "may be suitable" as provisional, not final, until you apply the [shared node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) and explicitly accept the shared-kernel exposure. Decision tree for choosing between private and shared nodes Any "yes" to the first three questions routes to private nodes, regardless of the other answers. Three "no" answers proceed to the hardening and risk-acceptance gate, where "yes" selects shared nodes and "no" selects private nodes. Hardening reduces risk on an accepted shared-node deployment. It never turns a "private nodes" result from the three questions into "shared nodes." ### What counts as narrowly scoped read-only visibility[​](#what-counts-as-narrowly-scoped-read-only-visibility "Direct link to What counts as narrowly scoped read-only visibility") This means an explicitly limited visibility surface, for example status and logs for the customer's own jobs or pods. It doesn't include: * Creating, updating, patching, or deleting Kubernetes resources, Slurm jobs, or other scheduler-managed workloads * `exec`, `attach`, or `port-forward` (or the equivalent job-shell access on other schedulers) * Reading Secrets, service account tokens, or broad cluster or node metadata * Writing custom resources * Permissions that can indirectly cause workloads or host-side resources to be created If a customer-facing offering needs any of these types of access, choose private nodes. ### What each result requires[​](#what-each-result-requires "Direct link to What each result requires") | Result | Immediate requirements | Next step | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | Shared nodes | Existing Kubernetes worker capacity on the control plane cluster. Requires a CNI that enforces NetworkPolicy. | [Shared Nodes quick start](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md) | | Private nodes | vCluster Platform (free mode supported), plus dedicated Linux worker capacity per tenant cluster, joined manually or provisioned automatically through a node provider. | [Private Nodes quick start](https://vcluster.com/docs/vcluster/quick-start/private-nodes.md) | ## Service models[​](#service-models "Direct link to Service models") You might describe your product in these terms rather than the three questions above. Match it to the row that fits for the same guidance. | Service model | Typical customer capability | Worker-node guidance | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Kubernetes-as-a-Service / Cluster-as-a-Service | Customer receives direct API access to the cluster or scheduler (Kubernetes, Slurm, or similar) and commonly administers workloads | Private nodes | | Namespace-as-a-Service | Customer can usually create Pods, controllers, Services, and other resources within a namespace, or jobs within a Slurm partition or queue | Private nodes, with a separate tenant cluster per customer. A namespace or Slurm partition doesn't isolate customers from each other within one cluster. Private nodes dedicate compute to the tenant cluster itself, not to namespaces or partitions inside it. | | Container-as-a-Service | Customer supplies an image, code, or job script, even if you hide the underlying cluster or scheduler from them | Private nodes, because the customer controls workload execution | | SaaS / provider-operated application | Customer uses your application or API. You own the images and workload definitions | Shared nodes may be suitable when access is tightly limited and the platform is hardened | | Trusted internal platform | Employees or teams may have direct cluster or scheduler write access and run arbitrary workloads under an accepted trust model | Shared nodes are supported with hardening. Use private nodes for stronger isolation, or where compliance or infrastructure isolation requires it. | ### Hybrids and edge cases[​](#hybrids-and-edge-cases "Direct link to Hybrids and edge cases") * A "SaaS" product that lets customers supply plugins, notebooks, images, models that execute code, or arbitrary jobs isn't provider-controlled for this decision. Treat it like Container-as-a-Service and use private nodes. * A portal that creates Deployments, Slurm jobs, or other scheduled workloads on a customer's behalf still gives the customer workload authority, even if the customer never sees `kubectl` or a scheduler CLI directly. Use private nodes. * Read-only diagnostic access doesn't make a product Kubernetes-as-a-Service or Slurm-as-a-Service by itself, as long as it's genuinely narrow and can't mutate or execute anything. * What the product is named matters less than the effective permissions and workload authority it grants. Judge by the three questions, not by the marketing category. ## Shared and private nodes as security boundaries[​](#shared-and-private-nodes-as-security-boundaries "Direct link to Shared and private nodes as security boundaries") * **Private nodes** are the security boundary for tenants with cluster, scheduler, or workload execution access. See [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for the full architecture. * **Shared nodes** are a supported model for trusted tenants, not a security boundary for untrusted tenants with cluster or scheduler access or arbitrary workload execution. See [Architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes) for the full architecture. A dedicated, labeled node pool selected from the control plane cluster is still the shared-node architecture. It scopes placement, not isolation, and it doesn't become private nodes. | | Private nodes | Shared nodes | | ----------------------------------------------------------------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | Kernel and node isolation | Dedicated per tenant | Shared across tenants | | CNI / CSI | Dedicated per tenant | Shared, tenant traffic isolated by policy | | Suitable when the tenant controls cluster or scheduler access or workload execution | Yes | No | | Suitable for trusted internal tenants | Yes | Yes, with [hardening](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) | | Additional runtime isolation | Optional, recommended for privileged or untrusted workloads | [vNode](https://www.vnode.com/docs/) recommended for stronger per-tenant boundaries | ## Examples[​](#examples "Direct link to Examples") * **A managed Kubernetes service reselling clusters to external customers** is delivering a Kubernetes environment to the customer. Use private nodes, regardless of how the accounts are billed or resold. * **An internal CI/CD platform that creates ephemeral clusters for your own engineering teams** is a trusted internal tenant. Shared nodes are appropriate, hardened according to the [shared node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md). * **A SaaS inference API where customers call your endpoint and never see Kubernetes** answers no to all three questions. Shared nodes may be suitable when hardened and using tightly scoped access. * **A self-service portal that lets external customers deploy their own container images through a form**, even without `kubectl`, gives the customer workload authority. Use private nodes. ## Trusted internal tenants are a separate model[​](#trusted-internal-tenants-are-a-separate-model "Direct link to Trusted internal tenants are a separate model") The three-question tree above is for customer-facing and other untrusted-tenant production offerings. Trusted internal teams, such as development, testing, CI/CD, and internal platform tenants, are a separate threat model. Treat a tenant as trusted only when all of the following hold: * Every workload owner belongs to an organization you've explicitly accepted into a common trust domain, not an external account. * You accept that tenant workloads share a kernel and worker nodes with each other, and that shared-kernel exposure is documented. * No contract, regulation, or internal policy requires dedicated infrastructure for these workloads. * You can enforce and verify the shared-node hardening baseline before onboarding, not just declare it in `vcluster.yaml`. If any of these don't hold, use private nodes, even for a team you'd otherwise call internal. Shared nodes remain supported when all four hold, with hardening applied and the architecture decision documented per the hardening guide's own checklist. Private nodes remain the right choice for internal tenants too, when compliance or infrastructure isolation requires it. Shared nodes are for trusted tenants Shared nodes are a supported, common model for trusted tenants. They give each tenant control-plane, API, and namespace isolation, but tenant workloads share the same kernel and physical nodes. They aren't a security boundary for untrusted tenants with Kubernetes access or arbitrary workload execution. Good fits are development, testing, CI/CD, and internal engineering teams. Use [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for external, resale, regulated, or otherwise untrusted tenant offerings, optionally with [vNode](https://www.vnode.com/docs/) for runtime isolation. NetworkPolicy is an added isolation layer worth enabling even for trusted tenants. vCluster can create the policies for you through `policies.networkPolicy`, and your control plane cluster's CNI enforces them. Confirm your CNI supports enforcement, since some accept NetworkPolicy resources without acting on them. See the [security baseline](https://vcluster.com/docs/vcluster/security/.md). If you accept shared nodes for a trusted-tenant use case, follow the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) as the required next step. The external-tenant line in the note above states the general default, not an override. The provider-operated SaaS exception from the three questions still holds. An external SaaS or inference-API offering where the customer has no cluster or scheduler access and can't run custom workloads can still use shared nodes. ## Where this decision gets set[​](#where-this-decision-gets-set "Direct link to Where this decision gets set") There's no separate toggle for this decision. `privateNodes.enabled` in `vcluster.yaml` is the actual switch, set at initial installation. Leave it unset or `false` for shared nodes, or set it `true` for private nodes. It can't change after creation without redeploying. See [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for the full configuration. ## Next steps[​](#next-steps "Direct link to Next steps") * Answered "private nodes"? Continue to [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) and [Deploy vCluster](https://vcluster.com/docs/vcluster/deploy.md). Private nodes need dedicated worker capacity per tenant. If that capacity isn't available yet, close the infrastructure gap or reconsider what you're offering customers, not fall back to shared nodes as a workaround. * Answered "shared nodes may be suitable"? Continue to the [Shared Nodes quick start](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md) or [Nodes from the Control Plane Cluster](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes.md) to deploy, then apply the [shared node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) and document your acceptance of the shared-kernel exposure, per the guide's own checklist, before treating the decision as final. * Match your answer to a concrete path in [Build for Production](https://vcluster.com/docs/vcluster/production-guide/overview.md). --- # CI/CD Platform Provide isolated, ephemeral Kubernetes environments for CI/CD pipelines. Each pipeline run gets its own tenant cluster. No shared state, no cross-pipeline interference, no cleanup scripts. Clusters are created on demand and deleted automatically after the job finishes. **Customer access model:** Internal, trusted CI/CD pipelines with ephemeral Kubernetes access. **Worker-node model:** Shared nodes, trusted internal tenants only. **Typical stack:** Shared nodes, Platform, auto-sleep and auto-delete configured in templates. Pipelines create clusters through the CLI, Terraform, or the Platform API. Shared nodes are for trusted tenants Shared nodes are a supported, common model for trusted tenants. They give each tenant control-plane, API, and namespace isolation, but tenant workloads share the same kernel and physical nodes. They aren't a security boundary for untrusted tenants with Kubernetes access or arbitrary workload execution. Good fits are development, testing, CI/CD, and internal engineering teams. Use [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for external, resale, regulated, or otherwise untrusted tenant offerings, optionally with [vNode](https://www.vnode.com/docs/) for runtime isolation. NetworkPolicy is an added isolation layer worth enabling even for trusted tenants. vCluster can create the policies for you through `policies.networkPolicy`, and your control plane cluster's CNI enforces them. Confirm your CNI supports enforcement, since some accept NetworkPolicy resources without acting on them. See the [security baseline](https://vcluster.com/docs/vcluster/security/.md). If you accept shared nodes for a trusted-tenant use case, follow the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) as the required next step. **What makes this path different:** Cluster lifetime is measured in minutes or hours, not weeks. The underlying architecture is shared nodes plus Platform, identical to the [Internal Kubernetes Platform](https://vcluster.com/docs/vcluster/production-guide/internal-k8s-platform.md). What differs is what you are optimizing for. Here it is cluster lifecycle management at scale, not long-term governance. If you are building an internal platform that includes both long-running team clusters and ephemeral CI clusters, start with the [Internal Kubernetes Platform](https://vcluster.com/docs/vcluster/production-guide/internal-k8s-platform.md) and extend it with the CI/CD patterns documented here. ## Day 0: Design decisions[​](#day-0-design-decisions "Direct link to Day 0: Design decisions") | Decision | Read next | Outcome | | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Define cluster TTL and cleanup policy | [Sleep mode](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md), [auto-delete](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md#configure-auto-delete) | Set a default TTL for CI clusters. Decide between auto-delete after inactivity, after a fixed duration, or on pipeline completion using the CLI. | | Choose pipeline integration method | [vCluster CLI](https://vcluster.com/docs/vcluster/cli/vcluster_create), [Terraform](https://vcluster.com/docs/platform/administer/node-providers/terraform.md), [Argo CD](https://vcluster.com/docs/platform/integrations/argocd.md) | Determine how pipelines provision and teardown clusters: CLI commands in workflow steps, Terraform in GitOps, or Platform API. | | Define resource quotas | [Quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md), [resource quota](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/resource-quota.md) | Cap how many concurrent CI clusters a project can run to prevent node saturation during peak pipeline load. | | Plan image caching | Your container runtime's registry mirror docs | Decide whether CI clusters pull images directly from registries or use a cache. On shared nodes, the control plane cluster's container runtime pulls images, so configure any pull-through cache there. A warm cache significantly reduces job time. | ## Day 1: Stand up the platform[​](#day-1-stand-up-the-platform "Direct link to Day 1: Stand up the platform") Use the [Internal Kubernetes Platform](https://vcluster.com/docs/vcluster/production-guide/internal-k8s-platform.md) Day 1 steps to install vCluster Platform, configure SSO and team permissions, create projects, and define templates. Then add the CI/CD-specific configuration: 1. Create a CI-specific [template](https://vcluster.com/docs/platform/administer/templates/overview.md) with auto-delete enabled after your target TTL: ``` deletion: auto: afterInactivity: 1h ``` 2. Set [project quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md) on maximum concurrent CI clusters per team or pipeline. 3. Configure your CI system to create clusters using the CLI or Terraform: ``` # Create a cluster at pipeline start vcluster create ci-${CI_JOB_ID} --namespace ci-${CI_JOB_ID} --template ci-cluster # Delete on pipeline completion (or let auto-delete handle it) vcluster delete ci-${CI_JOB_ID} --namespace ci-${CI_JOB_ID} ``` 4. Validate end-to-end: a pipeline run creates a cluster, deploys a test workload, and the cluster is deleted after the job finishes. ## Day 2: Operate[​](#day-2-operate "Direct link to Day 2: Operate") | Operation | Read next | | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Monitor concurrent cluster count and node utilization during peak load | [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md), [other architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md) | | Audit orphaned clusters (jobs that exited without cleanup) | [Sleep mode](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md), [auto-delete](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sleep.md#configure-auto-delete) | | Tune quotas as pipeline volume grows | [Quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md) | | Upgrade Platform and cluster templates | [Upgrade Platform](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade.md), [upgrade vCluster](https://vcluster.com/docs/vcluster/manage/upgrade/upgrade-version.md) | --- # Distributed Compute Aggregation One management plane for GPU compute from every source you own or contract. Whether that is a cloud region, a co-located data center, or a compute supplier, each site runs a Standalone control plane cluster. Your central Platform instance sees the full fleet. **Customer access model:** Each tenant gets an isolated cluster on dedicated GPU nodes, regardless of which compute source backs it. **Worker-node model:** Private nodes per tenant cluster at every site. **Typical stack:** Central Platform (global). Per-DC Standalone control plane cluster. Private GPU nodes co-located with each Standalone instance. vMetal for owned bare metal sites. ![Compute aggregation architecture: central control plane with tenant clusters scheduling work onto distributed capacity sources](/docs/img/production-paths/distributed-compute.png) Central control plane managing tenant workloads across distributed compute capacity sources **What makes this path different:** Every compute source, regardless of where it lives or who operates it, follows the same onboarding playbook. One ops team, one management plane, no per-provider tooling. This architecture also maps to the cluster-level isolation criteria that AI cloud buyers evaluate in frameworks like [ClusterMAX](https://www.vcluster.com/clustermax). ## Day 0: Design decisions[​](#day-0-design-decisions "Direct link to Day 0: Design decisions") | Decision | Read next | Outcome | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Design the central-regional topology | [Multi-region Platform](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md), [Standalone deployment](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) | Central Platform registers each DC's Standalone instance as a connected cluster. Tenant control planes run co-located with their GPU nodes at each site. | | Define the supplier onboarding playbook | [Standalone HA](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md), [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md), [Auto Nodes](https://vcluster.com/docs/platform/administer/node-providers/overview.md) | Standardize the steps to register a new compute source: install Standalone, connect to central Platform, join GPU inventory, validate. | | Plan networking between sites | [VPN](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/vpn.md) | Each site's tenant clusters connect to their private nodes over VPN. Define whether tenants can span sites or are bound to a single DC. | | Define the default network policy posture | [Network policy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md) | Set a default-deny east-west posture per tenant template at every site and confirm each site's CNI enforces it. Some CNIs accept NetworkPolicy objects without enforcing them. | | Define tenant isolation at each site | [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md), [vNode docs](https://www.vnode.com/docs/) | Each tenant cluster at a site gets its own dedicated GPU nodes. vNode adds runtime isolation for untrusted workloads. | | Choose the GPU scheduling model | [DeviceClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/device-classes.md), [DRA ResourceClaims](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaims.md) | Device-plugin-style `DeviceClass` sync covers simple, whole-GPU allocation. DRA `ResourceClaim` sync (Pro) supports fine-grained and shared GPU allocation, consistently across every compute source. | | Standardize tenant GPU software | [Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md), [create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) | If you manage tenant software, define one ordered application baseline that can be deployed consistently across compute sources. | | Plan tenant storage per site | [Storage classes from host](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/storage-classes.md), [CSI storage capacities](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-storage-capacities.md) | Decide which storage classes are available at each site, since local NVMe or shared storage availability can differ by compute source, and whether to sync host `CSIStorageCapacity` for scheduler-aware placement. | | Plan security hardening and audit logging | [Security baseline](https://vcluster.com/docs/vcluster/security/.md), [CIS hardening guide](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/overview.md), [Platform audit logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md) | Apply a consistent CIS benchmark baseline across every site's control plane cluster and decide audit log retention and SIEM integration before onboarding tenants. | ## Day 1: Stand up the first compute source[​](#day-1-stand-up-the-first-compute-source "Direct link to Day 1: Stand up the first compute source") 1. Deploy [vCluster Platform](https://vcluster.com/docs/platform/install/quick-start-guide.md) in the central region. Configure [Platform HA](https://vcluster.com/docs/platform/maintenance/reliability-scaling/high-availability.md) and [backup](https://vcluster.com/docs/platform/maintenance/backup-restore/backup-restore-platform.md). 2. At the first compute site, install [vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) and move to [HA](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md). 3. Register the site's Standalone control plane cluster with central Platform. 4. Join the site's GPU inventory as private nodes. Configure [Auto Nodes](https://vcluster.com/docs/platform/administer/node-providers/overview.md) or [vMetal](https://www.vmetal.ai/docs/) for automated provisioning and reclaim. 5. Configure [templates](https://vcluster.com/docs/platform/administer/templates/overview.md) and [quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md) in central Platform for this compute source. 6. Apply a default-deny [network policy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md) in the site's template and confirm the site's CNI enforces it. 7. Configure per-site [storage classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/storage-classes.md) for the template. 8. Configure GPU scheduling: sync [DeviceClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/device-classes.md) for device-plugin allocation, or enable [DRA ResourceClaim sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaims.md) for fine-grained allocation. 9. If you manage the tenant software layer, [create a StackTemplate](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) for the GPU applications that should be consistent across sites, then [deploy it to the tenant environments](https://vcluster.com/docs/platform/use-platform/apps/use-stacks.md#choose-how-to-install-a-stack). 10. Apply the [CIS hardening baseline](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/overview.md) to the site's control plane cluster and enable [Platform audit logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md). 11. Provision a test tenant cluster at the site. Validate that it appears in central Platform and that workloads land on the correct GPU nodes. 12. Document the supplier onboarding playbook. Repeat steps 2-11 for each additional compute source. ## Day 2: Operate[​](#day-2-operate "Direct link to Day 2: Operate") | Operation | Read next | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Monitor the distributed fleet | [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md), [other architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md) | | Onboard a new compute source | Supplier onboarding playbook (internal), [Standalone deployment](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) | | Upgrade across sites | [Upgrade vCluster](https://vcluster.com/docs/vcluster/manage/upgrade/upgrade-version.md), [upgrade Platform](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade.md) | | Handle site failures | [Platform HA](https://vcluster.com/docs/platform/maintenance/reliability-scaling/high-availability.md), [multi-region Platform](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md) | | Track GPU utilization and faults across sites | [GPU observability templates](https://vcluster.com/docs/platform/maintenance/observability/gpu-observability-templates.md), [NVSentinel GPU observability](https://vcluster.com/docs/platform/maintenance/observability/nvsentinel-gpu-observability.md) | | Review audit logs and security posture across the fleet | [Platform audit logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md), [security baseline](https://vcluster.com/docs/vcluster/security/.md) | --- # Edge Distribution Deploy tenant workloads at distributed edge sites without managing a separate cluster at each one. Telecommunications providers, ISPs, and edge compute operators use this model to run low-latency services at POPs, exchange points, and remote facilities from a single central control plane. **Customer access model:** Each tenant gets an isolated cluster whose workloads schedule onto dedicated edge nodes. **Worker-node model:** Private nodes, joined from each edge site. **Typical stack:** Central Standalone (HA) as the control plane cluster. Central vCluster Platform. Worker nodes at each edge site joined to the central control plane using VPN. Tenant control planes run centrally; pods schedule onto edge nodes. ![Edge distribution architecture: central control plane with tenant clusters scheduling pods onto distributed edge site nodes](/docs/img/production-paths/edge-distribution.png) Central control plane managing tenant workloads across distributed edge sites **What makes this path different:** The compute is distributed but the control plane is not. Your ops team manages the full fleet from one place. Tenants get a standard Kubernetes experience regardless of which edge site their workload lands on. ## Day 0: Design decisions[​](#day-0-design-decisions "Direct link to Day 0: Design decisions") | Decision | Read next | Outcome | | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | Design the central-edge topology | [Standalone deployment](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md), [VPN](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/vpn.md) | Central Standalone hosts all tenant control planes. Edge sites contribute worker nodes over VPN. | | Define edge site requirements | [Node requirements](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/node-requirements.md) | Document the OS, networking, and connectivity requirements for each edge site before joining nodes. | | Plan node labeling and scheduling | [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) | Label nodes by site, region, or tier so tenant workloads can be scheduled to the correct edge location. | | Plan tenant isolation at edge | [vNode docs](https://www.vnode.com/docs/) | Decide whether edge workloads need runtime isolation, especially for multi-tenant edge sites. | ## Day 1: Stand up the first edge site[​](#day-1-stand-up-the-first-edge-site "Direct link to Day 1: Stand up the first edge site") 1. Deploy [vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) centrally and move to [HA](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md). 2. Install [vCluster Platform](https://vcluster.com/docs/platform/install/quick-start-guide.md) and configure [SSO](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md) and [permissions](https://vcluster.com/docs/platform/administer/users-permissions/overview.md). 3. Establish [VPN connectivity](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/vpn.md) to the first edge site. 4. Join edge site nodes to the central control plane cluster. Apply site labels. 5. Create a [template](https://vcluster.com/docs/platform/administer/templates/overview.md) that schedules tenant workloads to the correct edge site using node selectors or affinity rules. 6. Provision a test tenant cluster. Validate that pods land on the intended edge nodes. 7. Document the site onboarding playbook. Repeat steps 3-6 for each additional edge site. ## Day 2: Operate[​](#day-2-operate "Direct link to Day 2: Operate") | Operation | Read next | | --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Monitor edge nodes and tenant workloads | [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md), [other architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md) | | Add and remove edge sites | Site onboarding playbook (internal), [VPN](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/vpn.md) | | Upgrade tenant clusters and Platform | [Upgrade vCluster](https://vcluster.com/docs/vcluster/manage/upgrade/upgrade-version.md), [upgrade Platform](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade.md) | | Handle site connectivity failures | [Platform HA](https://vcluster.com/docs/platform/maintenance/reliability-scaling/high-availability.md), [multi-region Platform](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md) | --- # Enterprise AI Factory Give every team in your organization a production-grade AI environment without the overhead of managing separate clusters. Production ML teams get private GPU nodes and full isolation. Dev and experiment teams share a node pool and provision on demand. Platform keeps everything governed, audited, and within budget. **Customer access model:** Internal teams with Kubernetes access, tiered by trust and workload. **Worker-node model:** Private nodes for production ML teams, shared nodes for trusted internal dev and experiment tiers. **Typical stack:** Platform on an existing cloud Kubernetes cluster or Standalone on-premises. Private nodes for production ML teams. Shared node pool for dev and experiment workloads. vMetal optional for on-premises GPU fleets. Shared nodes are for trusted tenants Shared nodes are a supported, common model for trusted tenants. They give each tenant control-plane, API, and namespace isolation, but tenant workloads share the same kernel and physical nodes. They aren't a security boundary for untrusted tenants with Kubernetes access or arbitrary workload execution. Good fits are development, testing, CI/CD, and internal engineering teams. Use [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for external, resale, regulated, or otherwise untrusted tenant offerings, optionally with [vNode](https://www.vnode.com/docs/) for runtime isolation. NetworkPolicy is an added isolation layer worth enabling even for trusted tenants. vCluster can create the policies for you through `policies.networkPolicy`, and your control plane cluster's CNI enforces them. Confirm your CNI supports enforcement, since some accept NetworkPolicy resources without acting on them. See the [security baseline](https://vcluster.com/docs/vcluster/security/.md). If you accept shared nodes for a trusted-tenant use case, follow the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) as the required next step. This path serves your organization's own teams. The shared node tier is for internal dev and experiment workloads only. If you serve external customers or resell access, use the [AI Cloud](https://vcluster.com/docs/vcluster/production-guide/ai-cloud.md) path, which gives each customer private nodes. ![Enterprise AI factory architecture: central control plane managing private GPU nodes for production ML teams and a shared node pool for dev and experiment workloads](/docs/img/production-paths/ai-factory.png) Central control plane managing private GPU nodes for ML teams alongside a shared pool for dev workloads **What makes this path different:** Your teams use Platform directly, or through an internal portal built on top of it. Self-service provisioning, standardized GPU environments, and chargeback visibility are what you are building toward. ## Day 0: Design decisions[​](#day-0-design-decisions "Direct link to Day 0: Design decisions") | Decision | Read next | Outcome | | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Decide the Control Plane Cluster foundation | [Architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md), [Deployment basics](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/basics.md) | Run the Control Plane Cluster on an existing managed Kubernetes service (EKS, AKS, GKE) or vCluster Standalone on on-premises servers. | | Define team tenancy tiers | [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md), [Deployment basics](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/basics.md) | Classify teams: production AI workloads on private GPU nodes, and dev and experiment workloads on shared nodes. | | Define project and quota structure | [Projects](https://vcluster.com/docs/platform/administer/projects/create.md), [Quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md) | Map projects to teams or business units. Set GPU, CPU, and memory quotas per project. | | Plan GPU tooling and AI stacks | [Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md), [Certified Stacks](https://vcluster.com/docs/platform/integrations/certified-stacks/.md) | Use StackTemplates to standardize ordered application deployment. Platform 4.12 includes Certified StackTemplates for dedicated and central NVIDIA Run:ai deployment models; create your own StackTemplates for other schedulers or developer environments. | | Choose the runtime isolation model | [vNode docs](https://www.vnode.com/docs/) | vNode is recommended when teams run agentic workloads, untrusted code execution, or need root access inside containers without the risk of node escape. | | Plan SSO and governance | [SSO](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md), [RBAC](https://vcluster.com/docs/platform/administer/users-permissions/overview.md) | Integrate with your corporate identity provider. Define which teams can self-service create clusters and which require approval. | ## Day 1: Stand up the first production environment[​](#day-1-stand-up-the-first-production-environment "Direct link to Day 1: Stand up the first production environment") 1. Install [vCluster Platform](https://vcluster.com/docs/platform/install/quick-start-guide.md) on your Control Plane Cluster. 2. Configure [SSO](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md), [teams, and permissions](https://vcluster.com/docs/platform/administer/users-permissions/overview.md) against your corporate identity provider. 3. Create [projects](https://vcluster.com/docs/platform/administer/projects/create.md), [templates](https://vcluster.com/docs/platform/administer/templates/overview.md), and [quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md) per team or business unit. 4. Configure the shared node pool: [pod security standard](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/pod-security-standard.md), [network policy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md), and [resource quota](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/resource-quota.md). 5. Configure private GPU nodes for production teams: [join nodes manually](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/join.md) or configure [Auto Nodes](https://vcluster.com/docs/platform/administer/node-providers/overview.md). 6. Set up [vMetal](https://www.vmetal.ai/docs/) if on-premises GPU servers are part of the fleet. 7. If you use NVIDIA Run:ai, choose a supported deployment model and install the [Run:ai Certified Stacks](https://vcluster.com/docs/platform/integrations/certified-stacks/runai.md). Otherwise, [create a StackTemplate](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) for your GPU applications and scheduler, then [deploy it to the tenant environment](https://vcluster.com/docs/platform/use-platform/apps/use-stacks.md#choose-how-to-install-a-stack). 8. Install [vNode](https://www.vnode.com/docs/) on nodes designated for agentic or privileged workloads. 9. Provision the first tenant cluster for each tier (shared and private) and validate GPU access, quota enforcement, and network isolation. For shared tiers, confirm the control plane cluster's CNI enforces NetworkPolicy. With `sync.toHost.networkPolicies` enabled, apply a default-deny policy and confirm the blocked connection is refused. See the [security baseline](https://vcluster.com/docs/vcluster/security/.md). 10. Document how teams request clusters, retrieve kubeconfigs, request quota increases, and report issues. ## Day 2: Operate[​](#day-2-operate "Direct link to Day 2: Operate") | Operation | Read next | | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Monitor platform and team workloads | [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md), [other architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md) | | Manage private node capacity | [Auto Nodes](https://vcluster.com/docs/platform/administer/node-providers/overview.md), [manage private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/manage.md) | | Upgrade clusters and Platform | [Upgrade vCluster](https://vcluster.com/docs/vcluster/manage/upgrade/upgrade-version.md), [upgrade Platform](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade.md) | | Back up and restore | [Snapshots](https://vcluster.com/docs/vcluster/manage/backup-restore/backup.md), [Platform backup](https://vcluster.com/docs/platform/maintenance/backup-restore/backup-restore-platform.md) | --- # Inference Provider: Managed Model Serving Build managed inference endpoints on your GPU infrastructure. Customers interact with your product API and model endpoints. Platform and vCluster form the operations layer your team runs behind it. **Customer access model:** External customers, through provider-managed product APIs and model endpoints. **Worker-node model:** Private nodes for dedicated or customer-controlled serving, shared nodes for provider-owned models with API-only access. See [Choose a worker node model](https://vcluster.com/docs/vcluster/production-guide/choose-worker-node-model.md). **Minimum stack:** * [vCluster Platform](https://vcluster.com/docs/platform/install/quick-start-guide.md) as the management plane. * If you already operate Kubernetes, each tenant cluster's [control plane](https://vcluster.com/docs/vcluster/introduction/architecture.md) runs as pods on your existing cluster. If you don't have Kubernetes to host it on, use [vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) to bootstrap the tenant cluster control plane directly on bare metal or VMs. Move to [Standalone HA](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md) before production traffic. * Private GPU nodes for dedicated inference customers. * Argo CD or Akuity to deliver the GPU stack and runtime declaratively into each tenant cluster. **Optional add-ons:** * vMetal when you own bare metal GPU lifecycle. Skip it if your GPU capacity comes from cloud GPU instances, manually joined nodes, or Auto Nodes instead. * vNode, a separate vCluster Labs product, for runtime isolation of custom containers, adapters, plugins, or other untrusted code. Product API and traffic layer boundary backed by dedicated, isolated tenant clusters on private GPU nodes **What makes this path different:** Customers don't use Platform directly. They call your product API or inference endpoint. Templates are the core payoff for your product. Applying a template creates a tenant cluster, attaches GPU capacity, enforces quotas, publishes routes, and reclaims capacity when endpoints are deleted. Confirm licensing before you invest further. Several pieces of this stack require Platform activation or a paid tier. See [Open Source vs Free tier](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md) for the tier check. Not building a multi-customer provider? If your users are internal teams that should provision and manage their own environments in Platform, start with [Enterprise AI Factory](https://vcluster.com/docs/vcluster/production-guide/enterprise-ai-factory.md) instead. If you only need to deploy one model inside a tenant cluster you already have, skip ahead to [Inference serving stacks](https://vcluster.com/docs/vcluster/integrations/model-serving.md). Use this inference provider path when your product hides Platform and exposes an inference endpoint lifecycle to multiple customers. Before standing up infrastructure, review [Building an inference platform](https://vcluster.com/docs/vcluster/introduction/inference-platform.md) for the tenancy models, GPU capacity classes, endpoint readiness, and product control plane mapping behind the decisions below. ## Day 0: Design decisions[​](#day-0-design-decisions "Direct link to Day 0: Design decisions") | Decision | Read next | Outcome | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Choose the inference tenancy model | [Choose an inference tenancy model](https://vcluster.com/docs/vcluster/introduction/inference-platform.md#choose-an-inference-tenancy-model), [Architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md) | Decide whether each customer, model family, or endpoint tier gets a tenant cluster, private GPU nodes, vNode runtime isolation, or a shared serving pool. | | Define the default network policy posture | [Network policy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md) | Set a default-deny east-west posture per endpoint template and confirm the control plane cluster's CNI enforces it. Some CNIs accept NetworkPolicy objects without enforcing them. | | Plan GPU capacity classes | [Size GPU capacity classes](https://vcluster.com/docs/vcluster/introduction/inference-platform.md#size-gpu-capacity-classes), [Node providers](https://vcluster.com/docs/platform/administer/node-providers/overview.md), [GPU and accelerators](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md) | Define node types by GPU model, count, region, and reservation model, and choose an isolation strategy per tenant tier. | | Choose the GPU scheduling model | [GPU and accelerators](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md), [DRA DeviceClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/device-classes.md), [DRA ResourceClaims](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaims.md) | Use extended resources from a vendor device plugin for simple whole-GPU allocation. Use a vendor DRA driver with `DeviceClass` and `ResourceClaim` for fine-grained or shareable allocation. Private nodes can run the DRA stack directly. DRA sync (Pro) makes control plane cluster devices available to tenants. | | Define serving stack templates | [Inference serving stacks](https://vcluster.com/docs/vcluster/integrations/model-serving.md), [Deploy applications](https://vcluster.com/docs/platform/integrations/argocd/deploy-applications.md), [Certified Stacks](https://vcluster.com/docs/platform/integrations/certified-stacks/.md) | Standardize the components each endpoint receives. Define whether a direct chart or an inference control plane owns workloads, routes, and autoscalers. | | Plan application boundaries and sequencing | [Sequence the GPU stack and the serving stack](https://vcluster.com/docs/vcluster/integrations/model-serving.md#sequence-the-gpu-stack-and-the-serving-stack), [What are Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md), [Create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md), [Deploy applications](https://vcluster.com/docs/platform/integrations/argocd/deploy-applications.md) | Use a Stack when separate Apps or Argo CD Applications need dependencies or health gates. Without a Stack, bundle hard dependencies in one Argo CD Application and let Kubernetes absorb soft dependencies. | | Plan model storage and warmup | [Model storage architecture](https://vcluster.com/docs/vcluster/integrations/model-serving.md#plan-model-storage), [Inference serving stacks](https://vcluster.com/docs/vcluster/integrations/model-serving.md) | Decide where model weights live, how they reach each endpoint, how caches are warmed, and how readiness is reported while large models load. | | Plan endpoint routing | [Inference serving stacks](https://vcluster.com/docs/vcluster/integrations/model-serving.md#expose-an-inference-endpoint), [Gateway API](https://vcluster.com/docs/vcluster/key-features/gateway-api.md) | Decide which component creates each route and which shared networking resources the platform owns. | | Plan autoscaling signals | [GPU and inference autoscaling](https://vcluster.com/docs/vcluster/integrations/gpu-hpa-dcgm.md), [Monitoring overview](https://vcluster.com/docs/platform/maintenance/monitoring/overview.md) | Combine hardware metrics, such as GPU utilization and memory, with serving metrics, such as request concurrency, queue depth, latency, tokens per second, and time to first token. | | Plan endpoint readiness | [Endpoint readiness](https://vcluster.com/docs/vcluster/introduction/inference-platform.md#endpoint-readiness), [Inference serving stacks](https://vcluster.com/docs/vcluster/integrations/model-serving.md) | Account for Argo CD sync time, node provisioning, image pulls, model downloads, and model load time before customers expect an endpoint to be ready. | | Plan durability and operations | [Backing store](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store.md), [Platform HA](https://vcluster.com/docs/platform/maintenance/reliability-scaling/high-availability.md), [Platform backup](https://vcluster.com/docs/platform/maintenance/backup-restore/backup-restore-platform.md) | Choose data stores, backup policy, control plane availability, and recovery procedures before customers depend on endpoints. | | Plan security hardening and audit logging | [Security baseline](https://vcluster.com/docs/vcluster/security/.md), [CIS hardening guide](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/overview.md), [Platform audit logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md) | Apply a CIS benchmark baseline to the control plane cluster and decide audit log retention and SIEM integration before onboarding tenants. | | Integrate with your product control plane | [Integrate with your product control plane](https://vcluster.com/docs/vcluster/introduction/inference-platform.md#integrate-with-your-product-control-plane), [Product control plane on the Platform API](https://vcluster.com/docs/platform/how-to/product-control-plane.md) | Decide the customer-facing product boundary, the mapping from product concepts to Platform resources, the metering approach, and the license tier you need before building automation. | ## Day 1: Stand up the first production inference endpoint[​](#day-1-stand-up-the-first-production-inference-endpoint "Direct link to Day 1: Stand up the first production inference endpoint") note Steps 3, 4, 5, and 6 configure Platform for your platform engineering team and automation. Customers should provision endpoints through your product. ### Set up the platform[​](#set-up-the-platform "Direct link to Set up the platform") 1. Install [vCluster Platform](https://vcluster.com/docs/platform/install/quick-start-guide.md). If building from bare metal, deploy [vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/basics.md) first, then move to [Standalone HA](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md) before production traffic. 2. Configure [backing store](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store.md), [Platform HA](https://vcluster.com/docs/platform/maintenance/reliability-scaling/high-availability.md), and [Platform backup](https://vcluster.com/docs/platform/maintenance/backup-restore/backup-restore-platform.md). 3. Configure [SSO](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md), [permissions](https://vcluster.com/docs/platform/administer/users-permissions/overview.md), and [access keys](https://vcluster.com/docs/platform/api/resources/ownedaccesskey.md) for the internal automation that provisions endpoints. 4. Create an [Argo CD connector](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md) to your Argo CD or Akuity instance. Every tenant cluster template references it by name. 5. Create [projects](https://vcluster.com/docs/platform/administer/projects/create.md), [templates](https://vcluster.com/docs/platform/administer/templates/overview.md), [quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md), and [allowed node types](https://vcluster.com/docs/platform/administer/projects/allowed/node-types.md) for the endpoint tier you're standing up. 6. Apply the [CIS hardening baseline](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/overview.md) to the control plane cluster and enable [Platform audit logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md). ### Attach GPU capacity[​](#attach-gpu-capacity "Direct link to Attach GPU capacity") 7. Configure GPU capacity for the inference tiers you sell. If you own bare metal, set up [vMetal](https://www.vmetal.ai/docs/) and the [Metal3 node provider](https://vcluster.com/docs/platform/administer/node-providers/metal3.md). If you use cloud GPU instances, create node types and automation for those instances through [node providers](https://vcluster.com/docs/platform/administer/node-providers/overview.md) and Auto Nodes, or your existing provisioning system and join them as Private Nodes. 8. Configure GPU scheduling. For device-plugin allocation, install the vendor plugin and have workloads request its extended resource, such as `nvidia.com/gpu`. For DRA allocation on private nodes, run the vendor DRA driver and controllers inside the tenant cluster. If devices reside on shared control plane cluster nodes instead, sync the permitted [DeviceClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/device-classes.md) and enable [ResourceClaim sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaims.md). ### Build the tenant cluster and serving stack templates[​](#build-the-tenant-cluster-and-serving-stack-templates "Direct link to Build the tenant cluster and serving stack templates") 9. Create a tenant cluster template for the endpoint tier. The template should enable [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md), restrict GPU node selection, enable any required sync settings, and reference the [Argo CD connector](https://vcluster.com/docs/platform/integrations/argocd/connect-argocd.md) from step 4. 10. Apply a default-deny [network policy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md) in the template and confirm the control plane cluster's CNI enforces it. 11. Declare the GPU stack and inference serving stack as tasks in a [Stack](https://vcluster.com/docs/platform/understand/what-are-stacks.md) when they need cross-application ordering. Add that Stack through `deploy.stacks`. Choose which serving component owns generated workloads, routes, and autoscalers. A direct engine chart declares those resources itself. A KServe-based stack declares an `InferenceService` and lets KServe generate them. A Stack task can reference an [`ArgoCDApplicationTemplate`](https://vcluster.com/docs/platform/integrations/argocd/deploy-applications.md) for each component. Without a Stack, keep controllers and their custom resources in one Application. Argo CD doesn't sequence separate Applications. See [Inference serving stacks](https://vcluster.com/docs/vcluster/integrations/model-serving.md) for the direct engine and KServe patterns. Each tenant cluster's scheduler makes decisions scoped to that cluster, not across your fleet. ### Configure routing and autoscaling[​](#configure-routing-and-autoscaling "Direct link to Configure routing and autoscaling") 12. Configure endpoint routing for the selected serving pattern. For a direct deployment, prefer Gateway API and package the route with its Deployment and Service. For KServe, configure the shared Gateway API or Ingress infrastructure and let KServe generate each endpoint's routing resource. 13. Configure autoscaling with hardware metrics, serving metrics, or both. Choose one autoscaler owner for each workload. Configure direct Deployments through HPA or KEDA. Configure KServe-managed workloads through the `InferenceService`. Use [GPU and inference autoscaling](https://vcluster.com/docs/vcluster/integrations/gpu-hpa-dcgm.md) for metric patterns. ### Wire up your product automation[​](#wire-up-your-product-automation "Direct link to Wire up your product automation") 14. Build the product automation path. For an endpoint create request, choose the project and template version, then pass tier parameters. Provision the tenant cluster from a template that already has the Argo CD connector and Application references set. Wait for GPU capacity and for the serving stack to report healthy. Then return endpoint status and URL. Start with an internal operator workflow before exposing the API to customers. See [Build a product control plane on the Platform API](https://vcluster.com/docs/platform/how-to/product-control-plane.md) for the concept-to-resource mapping this automation implements. ### Validate the first endpoint[​](#validate-the-first-endpoint "Direct link to Validate the first endpoint") 15. Validate the first endpoint from outside the tenant cluster. Confirm the endpoint accepts authorized traffic, rejects unauthorized traffic, reports healthy model status, emits metrics, and can be traced back to the tenant cluster, node type, template version, and route. 16. Validate isolation and quota enforcement. Confirm the customer can't see Platform internals, other tenant clusters, disallowed templates, disallowed GPU node types, or control plane cluster resources. 17. Test the full endpoint lifecycle through your product API. Create, warm, scale, update the model or adapter, rotate credentials, drain, delete, and reclaim GPU capacity. ### Endpoint provisioning flow[​](#endpoint-provisioning-flow "Direct link to Endpoint provisioning flow") Use this flow as the acceptance test for the first provider-managed endpoint: 1. Your product automation receives an endpoint request with model, region, endpoint tier, and scaling parameters. 2. Product automation calls Platform with an internal access key. 3. Platform creates or selects the project, tenant cluster template, serving stack template, quota, and allowed node types for the requested tier. 4. Private Nodes or Auto Nodes attach GPU capacity from the selected node type. 5. Argo CD syncs the GPU stack and serving stack declared in the tenant cluster template. 6. The direct engine chart or inference control plane creates the workload, Service, routing, autoscaling, and metrics resources. 7. Gateway API, Ingress, Knative networking, or your traffic layer publishes the endpoint hostname. 8. Product automation returns endpoint status, URL, model state, and operational identifiers. 9. Day 2 automation watches rollout, route, GPU, and serving metrics and reconciles endpoint status. ## Day 2: Operate[​](#day-2-operate "Direct link to Day 2: Operate") | Operation | Read next | | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Manage GPU capacity and machine lifecycle | [vMetal inference fleet capacity](https://www.vmetal.ai/docs/operate/gpu-fleet#inference-provider-capacity), [Metal3 node provider](https://vcluster.com/docs/platform/administer/node-providers/metal3.md), [Manage private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/manage.md) | | Monitor platform and tenant workloads | [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md), [other architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md) | | Expose and troubleshoot endpoint routing | [Inference serving stacks](https://vcluster.com/docs/vcluster/integrations/model-serving.md#expose-an-inference-endpoint), [Gateway API](https://vcluster.com/docs/vcluster/key-features/gateway-api.md), [Gateway API sync troubleshooting](https://vcluster.com/docs/vcluster/troubleshoot/gateway-api-sync.md) | | Scale model-serving workloads | [GPU and inference autoscaling](https://vcluster.com/docs/vcluster/integrations/gpu-hpa-dcgm.md), [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md) | | Meter customer usage | [vBilling](https://www.vcluster.com/vbilling), [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md) | | Roll model runtimes and endpoint templates | [Inference serving stacks](https://vcluster.com/docs/vcluster/integrations/model-serving.md#choose-a-deployment-pattern), [Deploy applications](https://vcluster.com/docs/platform/integrations/argocd/deploy-applications.md), [GPU and inference autoscaling](https://vcluster.com/docs/vcluster/integrations/gpu-hpa-dcgm.md) | | Roll drivers, vCluster versions, and OS images | [Deploy changes](https://vcluster.com/docs/vcluster/manage/deploy-changes.md), [vMetal GPU fleet operations](https://www.vmetal.ai/docs/operate/gpu-fleet#rotating-os-images), [Upgrade vCluster](https://vcluster.com/docs/vcluster/manage/upgrade/upgrade-version.md) | | Enforce customer capacity boundaries | [Quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md), [Allowed node types](https://vcluster.com/docs/platform/administer/projects/allowed/node-types.md), [Projects](https://vcluster.com/docs/platform/administer/projects/create.md) | | Back up and restore tenant clusters and Platform | [Snapshots](https://vcluster.com/docs/vcluster/manage/backup-restore/backup.md), [Restore](https://vcluster.com/docs/vcluster/manage/backup-restore/restore.md), [Platform backup](https://vcluster.com/docs/platform/maintenance/backup-restore/backup-restore-platform.md) | | Manage vNode compatibility during upgrades | [vNode limitations](https://www.vnode.com/docs/limitations/), [vNode configuration](https://www.vnode.com/docs/configuration/) | | Review audit logs and security posture | [Platform audit logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md), [security baseline](https://vcluster.com/docs/vcluster/security/.md) | --- # Internal Kubernetes Platform Provide isolated Kubernetes environments for internal engineering teams, business units, and the services they own. Clusters are long-running and may carry internal production traffic. Tenant isolation gives each team its own control plane and prevents noisy-neighbor effects across teams. Shared nodes avoid standing up separate infrastructure per team. Large enterprises use this model to replace shared cluster namespaces with stronger tenant isolation. Each team gets their own control plane, their own RBAC, and their own view of Kubernetes, without owning the infrastructure underneath. Shared nodes are for trusted tenants Shared nodes are a supported, common model for trusted tenants. They give each tenant control-plane, API, and namespace isolation, but tenant workloads share the same kernel and physical nodes. They aren't a security boundary for untrusted tenants with Kubernetes access or arbitrary workload execution. Good fits are development, testing, CI/CD, and internal engineering teams. Use [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for external, resale, regulated, or otherwise untrusted tenant offerings, optionally with [vNode](https://www.vnode.com/docs/) for runtime isolation. NetworkPolicy is an added isolation layer worth enabling even for trusted tenants. vCluster can create the policies for you through `policies.networkPolicy`, and your control plane cluster's CNI enforces them. Confirm your CNI supports enforcement, since some accept NetworkPolicy resources without acting on them. See the [security baseline](https://vcluster.com/docs/vcluster/security/.md). If you accept shared nodes for a trusted-tenant use case, follow the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) as the required next step. This path targets your own internal teams and services. If tenants are external customers or you're reselling access, follow the [AI Cloud](https://vcluster.com/docs/vcluster/production-guide/ai-cloud.md) or [Single-Tenant Per Customer](https://vcluster.com/docs/vcluster/production-guide/single-tenant.md) paths, which use private nodes. **Customer access model:** Internal, trusted engineering teams with Kubernetes access to their own tenant cluster. **Worker-node model:** Shared nodes, trusted internal tenants only. **Typical stack:** Platform on an existing Kubernetes cluster. Shared node pool with optional dedicated node pools for teams with elevated security or performance requirements. **What makes this path different:** These clusters are long-lived and may host internal production services, not just dev workloads. Governance, isolation, and uptime are the priorities. If you also need ephemeral CI clusters, that is the [CI/CD Platform](https://vcluster.com/docs/vcluster/production-guide/cicd-platform.md) path. ## Day 0: Design decisions[​](#day-0-design-decisions "Direct link to Day 0: Design decisions") | Decision | Read next | Outcome | | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Define the tenancy model | [Projects](https://vcluster.com/docs/platform/administer/projects/create.md), [templates](https://vcluster.com/docs/platform/administer/templates/overview.md) | Decide whether teams get one cluster per team, one per project, or on-demand. Set a default template enforced across all teams. | | Define application baselines | [Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md), [create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) | If teams need the same multi-application environment, define the Apps or Argo CD Applications, dependencies, and health gates that Platform deploys for them. | | Define access and governance model | [SSO](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md), [RBAC](https://vcluster.com/docs/platform/administer/users-permissions/overview.md), [network policy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md) | Map your identity provider to Platform roles. Define which teams can create clusters, how many, and with what resource limits. | | Define resource quotas | [Quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md), [resource quota](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/resource-quota.md) | Set per-team cluster count, CPU, and memory limits. Prevent runaway consumption without requiring ops approval for every cluster. | | Plan sync configuration | [Sync configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync) | Decide which control plane cluster resources (storage classes, ingress controllers, secrets) are visible inside tenant clusters. | | Define uptime requirements | [HA](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/high-availability.md), [backing store](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store.md) | Determine if tenant clusters need HA control planes. Most internal platforms start single-replica and graduate to HA as adoption grows. | ## Day 1: Stand up the platform[​](#day-1-stand-up-the-platform "Direct link to Day 1: Stand up the platform") 1. Install [vCluster Platform](https://vcluster.com/docs/platform/install/quick-start-guide.md) on your existing Kubernetes cluster. 2. Configure [SSO](https://vcluster.com/docs/platform/configure/platform-configs/single-sign-on.md) and [team permissions](https://vcluster.com/docs/platform/administer/users-permissions/overview.md). 3. Create [projects](https://vcluster.com/docs/platform/administer/projects/create.md) aligned to your org structure (team, BU, or environment). 4. Define [templates](https://vcluster.com/docs/platform/administer/templates/overview.md) that enforce your baseline: [pod security standard](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/pod-security-standard.md), [network policy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md), resource quota, and sync configuration. 5. If the environment includes a standard set of applications, [create a StackTemplate](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) and [deploy it to the tenant environments](https://vcluster.com/docs/platform/use-platform/apps/use-stacks.md#choose-how-to-install-a-stack) that need it. 6. Apply [quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md) per project to cap resource consumption. 7. Enable [monitoring](https://vcluster.com/docs/platform/maintenance/monitoring/overview.md) for the Platform layer and expose cluster-level metrics to Platform's fleet view. 8. Validate self-service and isolation. A team lead creates a cluster, deploys a workload, and verifies that another team's project is not visible. Confirm the control plane cluster's CNI enforces NetworkPolicy. With `sync.toHost.networkPolicies` enabled, apply a default-deny policy in a tenant namespace and confirm the blocked connection is refused. If traffic still flows, shared-node network isolation isn't in effect. See the [security baseline](https://vcluster.com/docs/vcluster/security/.md). ## Day 2: Operate[​](#day-2-operate "Direct link to Day 2: Operate") | Operation | Read next | | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Monitor cluster health and resource usage across teams | [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md), [other architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md) | | Manage cluster lifecycle and upgrades | [Upgrade vCluster](https://vcluster.com/docs/vcluster/manage/upgrade/upgrade-version.md), [upgrade Platform](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade.md) | | Respond to quota violations or runaway clusters | [Quotas](https://vcluster.com/docs/platform/administer/projects/quotas.md), [resource quota](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/resource-quota.md) | | Back up and restore tenant clusters | [Snapshots](https://vcluster.com/docs/vcluster/manage/backup-restore/backup.md), [restore](https://vcluster.com/docs/vcluster/manage/backup-restore/restore.md) | | Rotate certificates | [Certificate rotation](https://vcluster.com/docs/vcluster/manage/certificates.md) | --- # Build for Production vCluster lets you provision isolated tenant clusters on your existing infrastructure without a separate physical cluster per tenant. This section maps the common production architectures to concrete implementation paths. Choose the one that matches what you are building and follow it from initial design to a running, operated platform. Need to evaluate vCluster first? Start with a [quick start](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md) to prove the deployment model in your environment, then return here to plan production. Choose your worker node model first Shared and private nodes are a security boundary decision, locked at deployment time. Before you pick a path below, run through [Choose a worker node model](https://vcluster.com/docs/vcluster/production-guide/choose-worker-node-model.md) to confirm which one your offering requires. ## What are you building?[​](#what-are-you-building "Direct link to What are you building?") Find the goal that matches your platform, then choose the desired outcome. The linked production path takes you from architecture decisions through Day 2 operations. ### Offer AI or Kubernetes infrastructure to customers[​](#offer-ai-or-kubernetes-infrastructure-to-customers "Direct link to Offer AI or Kubernetes infrastructure to customers") | Desired outcome | Start here | Worker-node guidance | | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | Serve models through managed endpoints while your team operates the Kubernetes infrastructure | [Inference Provider: Managed Model Serving](https://vcluster.com/docs/vcluster/production-guide/inference-provider.md) | Private nodes for dedicated or customer-controlled serving. Shared nodes may suit provider-owned models with API-only access | | Give customers isolated, GPU-backed Kubernetes environments through your product | [AI Cloud: Managed Kubernetes Service](https://vcluster.com/docs/vcluster/production-guide/ai-cloud.md) | Private nodes for external customer workloads | | Give each enterprise customer a dedicated cluster stack for node-level, regulated, or sovereign isolation | [Single-Tenant Per Customer](https://vcluster.com/docs/vcluster/production-guide/single-tenant.md) | Private nodes for node-level customer isolation | ### Build a platform for internal teams[​](#build-a-platform-for-internal-teams "Direct link to Build a platform for internal teams") | Desired outcome | Start here | Worker-node guidance | | --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | Standardize governed GPU environments across production, development, and experiment tiers | [Enterprise AI Factory](https://vcluster.com/docs/vcluster/production-guide/enterprise-ai-factory.md) | Private nodes for production. Shared nodes for trusted development and experiment tiers | | Give trusted engineering teams long-running, isolated Kubernetes environments | [Internal Kubernetes Platform](https://vcluster.com/docs/vcluster/production-guide/internal-k8s-platform.md) | Shared nodes for trusted team workloads only | | Give each CI/CD pipeline an ephemeral, isolated Kubernetes environment with automatic cleanup | [CI/CD Platform](https://vcluster.com/docs/vcluster/production-guide/cicd-platform.md) | Shared nodes for trusted pipeline workloads only | ### Operate compute across providers or locations[​](#operate-compute-across-providers-or-locations "Direct link to Operate compute across providers or locations") | Desired outcome | Start here | Worker-node guidance | | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------- | | Manage GPU capacity from multiple compute sources through one operations layer | [Distributed Compute Aggregation](https://vcluster.com/docs/vcluster/production-guide/distributed-compute.md) | Private nodes for dedicated tenant capacity at each site | | Manage tenant workloads at distributed edge sites from a central control plane | [Edge Distribution](https://vcluster.com/docs/vcluster/production-guide/edge-distribution.md) | Private nodes for dedicated tenant capacity at edge sites | Not sure where to start? Use [Choose a worker node model](https://vcluster.com/docs/vcluster/production-guide/choose-worker-node-model.md) to decide between shared and private nodes. If no production path fits, review [Architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md) and [Building a GPU cloud platform](https://vcluster.com/docs/vcluster/introduction/gpu-cloud-platform.md). ## What production-ready means[​](#what-production-ready-means "Direct link to What production-ready means") A production vCluster platform delivers: * **Tenant isolation:** every customer or team sees only their own cluster and workloads. Node level isolation requires private nodes * **Repeatable provisioning:** new tenant clusters deploy from a defined template, not from manual steps. When each environment also needs the same ordered set of applications, use [Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md) to standardize that software layer * **A worker node model matched to your isolation needs:** private nodes for tenants with cluster, scheduler, or workload execution access, or shared nodes for trusted internal tenants or a provider-operated offering where you own all workloads. A dedicated node pool scopes placement within shared nodes. It doesn't provide private-node isolation. See [Choose a worker node model](https://vcluster.com/docs/vcluster/production-guide/choose-worker-node-model.md). * **Governed access:** who can create, access, and administer tenant clusters, enforced through Platform policies * **Durable control planes:** HA, data store, and backup procedures defined before tenants depend on the system * **Operational readiness:** monitoring, upgrade, restore, and incident response procedures documented and tested ## Connect quick start results to production paths[​](#connect-quick-start-results-to-production-paths "Direct link to Connect quick start results to production paths") Each quick start validates a specific deployment model. Use this table to connect what you proved to the production path that extends it. | If you completed | You have proven | Production paths to consider | | ------------------------------------------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Docker (vind)](https://vcluster.com/docs/vcluster/quick-start/docker.md) | Local development, testing, or CI cluster behavior | Use vind to validate cluster behavior, then choose a production path for deployment. | | [Shared Nodes](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md) | Tenant clusters on an existing Kubernetes cluster | [Internal Kubernetes Platform](https://vcluster.com/docs/vcluster/production-guide/internal-k8s-platform.md), [CI/CD Platform](https://vcluster.com/docs/vcluster/production-guide/cicd-platform.md), or [Enterprise AI Factory](https://vcluster.com/docs/vcluster/production-guide/enterprise-ai-factory.md) (shared tier) | | [Private Nodes](https://vcluster.com/docs/vcluster/quick-start/private-nodes.md) | Tenant clusters with dedicated worker nodes | [Inference Provider](https://vcluster.com/docs/vcluster/production-guide/inference-provider.md), [AI Cloud](https://vcluster.com/docs/vcluster/production-guide/ai-cloud.md), [Enterprise AI Factory](https://vcluster.com/docs/vcluster/production-guide/enterprise-ai-factory.md) (production tier), [Distributed Compute Aggregation](https://vcluster.com/docs/vcluster/production-guide/distributed-compute.md), [Single-Tenant Per Customer](https://vcluster.com/docs/vcluster/production-guide/single-tenant.md), or [Edge Distribution](https://vcluster.com/docs/vcluster/production-guide/edge-distribution.md) | | [Standalone](https://vcluster.com/docs/vcluster/quick-start/standalone.md) | Control plane cluster on bare metal or VMs | [Inference Provider](https://vcluster.com/docs/vcluster/production-guide/inference-provider.md), [AI Cloud](https://vcluster.com/docs/vcluster/production-guide/ai-cloud.md), [Distributed Compute Aggregation](https://vcluster.com/docs/vcluster/production-guide/distributed-compute.md), [Enterprise AI Factory](https://vcluster.com/docs/vcluster/production-guide/enterprise-ai-factory.md) (on-premises), or [Edge Distribution](https://vcluster.com/docs/vcluster/production-guide/edge-distribution.md) | ## Day 2 operations reference[​](#day-2-operations-reference "Direct link to Day 2 operations reference") Common operations that apply across all paths. | Operation | Read next | | ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Monitor Platform and tenant workloads | [Fleet Observability](https://vcluster.com/docs/platform/maintenance/observability/overview.md), [other architectures](https://vcluster.com/docs/platform/maintenance/monitoring/fleet-metrics-architectures.md) | | Back up and restore tenant clusters | [Snapshots](https://vcluster.com/docs/vcluster/manage/backup-restore/backup.md), [restore](https://vcluster.com/docs/vcluster/manage/backup-restore/restore.md), [Velero](https://vcluster.com/docs/vcluster/manage/backup-restore/velero.md) | | Back up and restore Platform | [Backup and restore Platform](https://vcluster.com/docs/platform/maintenance/backup-restore/backup-restore-platform.md), [Platform database](https://vcluster.com/docs/platform/install/air-gapped/platform-database.md) | | Upgrade Platform and tenant clusters | [Upgrade vCluster](https://vcluster.com/docs/vcluster/manage/upgrade/upgrade-version.md), [upgrade Platform](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/upgrade.md), [lifecycle policy](https://vcluster.com/docs/platform/maintenance/upgrade-migrate/lifecycle-policy.md) | | Rotate certificates | [Certificate rotation](https://vcluster.com/docs/vcluster/manage/certificates.md) | | Manage private worker nodes | [Manage private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/manage.md), [Auto Nodes](https://vcluster.com/docs/platform/administer/node-providers/overview.md) | | Scale and recover the platform | [Platform HA](https://vcluster.com/docs/platform/maintenance/reliability-scaling/high-availability.md), [multi-region Platform](https://vcluster.com/docs/platform/administer/clusters/advanced/multi-region/overview.md) | | Troubleshoot incidents | [vCluster troubleshoot](https://vcluster.com/docs/vcluster/troubleshoot/noexec-emptydir-volumes.md), [debug commands](https://vcluster.com/docs/vcluster/cli/vcluster_debug), [Platform troubleshooting](https://vcluster.com/docs/platform/troubleshoot/connected-cluster-connectivity.md) | --- # Single-Tenant Per Customer Give each enterprise customer their own complete cluster stack. Dedicated nodes, an isolated tenant cluster, and full cluster-admin access if you choose to offer it. Your platform team handles provisioning and lifecycle. Customers see only their own environment. **Customer access model:** External enterprise customers, optionally with cluster-admin on their own tenant cluster. **Worker-node model:** Private nodes per customer. See [Choose a worker node model](https://vcluster.com/docs/vcluster/production-guide/choose-worker-node-model.md). **Typical stack:** Platform + private nodes per customer. vMetal optional for physical server fleets. vNode for customers requiring privileged workload support. **What makes this path different:** Each customer is fully isolated at the node level. Customers may have cluster-admin rights to their own cluster while remaining invisible to each other and to the platform layer. This path is closely related to the [AI Cloud path](https://vcluster.com/docs/vcluster/production-guide/ai-cloud.md). The main distinction is whether your customers consume Kubernetes directly (this path) or through your own product API (AI Cloud). ## Day 0: Design decisions[​](#day-0-design-decisions "Direct link to Day 0: Design decisions") | Decision | Read next | Outcome | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Define per-customer node allocation | [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md), [Auto Nodes](https://vcluster.com/docs/platform/administer/node-providers/overview.md) | Each customer cluster maps to a dedicated node pool. Auto Nodes provisions and reclaims nodes as customers are added or removed. | | Define the default network policy posture | [Network policy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md) | Set a default-deny east-west posture per customer template and confirm the control plane cluster's CNI enforces it. Some CNIs accept NetworkPolicy objects without enforcing them. | | Choose the GPU scheduling model | [DeviceClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/device-classes.md), [DRA ResourceClaims](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaims.md) | Device-plugin-style `DeviceClass` sync covers simple, whole-GPU allocation. DRA `ResourceClaim` sync (Pro) supports fine-grained and shared GPU allocation, relevant if customers get cluster-admin access to schedule their own workloads. | | Plan tenant storage | [Storage classes from host](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/storage-classes.md), [CSI storage capacities](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/csi-storage-capacities.md) | Decide which storage classes each customer can request, and whether to sync host `CSIStorageCapacity` for scheduler-aware placement. | | Plan provisioning automation | [Templates](https://vcluster.com/docs/platform/administer/templates/overview.md), [Terraform](https://vcluster.com/docs/platform/administer/node-providers/terraform.md) | Customer cluster creation is triggered by your CRM or provisioning system, not by manual Platform operations. | | Define the customer application baseline | [Stacks](https://vcluster.com/docs/platform/understand/what-are-stacks.md), [create a Stack template](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) | If every customer receives the same multi-application baseline, define its Apps or Argo CD Applications, dependencies, and health gates once. | | Define offboarding and node reclaim | [Manage private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/manage.md), [vMetal docs](https://www.vmetal.ai/docs/) | Document how nodes are reclaimed and reimaged when a customer offboards. | | Plan security hardening and audit logging | [Security baseline](https://vcluster.com/docs/vcluster/security/.md), [CIS hardening guide](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/overview.md), [Platform audit logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md) | Apply a CIS benchmark baseline to the control plane cluster and decide audit log retention and SIEM integration, especially where customers hold cluster-admin access to their own cluster. | ## Day 1: Stand up the first customer cluster[​](#day-1-stand-up-the-first-customer-cluster "Direct link to Day 1: Stand up the first customer cluster") 1. Install [vCluster Platform](https://vcluster.com/docs/platform/install/quick-start-guide.md). 2. Configure [Auto Nodes](https://vcluster.com/docs/platform/administer/node-providers/overview.md) or [vMetal](https://www.vmetal.ai/docs/) for automated node provisioning and reclaim. 3. Create a per-customer cluster template with [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md), [backing store](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store.md), and [HA](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/high-availability.md). 4. Apply a default-deny [network policy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md) in the template and confirm the control plane cluster's CNI enforces it. 5. Configure per-customer [storage classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/storage-classes.md) in the template. 6. Configure GPU scheduling: sync [DeviceClasses](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/device-classes.md) for device-plugin allocation, or enable [DRA ResourceClaim sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/DRA/resourceclaims.md) for fine-grained allocation. 7. Apply the [CIS hardening baseline](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/overview.md) to the control plane cluster and enable [Platform audit logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md). 8. If customers receive a standard application baseline, [create a StackTemplate](https://vcluster.com/docs/platform/administer/templates/create-stack-templates.md) and [include it in the tenant cluster configuration](https://vcluster.com/docs/platform/use-platform/apps/use-stacks.md#choose-how-to-install-a-stack). 9. Wire provisioning automation to Platform using Terraform, CLI, or API. 10. Validate the full customer lifecycle: provision, validate isolation, access as cluster-admin, offboard, and verify node reclaim. ## Day 2: Operate[​](#day-2-operate "Direct link to Day 2: Operate") | Operation | Read next | | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Add and remove customers | [Auto Nodes](https://vcluster.com/docs/platform/administer/node-providers/overview.md), provisioning automation | | Back up and restore per-customer | [Snapshots](https://vcluster.com/docs/vcluster/manage/backup-restore/backup.md), [restore](https://vcluster.com/docs/vcluster/manage/backup-restore/restore.md) | | Upgrade customer clusters | [Upgrade vCluster](https://vcluster.com/docs/vcluster/manage/upgrade/upgrade-version.md) | | Track GPU utilization and faults for metering and troubleshooting | [GPU observability templates](https://vcluster.com/docs/platform/maintenance/observability/gpu-observability-templates.md), [NVSentinel GPU observability](https://vcluster.com/docs/platform/maintenance/observability/nvsentinel-gpu-observability.md) | | Review audit logs and security posture | [Platform audit logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md), [security baseline](https://vcluster.com/docs/vcluster/security/.md) | --- # Docker (vind) Quick Start You want a full Kubernetes cluster running locally for development, testing, or CI. You don't want the setup overhead of a cloud environment or the limitations of KinD. vind (vCluster in Docker) runs a complete Kubernetes cluster in Docker containers. No cloud account, no existing Kubernetes cluster, no kubeadm. If Docker is running, you can have a cluster in under a minute. When you're ready to test against real cloud hardware, vind clusters can also accept actual EC2 or GCP instances as worker nodes through vCluster VPN. That's a hybrid setup that KinD simply can't do. This guide covers deployment, pause and resume, LoadBalancer services, and virtual worker nodes. Here's how vind compares to KinD: | Feature | vind | KinD | | ------------------------------------- | ---------------------------------------- | ----------------------------- | | Pause and resume without losing state | Yes | No — must delete and recreate | | LoadBalancer services | Automatic (Linux); port-forward on macOS | Manual setup required | | Virtual worker nodes | Yes | Limited | | Join real cloud instances | Yes, using VPN | No | | Pull-through image cache | Uses host Docker daemon | Direct registry pulls | ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Docker installed and running * At least 4 GB RAM available to Docker (Rancher Desktop or Docker Desktop VM size) ## Install the vCluster CLI[​](#install-the-vcluster-cli "Direct link to Install the vCluster CLI") * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` ## Deploy a cluster[​](#deploy-a-cluster "Direct link to Deploy a cluster") Restricted networks If outbound access must use a forward proxy, complete the [restricted network setup](https://vcluster.com/docs/vcluster/deploy/control-plane/docker-container/air-gapped.md) before running these steps. 1. Configure vCluster to use the Docker driver. ``` vcluster use driver docker ``` 2. Deploy a cluster. ``` vcluster create my-vcluster ``` note vCluster deploys Kubernetes **v1.36.0** by default. To use a different version, set `controlPlane.distro.k8s.image.tag` in your `vcluster.yaml`. When deployment finishes, the CLI connects and switches your kube context into the cluster. 3. Verify the cluster is running. ``` kubectl get nodes ``` ## Pause and resume[​](#pause-and-resume "Direct link to Pause and resume") A vind cluster can pause and resume without losing state. A KinD cluster requires a full delete and recreate. ``` # Pause the cluster — frees Docker resources immediately vcluster pause my-vcluster # Check the status vcluster list ``` Output is similar to: ``` NAME | STATUS | CONNECTED | AGE --------------+--------+-----------+----- my-vcluster | exited | True | 5m ``` ``` # Resume — cluster is back in seconds vcluster resume my-vcluster vcluster connect my-vcluster ``` Paused clusters retain all their state. After resume, workloads restart where they left off. If `vcluster disconnect` prompts you to select a context, choose your previous cluster context (for example `rancher-desktop` or `docker-desktop`). ## LoadBalancer services[​](#loadbalancer-services "Direct link to LoadBalancer services") On Linux, LoadBalancer services get an external IP automatically. No MetalLB or extra configuration required. ``` kubectl create deployment hello --image=nginx kubectl expose deployment hello --port=80 --type=LoadBalancer kubectl get service hello --watch ``` Output on Linux is similar to: ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hello LoadBalancer 10.96.x.x 127.0.0.1 80:31234/TCP 5s ``` ``` curl http://localhost ``` macOS limitation On macOS with Docker Desktop or Rancher Desktop, the LoadBalancer IP will not be assigned. Docker runs inside a Linux VM and the bridge network IP is not routable on the macOS host. Use `kubectl port-forward` instead: ``` kubectl port-forward deployment/hello 8080:80 curl http://localhost:8080 ``` ## Add virtual worker nodes[​](#add-virtual-worker-nodes "Direct link to Add virtual worker nodes") Test node affinity rules, DaemonSet scheduling, or topology spread constraints without real hardware. Add virtual worker nodes by configuring them in a `vcluster.yaml` at creation time. Delete the current cluster first to free resources. Running two clusters simultaneously requires at least 4 GB RAM in your Docker VM. ``` vcluster delete my-vcluster ``` vcluster.yaml ``` experimental: docker: nodes: - name: worker-1 - name: worker-2 ``` ``` vcluster create my-cluster --values vcluster.yaml kubectl get nodes ``` Output is similar to: ``` NAME STATUS ROLES AGE my-cluster Ready control-plane 30s worker-1 Ready 25s worker-2 Ready 25s ``` Schedule workloads to specific nodes, add labels and taints, or simulate a realistic multi-node topology. All from a single Docker host. ## Join real cloud nodes[​](#join-real-cloud-nodes "Direct link to Join real cloud nodes") vind clusters accept real cloud instances as worker nodes through vCluster VPN. An EC2 instance, GCP VM, or any Linux machine can join your local vind cluster through a VPN tunnel. The result is a hybrid cluster with a local control plane and real remote compute. This requires vCluster Platform. See [vCluster VPN documentation](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/vpn.md) for setup instructions. ## Clean up[​](#clean-up "Direct link to Clean up") ``` vcluster delete my-cluster ``` All Docker containers and networks created for the cluster are removed. On macOS you may see a `permission denied` warning about a leftover file in `~/.vcluster/docker/`. This is benign. The cluster is fully deleted. ## Next steps[​](#next-steps "Direct link to Next steps") * [vind configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/docker.md) — ports, volumes, registry proxy, multi-node options * [Restricted network access](https://vcluster.com/docs/vcluster/deploy/control-plane/docker-container/air-gapped.md) — forward proxy, registry mirror, and image override configuration * [vCluster VPN](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/vpn.md) — join real cloud nodes to a vind cluster * [Shared nodes](https://vcluster.com/docs/vcluster/quick-start/shared-nodes.md) — when you're ready to move to a real Kubernetes cluster * [vcluster.yaml reference](https://vcluster.com/docs/vcluster/configure/what-is-vcluster-yaml.md) — full configuration reference Ready for production? See [Build for Production](https://vcluster.com/docs/vcluster/production-guide/overview.md) to find the path that matches what you're building. --- # Private Nodes Quick Start You're building a platform where tenants need real hardware isolation. Think GPU workloads that can't share a node pool, regulated environments that require dedicated infrastructure, or AI cloud customers who expect hyperscaler-grade separation. This guide shows you how to deploy a tenant cluster backed by dedicated worker nodes. There's no cross-tenant scheduling, and each tenant gets a separate CNI and separate storage. At the end, you'll see exactly what is and isn't visible from the Control Plane Cluster. Private nodes require vCluster Platform. Free mode is supported, so no paid license is needed to get started. Worker nodes connect to the control plane through vCluster VPN, which routes traffic through the Platform URL using WireGuard. Worker nodes only need outbound HTTPS access to Platform — no direct exposure of the control plane API is required. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") **You need a Control Plane Cluster with vCluster Platform running on it.** * Have Platform already? Continue below. * No cluster yet? Start with the [Standalone quick start](https://vcluster.com/docs/vcluster/quick-start/standalone.md) to bootstrap a Control Plane Cluster and install Platform in one pass. * Have a cluster but no Platform? Run `vcluster platform start` against it. See the [Platform install guide](https://vcluster.com/docs/platform/install/quick-start-guide.md). **You also need at least one Linux machine to join as a worker node.** * Ubuntu 22.04 or 24.04 recommended * Root access on the worker node * Outbound HTTPS access to the Platform URL from the worker node See the full [node requirements](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/node-requirements.md). ## Create the tenant cluster[​](#create-the-tenant-cluster "Direct link to Create the tenant cluster") Private nodes must be enabled at creation time An existing tenant cluster cannot be migrated from shared to private nodes. This configuration must be set when the cluster is first created. 1. Make sure your kube context is pointed at the Control Plane Cluster. ``` kubectl config current-context ``` 2. Create a `vcluster.yaml` with private nodes and VPN enabled. vcluster.yaml ``` privateNodes: enabled: true vpn: enabled: true networking: podCIDR: 10.64.0.0/16 serviceCIDR: 10.128.0.0/16 ``` note vCluster deploys Kubernetes **v1.36.0** by default. To use a different version, set `controlPlane.distro.k8s.image.tag` in your `vcluster.yaml`. 3. Deploy the tenant cluster. ``` vcluster create my-vcluster --namespace team-x --values vcluster.yaml ``` When deployment finishes, the CLI connects and switches your kube context into the tenant cluster. The cluster has no worker nodes yet. Workloads cannot be scheduled until at least one node joins. 4. Confirm the tenant cluster is running with no nodes. ``` kubectl get nodes ``` Expected output: ``` No resources found. ``` This is expected. Private nodes join explicitly. The control plane is running. Compute comes next. ## Join a worker node[​](#join-a-worker-node "Direct link to Join a worker node") Modify the following with your specific values to replace on the whole page and generate copyable commands: PLATFORM\_URLhttps\://my-platform.example.com 1. Generate a join token. Run this from your local machine with the kube context still inside the tenant cluster. ``` vcluster token create --expires=1h ``` The output is a `curl` command to run on the worker node: ``` curl -sfLk https://my-platform.example.com/node/join?token= | sh - ``` 2. SSH into the worker node and run the join command as root. ``` sudo su - curl -sfLk https://my-platform.example.com/node/join?token= | sh - ``` The script installs Kubernetes node components (containerd, kubelet) and the `vcluster-vpn` service, which establishes a WireGuard tunnel to the control plane through Platform. Output ends with: ``` This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The kubelet was informed of the new secure connection details. ``` 3. Back on your workstation, verify the node joined. ``` kubectl get nodes ``` Output is similar to: ``` NAME STATUS ROLES AGE VERSION worker-node-1 Ready 30s v1.36.0 ``` ## Deploy a workload[​](#deploy-a-workload "Direct link to Deploy a workload") With a worker node ready, schedule a workload to confirm the tenant cluster is fully operational. ``` kubectl create deployment hello --image=nginx kubectl get pods --watch ``` The pod schedules onto `worker-node-1`. No other tenant can see this node or the workloads running on it. ## The isolation boundary[​](#the-isolation-boundary "Direct link to The isolation boundary") From the Control Plane Cluster, the worker node that joined the tenant cluster is not visible as a Kubernetes node: ``` vcluster disconnect kubectl get nodes ``` The worker node does not appear here. It exists only within the tenant cluster's namespace. No other tenant cluster can schedule onto it. No operator on the Control Plane Cluster can inspect its workloads through the Kubernetes API. This is what makes private nodes the right model for GPU workloads. Each tenant gets hardware-level isolation without needing a separate physical cluster. ## Clean up[​](#clean-up "Direct link to Clean up") Reset the worker node before you delete the cluster. Platform stops serving the join script once the cluster is gone. Connect to the cluster and generate a new join token: ``` vcluster connect my-vcluster --namespace team-x vcluster token create --expires=10m ``` On the worker node, run the returned command as root with `--reset-only`, then stop the `vcluster-vpn` service. The `--reset-only` flag removes the node components but leaves `vcluster-vpn` running with this cluster's VPN state, which makes the next join to a new cluster hang. ``` sudo su - curl -sfLk https://my-platform.example.com/node/join?token= | sh -s -- --reset-only systemctl disable --now vcluster-vpn rm -f /etc/vcluster-vpn/config.json ``` Confirm the service stopped: ``` systemctl is-active vcluster-vpn ``` The output is `inactive`. Delete the cluster: ``` vcluster disconnect vcluster delete my-vcluster --namespace team-x ``` ## Next steps[​](#next-steps "Direct link to Next steps") * [Auto Nodes](https://vcluster.com/docs/platform/administer/node-providers/overview.md) — automate worker node provisioning through Platform instead of joining manually * [Node-to-node VPN](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/vpn.md) — extend VPN to encrypt traffic between worker nodes (useful for multi-cloud and cross-datacenter deployments) * [Node requirements](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes/node-requirements.md) — supported OS versions and system dependencies * [High availability](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md) — multiple control plane nodes for production * [Building a GPU cloud platform](https://vcluster.com/docs/vcluster/introduction/gpu-cloud-platform.md) — end-to-end architecture guide Ready for production? See [Build for Production](https://vcluster.com/docs/vcluster/production-guide/overview.md) to find the path that matches what you're building. --- # Shared Nodes Quick Start You're running a Kubernetes cluster and want to give internal teams or CI pipelines their own isolated Kubernetes environments. You don't want to spin up new infrastructure for each one. This guide shows you how to deploy a tenant cluster onto your existing cluster in under two minutes. What you deploy is a nested cluster, the tenancy model vCluster OSS provides. See [What is vCluster?](https://vcluster.com/docs/vcluster/introduction/what-is-vcluster.md) for how it compares to private nodes. You'll also see the isolation boundary in action. From inside the tenant cluster, everything looks like a normal Kubernetes cluster. From the outside, each tenant fits in a single namespace with no visibility into anyone else's workloads. By the end you'll have a working tenant cluster and a clear picture of how the isolation model works. There's also an optional step to connect it to vCluster Platform for self-service access and auto-sleep. Shared nodes are for trusted tenants Shared nodes are a supported, common model for trusted tenants. They give each tenant control-plane, API, and namespace isolation, but tenant workloads share the same kernel and physical nodes. They aren't a security boundary for untrusted tenants with Kubernetes access or arbitrary workload execution. Good fits are development, testing, CI/CD, and internal engineering teams. Use [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for external, resale, regulated, or otherwise untrusted tenant offerings, optionally with [vNode](https://www.vnode.com/docs/) for runtime isolation. NetworkPolicy is an added isolation layer worth enabling even for trusted tenants. vCluster can create the policies for you through `policies.networkPolicy`, and your control plane cluster's CNI enforces them. Confirm your CNI supports enforcement, since some accept NetworkPolicy resources without acting on them. See the [security baseline](https://vcluster.com/docs/vcluster/security/.md). If you accept shared nodes for a trusted-tenant use case, follow the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) as the required next step. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * A Kubernetes cluster with `kubectl` configured to access it * At least 2 vCPU and 2 GB RAM available on the cluster ## Install the vCluster CLI[​](#install-the-vcluster-cli "Direct link to Install the vCluster CLI") * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` ## Deploy a tenant cluster[​](#deploy-a-tenant-cluster "Direct link to Deploy a tenant cluster") 1. Deploy a tenant cluster named `my-vcluster` into a new namespace `team-x`. ``` vcluster create my-vcluster --namespace team-x ``` note vCluster deploys Kubernetes **v1.36.0** by default. To use a different version, set `controlPlane.distro.k8s.image.tag` in your `vcluster.yaml`. When the deployment finishes, the CLI automatically connects and switches your kube context into the tenant cluster. 2. Confirm you are inside the tenant cluster. ``` kubectl get namespaces ``` You see standard Kubernetes system namespaces. This is the tenant cluster's own namespace view, separate from the Control Plane Cluster. ## See the isolation[​](#see-the-isolation "Direct link to See the isolation") 1. Deploy a test workload inside the tenant cluster. ``` kubectl create deployment nginx --image=nginx kubectl get pods ``` The pod is running inside the tenant cluster. 2. Disconnect from the tenant cluster and inspect the same namespace from the Control Plane Cluster. ``` vcluster disconnect kubectl get pods -n team-x ``` Output is similar to: ``` NAME READY STATUS RESTARTS AGE coredns-79cf5f4c56-nkgdr-x-kube-system-x-my-vcluster 1/1 Running 0 4m my-vcluster-0 1/1 Running 0 5m nginx-56c45fd5ff-kp74s-x-default-x-my-vcluster 1/1 Running 0 2m ``` The nginx pod is here, but with a rewritten name. vCluster syncs tenant workloads into this namespace so the underlying cluster can schedule them. The suffix (`-x-default-x-my-vcluster`) encodes the source namespace and cluster. From inside the tenant cluster, the pod appeared as plain `nginx` in namespace `default`. This is the isolation boundary. The tenant gets a clean Kubernetes API with no visibility into the Control Plane Cluster's structure. Other tenant clusters have their own dedicated namespaces, so neither tenant can see the other's workloads. This isolation is at the API and namespace level. Tenants still share the same kernel and physical nodes. That's why shared nodes are for internal, trusted tenants rather than externally facing or resold offerings. ## Verify network policy enforcement[​](#verify-network-policy-enforcement "Direct link to Verify network policy enforcement") Shared-node isolation relies on the control plane cluster's CNI actually enforcing NetworkPolicy. Some CNIs accept policy resources without enforcing them, which silently leaves tenant traffic open. Confirm enforcement before you rely on it. On shared nodes, tenant workloads run on the control plane cluster, so the control plane cluster's CNI does the enforcement. A NetworkPolicy created inside the tenant cluster only takes effect once vCluster syncs it to the control plane cluster. That sync is off by default, so enable it first. 1. Enable NetworkPolicy sync on the tenant cluster. ``` cat < values.yaml sync: toHost: networkPolicies: enabled: true EOF vcluster create my-vcluster --namespace team-x --upgrade -f values.yaml ``` 2. Deploy a target pod with a service and a client pod. ``` kubectl run web --image=nginx --port=80 kubectl expose pod web --port=80 kubectl run client --image=busybox --restart=Never -- sleep 3600 ``` Confirm the client can reach the target while traffic is still open. ``` kubectl exec client -- wget -qO- --timeout=3 web ``` 3. Apply a default-deny ingress policy, then test the connection again. ``` kubectl apply -f - <:6443 ``` Verify access from your workstation: ``` kubectl get nodes ``` ## What you have now[​](#what-you-have-now "Direct link to What you have now") * A bare-metal or VM-based Kubernetes cluster running as a vCluster Standalone instance * A Control Plane Cluster ready to host tenant clusters and vCluster Platform ## Next steps[​](#next-steps "Direct link to Next steps") * [Install vCluster Platform](https://vcluster.com/docs/platform/install/helm.md) — add the Platform management UI to this cluster for tenant cluster provisioning, access controls, and node automation * [High availability](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md) — add control plane nodes for production resilience * [Building a GPU cloud platform](https://vcluster.com/docs/vcluster/introduction/gpu-cloud-platform.md) — end-to-end architecture for AI cloud deployments Ready for production? See [Build for Production](https://vcluster.com/docs/vcluster/production-guide/overview.md) to find the path that matches what you're building. --- # Annotations and labels reference This page documents the well-known annotations and labels in the `vcluster.loft.sh` namespace used by vCluster for resource synchronization between tenant and control plane clusters. ## Sync identification labels[​](#sync-identification-labels "Direct link to Sync identification labels") These labels identify synced resources and their relationship to the vCluster. ### vcluster.loft.sh/name[​](#vcluster-loft-sh-name "Direct link to vcluster.loft.sh/name") **Type:** Label **Example:** `vcluster.loft.sh/name: "my-vcluster"` **Used on:** All synced resources Identifies the name of the vCluster that owns or synced this resource. ### vcluster.loft.sh/namespace[​](#vcluster-loft-sh-namespace "Direct link to vcluster.loft.sh/namespace") **Type:** Label **Example:** `vcluster.loft.sh/namespace: "vcluster-my-vcluster"` **Used on:** All synced resources Identifies the namespace where the vCluster is deployed in the control plane cluster. ### vcluster.loft.sh/managed-by[​](#vcluster-loft-sh-managed-by "Direct link to vcluster.loft.sh/managed-by") **Type:** Label **Example:** `vcluster.loft.sh/managed-by: "vcluster"` **Used on:** Synced resources Indicates that this resource is managed by vCluster and should not be modified directly. ### vcluster.loft.sh/protected[​](#vcluster-loft-sh-protected "Direct link to vcluster.loft.sh/protected") **Type:** Label **Example:** `vcluster.loft.sh/protected: "apiservice"` **Used on:** Service and Endpoint resources backing APIServices Identifies APIService backend resources that should be protected from deletion by ValidatingAdmissionPolicy. The label value names the feature owning the resource for scoped policy bindings. ## Sync control annotations[​](#sync-control-annotations "Direct link to Sync control annotations") These annotations control how vCluster syncs resources between the tenant and control plane clusters. ### vcluster.loft.sh/force-sync[​](#vcluster-loft-sh-force-sync "Direct link to vcluster.loft.sh/force-sync") **Type:** Annotation **Example:** `vcluster.loft.sh/force-sync: "true"` **Used on:** Any resource Forces vCluster to sync this resource from the control plane cluster to the tenant cluster, even if it would normally be excluded by sync configuration. ### vcluster.loft.sh/skip-translate[​](#vcluster-loft-sh-skip-translate "Direct link to vcluster.loft.sh/skip-translate") **Type:** Annotation **Example:** `vcluster.loft.sh/skip-translate: "true"` **Used on:** Any resource Prevents vCluster from translating (name-mangling) this resource when syncing. The resource retains its original name in the control plane cluster. ### vcluster.loft.sh/skip-backsync[​](#vcluster-loft-sh-skip-backsync "Direct link to vcluster.loft.sh/skip-backsync") **Type:** Annotation **Example:** `vcluster.loft.sh/skip-backsync: "true"` **Used on:** Synced resources Prevents changes to this resource in the control plane cluster from being synced back to the tenant cluster. ### vcluster.loft.sh/import[​](#vcluster-loft-sh-import "Direct link to vcluster.loft.sh/import") **Type:** Annotation **Example:** `vcluster.loft.sh/import: "true"` **Used on:** Resources in control plane cluster Marks a resource in the control plane cluster for import into the vCluster. The resource is synced into the tenant cluster. ### vcluster.loft.sh/preserve-host-rule[​](#vcluster-loft-sh-preserve-host-rule "Direct link to vcluster.loft.sh/preserve-host-rule") **Type:** Annotation **Example:** `vcluster.loft.sh/preserve-host-rule: "mirror-rule"` **Used on:** HTTPRoute resources Names a host HTTPRoute rule that vCluster should preserve when syncing the HTTPRoute between tenant and control plane clusters. ### vcluster.loft.sh/preserve-request-mirror-filters[​](#vcluster-loft-sh-preserve-request-mirror-filters "Direct link to vcluster.loft.sh/preserve-request-mirror-filters") **Type:** Annotation **Example:** `vcluster.loft.sh/preserve-request-mirror-filters: "true"` **Used on:** HTTPRoute resources Keeps host-added HTTPRoute RequestMirror filters when syncing the HTTPRoute from the control plane cluster to the tenant cluster. ## Internal annotations[​](#internal-annotations "Direct link to Internal annotations") These annotations are used internally by vCluster for sync tracking. They should not be modified manually. ### vcluster.loft.sh/object-name[​](#vcluster-loft-sh-object-name "Direct link to vcluster.loft.sh/object-name") **Type:** Annotation (internal) **Used on:** Synced resources Stores the original name of the resource in the tenant cluster. ### vcluster.loft.sh/object-namespace[​](#vcluster-loft-sh-object-namespace "Direct link to vcluster.loft.sh/object-namespace") **Type:** Annotation (internal) **Used on:** Synced resources Stores the original namespace of the resource in the tenant cluster. ### vcluster.loft.sh/object-uid[​](#vcluster-loft-sh-object-uid "Direct link to vcluster.loft.sh/object-uid") **Type:** Annotation (internal) **Used on:** Synced resources Stores the UID of the resource in the tenant cluster. ### vcluster.loft.sh/object-kind[​](#vcluster-loft-sh-object-kind "Direct link to vcluster.loft.sh/object-kind") **Type:** Annotation (internal) **Used on:** Synced resources Stores the Kind of the resource in the tenant cluster. ### vcluster.loft.sh/object-host-name[​](#vcluster-loft-sh-object-host-name "Direct link to vcluster.loft.sh/object-host-name") **Type:** Annotation (internal) **Used on:** Synced resources Stores the translated name of the resource in the control plane cluster. ### vcluster.loft.sh/object-host-namespace[​](#vcluster-loft-sh-object-host-namespace "Direct link to vcluster.loft.sh/object-host-namespace") **Type:** Annotation (internal) **Used on:** Synced resources Stores the namespace of the resource in the control plane cluster. ### vcluster.loft.sh/object-imported[​](#vcluster-loft-sh-object-imported "Direct link to vcluster.loft.sh/object-imported") **Type:** Annotation (internal) **Used on:** Imported resources Marks a resource as imported from the control plane cluster. ### vcluster.loft.sh/service-cidr[​](#vcluster-loft-sh-service-cidr "Direct link to vcluster.loft.sh/service-cidr") **Type:** Annotation (internal) **Used on:** vCluster configuration Stores the service CIDR used by the vCluster. ### vcluster.loft.sh/created[​](#vcluster-loft-sh-created "Direct link to vcluster.loft.sh/created") **Type:** Annotation (internal) **Used on:** Helm releases Indicates that this release was created by the vCluster CLI. ### vcluster.loft.sh/distro[​](#vcluster-loft-sh-distro "Direct link to vcluster.loft.sh/distro") **Type:** Label (internal) **Used on:** vCluster pods Identifies the Kubernetes distribution running inside the vCluster (k8s). ### vcluster.loft.sh/belongs-to[​](#vcluster-loft-sh-belongs-to "Direct link to vcluster.loft.sh/belongs-to") **Type:** Label **Example:** `vcluster.loft.sh/belongs-to: "my-vcluster"` **Used on:** Service resources Identifies which vCluster the node service belongs to when there are multiple vClusters in one namespace. ### vcluster.loft.sh/node[​](#vcluster-loft-sh-node "Direct link to vcluster.loft.sh/node") **Type:** Label **Example:** `vcluster.loft.sh/node: "node-1"` **Used on:** Service resources Specifies which node this service represents in the vCluster. ### vcluster.loft.sh/restore-request[​](#vcluster-loft-sh-restore-request "Direct link to vcluster.loft.sh/restore-request") **Type:** Label **Example:** `vcluster.loft.sh/restore-request: "true"` **Used on:** ConfigMap resources Labels ConfigMaps as restore requests for vCluster snapshot restoration. ### vcluster.loft.sh/snapshot-request[​](#vcluster-loft-sh-snapshot-request "Direct link to vcluster.loft.sh/snapshot-request") **Type:** Label **Example:** `vcluster.loft.sh/snapshot-request: "true"` **Used on:** ConfigMap resources Labels ConfigMaps as snapshot requests for vCluster state capture. ## Platform integration[​](#platform-integration "Direct link to Platform integration") These annotations are used when vCluster is managed by vCluster Platform. ### loft.sh/custom-links[​](#loft-sh-custom-links "Direct link to loft.sh/custom-links") **Type:** Annotation **Example:** `loft.sh/custom-links: "https://docs.example.com\nhttps://dashboard.example.com"` **Used on:** Cluster resources Enumerates associated links to external websites. Multiple URLs can be specified using newline separators. ### loft.sh/direct-cluster-endpoint[​](#loft-sh-direct-cluster-endpoint "Direct link to loft.sh/direct-cluster-endpoint") **Type:** Annotation **Example:** `loft.sh/direct-cluster-endpoint: "https://cluster.example.com:6443"` **Used on:** Cluster resources Tells the vCluster CLI to use this endpoint instead of the default platform server address to connect to this cluster. ### loft.sh/direct-cluster-endpoint-insecure[​](#loft-sh-direct-cluster-endpoint-insecure "Direct link to loft.sh/direct-cluster-endpoint-insecure") **Type:** Annotation **Example:** `loft.sh/direct-cluster-endpoint-insecure: "true"` **Used on:** Cluster resources Specifies whether to use an insecure connection when connecting to the cluster endpoint. --- # vCluster glossary This page contains definitions for common terms used throughout the vCluster documentation. ### Admission Control A Kubernetes feature that intercepts requests to the API server prior to persistence of the object, allowing for validation, mutation, or rejection of requests based on configured policies. **Related terms:** API Server, Control Plane ### API Server The core component of Kubernetes that exposes the Kubernetes API. It is the front-end for the Kubernetes control plane and handles all REST operations, validating and configuring data for API objects. **Related terms:** Control Plane, rate-limiting ### Cluster A certified Kubernetes distribution that runs as an isolated environment with a virtualized control plane hosted on a control plane cluster. Clusters operate as independent Kubernetes environments, each with its own API server, control plane, syncer, and set of resources. **Related terms:** vCluster, Control plane cluster ### Cluster A fully isolated Kubernetes environment provisioned for a single tenant. Each cluster has its own API server, controller manager, and resource namespace, backed by a virtualized control plane hosted on a control plane cluster. From the tenant's perspective it behaves exactly like a standard Kubernetes cluster. **Related terms:** Control plane cluster, Cluster ### Compaction The process of removing superseded data from etcd to free up storage space. Regular compaction prevents etcd from running out of disk space. **Related terms:** etcd, Defragmentation ### Control Plane The container orchestration layer that exposes the API and interfaces to define, deploy, and manage the lifecycle of containers. In vCluster, each cluster has its own control plane components. **Related terms:** API Server, vCluster ### Control plane cluster The Kubernetes cluster that hosts the virtualized control planes for clusters. The control plane cluster is operated by the platform provider and is completely invisible to tenants. There are no shared control plane nodes, no in-cluster agent pods, and no lateral path between tenant environments. With shared nodes, this cluster also runs tenant workloads alongside the control plane pods — the same node pool is used for both. **Related terms:** Cluster, Control plane cluster ### Control plane cluster The Kubernetes cluster that hosts the virtualized control planes for clusters. The control plane cluster is operated by the platform provider and is completely invisible to tenants. There are no shared control plane nodes, no in-cluster agent pods, and no lateral path between tenant environments. With shared nodes, this cluster also runs tenant workloads alongside the control plane pods — the same node pool is used for both. **Related terms:** Cluster, Control plane cluster, Cluster ### Defragmentation The process of releasing unused space in etcd's backend database after compaction. While compaction marks space as free, defragmentation reclaims it. **Related terms:** etcd, Compaction ### etcd A distributed key-value store that provides reliable storage for Kubernetes cluster data. In vCluster, etcd can be deployed externally or embedded within the vCluster pod. **Related terms:** Control Plane ### K0s A lightweight, certified Kubernetes distribution that can be used in clusters as an alternative to K3s or standard Kubernetes. **Related terms:** K3s, K8s ### K3s A lightweight, certified Kubernetes distribution often used as the default distribution for clusters due to its minimal resource requirements and fast startup time. **Related terms:** K8s, K0s ### K8s The standard Kubernetes distribution that can be used in clusters, offering full compatibility with upstream Kubernetes features. **Related terms:** K3s, K0s ### Pseudo Nodes The default node representation in a cluster. vCluster creates pseudo nodes for each host node that has pods scheduled on it. Pseudo nodes have real values for name, CPU, architecture, and operating system, but all other fields are randomly generated. When no pods remain on a host node, vCluster deletes the corresponding pseudo node. **Related terms:** Syncer, Control plane cluster ### Syncer A component in vCluster that synchronizes resources between the cluster and the control plane cluster, enabling clusters to function while maintaining isolation. **Related terms:** vCluster, Cluster ### Tenant Isolation The capability to host multiple separate users, teams, or workloads on the same infrastructure while providing strong isolation between them. vCluster delivers tenant isolation through dedicated control planes, isolated resource namespaces, and optional private nodes per tenant. **Related terms:** Cluster ### vCluster An open-source software product that creates and manages clusters within Kubernetes infrastructure. vCluster provides tenant isolation capabilities while reducing infrastructure costs. **Related terms:** Cluster, Control plane cluster --- # v0.20 CLI Updates With the breaking changes introduced in v0.20, the vCluster CLI was also updated to match the direction of the project. There were also two significant CLI changes that influenced the changes in the `vcluster` CLI. 1. The `loft` CLI, which was used to manage the platform (formerly known as loft) was deprecated and no longer supported, so all commands in the `loft` CLI were moved into the `vcluster` CLI. 2. The `vcluster pro` command was renamed to `vcluster platform`. With these two significant changes, there were duplicate features from the older CLI (`loft`) and vCluster command (`vcluster pro`) that were merged as `vcluster platform`. ## New commands in v0.20.0[​](#new-commands-in-v0200 "Direct link to New commands in v0.20.0") * `vcluster convert` : Converted a pre-v0.20 `values.yaml` to the new `vcluster.yaml` format. This command was removed from the vCluster CLI in v0.32.0. See the [dedicated migration guide](https://vcluster.com/docs/vcluster/reference/migrations/0-20-migration.md) for how to convert legacy values files with an older CLI release. * `vcluster set` : Set configuration * `vcluster use` : vCluster use subcommand - This command allows you to switch between the drivers (Helm or platform). ## Renamed commands from vCluster CLI v0.19.x[​](#renamed-commands-from-vcluster-cli-v019x "Direct link to Renamed commands from vCluster CLI v0.19.x") * `vcluster import` was renamed to `vcluster platform add` : Adds a cluster or vCluster to vCluster platform * `vcluster get` was renamed to `vcluster platform get` : Retrieves and displays information ## New commands that were used in `vcluster pro` and `loft`[​](#new-commands-that-were-used-in-vcluster-pro-and-loft "Direct link to new-commands-that-were-used-in-vcluster-pro-and-loft") * `loft credits` was renamed to `vcluster credits` : Saves the OSS credits * `loft generate` was replaced with `vcluster platform create` and `vcluster platform set` * `loft import` was renamed to `vcluster platform add` : Adds a cluster or vCluster to vCluster platform ### Changed subcommands[​](#changed-subcommands "Direct link to Changed subcommands") * `vcluster pro token` / `loft token` was renamed to `vcluster platform access-key`  * `vcluster pro connect` was renamed to `vcluster platform connect` with additional context. * `vcluster pro generate` was removed and replaced with `vcluster platform create` and `vcluster platform set` ### Renamed commands from `loft` CLI to `vcluster platform` CLI[​](#renamed-commands-from-loft-cli-to-vcluster-platform-cli "Direct link to renamed-commands-from-loft-cli-to-vcluster-platform-cli") This is a list of commands that were formerly in the `loft` CLI and now renamed in `vcluster` under `vcluster platform`. All features remain the same. * `vcluster platform backup` : Backup subcommands * `vcluster platform connect` : Connects a cluster to vCluster Platform.  * `vcluster platform create` : Creates vCluster platform resources * `vcluster platform delete` : Deletes vCluster platform resources * `vcluster platform get` : Retrieves and displays information * `vcluster platform list` : Lists configuration * `vcluster platform reset` : Reset configuration * `vcluster platform set` : Set configuration * `vcluster platform share` : Shares a tenant cluster/namespace with another vCluster platform user or team * `vcluster platform sleep` : Put a tenant cluster/namespace to sleep * `vcluster platform start` : Start a vCluster Platform instance and connect using port-forwarding * `vcluster platform wakeup` : Wake up a tenant cluster/namespace ## Flag differences[​](#flag-differences "Direct link to Flag differences") ### New global flag[​](#new-global-flag "Direct link to New global flag") •  `--config` string : The vCluster CLI config to use (will be created if it does not exist) ### New flags within each command[​](#new-flags-within-each-command "Direct link to New flags within each command") `vcluster connect` * `--driver` string : Specifies the driver to use for managing the tenant cluster (helm or platform). `vcluster create` * `--add` : Automatically adds the tenant cluster to the current vCluster platform when using the helm driver (default true). * `--background-proxy` : Uses a background-proxy to access the vCluster if docker is installed and reachable (default true). * `--create-context` : Creates a kube context for the namespace (default true). * `--description` string : Description to show in the platform UI for this tenant cluster.  * `--display-name` string : Display name to show in the platform UI for this tenant cluster. * `--driver` string : Specifies the driver to use for managing the tenant cluster (helm or platform). * `--parameters` string : Uses a file for the parameters if a template is used. * `--print` : Prints the context to the console if enabled. * `--recreate` : Deletes an existing tenant cluster with the same name before creating a new one. * `--set-parameter` stringArray : Sets specific parameters if a template is used. * `--skip-wait` : Does not wait until the tenant cluster is running if true.  * `--switch-context` : Switches the current context to the new context (default true). * `--team` string : Specifies the team to create the namespace for.  * `--use` : Uses the tenant cluster if it's already there.  * `--user` string : Specifies the user to create the namespace for.  `vcluster delete` * `--delete-context` : Deletes the corresponding kube context if there is any (default true). * `--driver` string : Specifies the driver to use for managing the tenant cluster (helm or platform). `vcluster list` * `--driver` string : Specifies the driver to use for managing the tenant cluster (helm or platform). `vcluster platform login`  * `--use-driver` string : Switches the vCluster driver between platform and helm.  `vcluster pause` * `--driver` string : Specifies the driver for the tenant cluster (helm or platform). `vcluster resume`: * `--driver` string : Specifies the driver for the tenant cluster (helm or platform).    ### Dropped flags within each command[​](#dropped-flags-within-each-command "Direct link to Dropped flags within each command") `vcluster connect` * `--kube-config` string: This flag was used to specify the file to write the created kube config to. The default value was "./kubeconfig.yaml".  * `--kube-config-context-name` string: This flag was used to override the context name of the generated tenant cluster kube config with a specified name.  * `--update-current`: This flag was used to update the current kube config if set to true. The default value was true.  `vcluster create` * `--disable-pro`: This flag was used to prevent vcluster from trying to create a vCluster.Pro. It could also be used with 'vcluster logout' to prevent vCluster from creating any pro clusters. In v0.20.0, the images used are the pro images and in order to use the OSS images, you'll need to replace the images as part of the vcluster.yaml. * `--distro` string: This flag was used to specify the Kubernetes distro to use for the tenant cluster. Allowed distros were k3s, k0s, k8s, eks, with the default being "k3s". In v0.20.0, configuring the distro is now in the vcluster.yaml with the default being "k8s". * `--extra-values` strings: This flag was deprecated and used to specify extra values. It was replaced by the  --values  flag. * `--isolate` : This flag was used to run vcluster and its workloads in an isolated environment. In v0.20.0, running workloads in an isolated environment is [configured within vcluster.yaml](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes/isolated-workloads.md). * `--params` string: This flag was used to specify a file for the parameters if a template was used. It was replaced by the  --parameters string  flag. --- # Conversion Guide from pre-v0.20 to v0.20 As part of the v0.20.0 release, LoftLabs introduced a new `vcluster.yaml` configuration file with a new structure. This file replaces your existing `values.yaml` configuration file. Previously, configuring a cluster had configuration options using CLI flags, annotations and different `values.yaml` options. With the new `vcluster.yaml`, all configuration options are now easily configured in a single source. warning Pre-v0.20.0 `values.yaml` files are not compatible with v0.20.0. You need to convert to the new format to use the new vCluster CLI and upgrade existing vCluster instances. Upgrade vCluster v0.19 to v0.20+ by following these steps: 1. [Upgrade the vCluster CLI](#upgrade-the-cli). 2. [Convert your Helm values configuration file to the new `vcluster.yaml` configuration file](#convert-your-valuesyaml-to-vclusteryaml). 3. [Use your new `vcluster.yaml` file to upgrade your vCluster instance](#upgrade-your-vcluster-instance). ## Supported legacy versions[​](#supported-legacy-versions "Direct link to Supported legacy versions") LoftLabs has tested the conversion path from v0.19.x to v0.20.0 and therefore recommends moving to v0.19.x before converting to v0.20.0. Conversions from vCluster versions prior to 0.19.x might work but have not been tested extensively. ## Before you begin[​](#before-you-begin "Direct link to Before you begin") * Helm version is >= v.3.10.0. * If you are using [vcluster-hostpath-mapper](https://github.com/loft-sh/vcluster-hostpath-mapper), upgrade it to v0.1.1-beta.1 or later for compatibility with the new vcluster.yaml configuration file in v0.20. * Review the [list of changes](https://vcluster.com/docs/vcluster/reference/migrations/0-20-cli-changes.md) in the vCluster CLI as commands, subcommands, and flags were removed, renamed or added. ## Upgrade the CLI[​](#upgrade-the-cli "Direct link to Upgrade the CLI") Before upgrading cluster instances, upgrade to the latest version of the vCluster CLI. ``` vcluster upgrade --version ``` Replace: * `` with the version of vCluster you want to upgrade to. The `convert config` command converts the old `values.yaml` format to the new `vcluster.yaml`. `convert config` was removed in v0.32.0 The `convert config` command was removed from the vCluster CLI in v0.32.0. Run this one-time conversion with a vCluster CLI release earlier than v0.32.0 (v0.31.3 is the last release that ships the command), then upgrade your instance with your current CLI. Download a pre-v0.32.0 CLI for your platform and invoke it as `./vcluster` for the conversion steps below. Adjust the OS and architecture in the URL if needed: ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/download/v0.31.3/vcluster-$(uname -s | tr '[:upper:]' '[:lower:]')-amd64" chmod +x vcluster ``` ``` Usage: ./vcluster convert config [flags] Flags: --distro string Kubernetes distro used -f, --file string Path to the input file -h, --help Help for config -o, --output string Prints the output in the specified format. Allowed values: yaml, json (default "yaml") ``` ## Get your previous `values.yaml` (Optional)[​](#get-your-previous-valuesyaml-optional "Direct link to get-your-previous-valuesyaml-optional") If you have the existing `values.yaml` that you provided to your existing vCluster, you can skip this step. ### Get the user-suppied values[​](#get-the-user-suppied-values "Direct link to Get the user-suppied values") If you don't have the `values.yaml`, use Helm to retrieve the user defined configuration for your currently running cluster. ``` helm get values -n > values.yaml ``` Replace: * `` with the name of your vCluster instance. * `` with the namespace that vCluster runs in. This uses `--revision 0` under the hood. Adjust this accordingly to receive the values of your latest revision. See the [Helm docs](https://helm.sh/docs/helm/helm_history/) for how to fetch revision history. ### Get the default values[​](#get-the-default-values "Direct link to Get the default values") If the `helm get` provides an empty file, then you used the default configuration to deploy your vCluster. You still need to supply a `values.yaml` to the convert due to changes in defaults. ``` helm get values -n --all > values.yaml ``` Replace: * `` with the name of your vCluster instance. * `` with the namespace that vCluster runs in. ## Convert your `values.yaml` to `vcluster.yaml`[​](#convert-your-valuesyaml-to-vclusteryaml "Direct link to convert-your-valuesyaml-to-vclusteryaml") Convert your `values.yaml` to the new `vcluster.yaml` configuration file format by running the pre-v0.32.0 CLI you downloaded above: ``` ./vcluster convert config --distro -f > vcluster.yaml ``` Replace: * `` with your cluster's existing Kubernetes distribution. **You can't change the Kubernetes distribution or backing store during an upgrade**. * `` with the path to the `values.yaml`file you used to configure your pre-v0.20 vCluster instance. For example: ``` ./vcluster convert config --distro k8s -f /my/k8s/values.yaml > vcluster.yaml ``` ### Notes about Changes for your `vcluster.yaml`[​](#notes-about-changes-for-your-vclusteryaml "Direct link to notes-about-changes-for-your-vclusteryaml") #### Scheduling of the StatefulSet[​](#scheduling-of-the-statefulset "Direct link to Scheduling of the StatefulSet") Since the scheduling of the control plane pod changed in v0.20+, the converted `vcluster.yaml` includes retaining the original scheduling pattern as it is can't be changed between upgrades. #### Image repository[​](#image-repository "Direct link to Image repository") By default, the conversion does not set the repo of the images to use the open source repo, but upgrade you to use the `loft-sh/vcluster-pro` image repository. This allows users to seamlessly test and adopt any Enterprise feature without disrupting their existing open source features. Enterprise features are integrated into the `loft-sh/vcluster-pro` image but remain inactive by default to ensure that your experience remains consistent with the open source version unless you specifically activate them with a vCluster Platform Enterprise plan. If you want to continue using the open source image, you need to explicitly set the open source repository for your images. ``` controlPlane: statefulSet: image: repository: loft-sh/vcluster-oss ``` #### EKS distribution support[​](#eks-distribution-support "Direct link to EKS distribution support") The v0.20 removed the support of EKS as a cluster distribution. reminder The Kubernetes distribution of the cluster doesn't need to match the distribution of your control plane cluster. You can use any distribution of vCluster to deploy vCluster on EKS clusters. In order to continue using your clusters that have EKS as the distribution, you can convert to using the k8s distribution and replace the images with the EKS images. While this was tested when v0.20.0 was released, it is not continuously tested. It's recommended spinning up new clusters with the k8s distribution and migrate your workloads to them. ``` ./vcluster convert config --distro k8s -f /my/k8s/values.yaml > vcluster.yaml ``` After the conversion, you need to update the `vcluster.yaml` and change the images to be used. Refer to the EKS registries for [kube-apiserver](https://gallery.ecr.aws/eks-distro/kubernetes/kube-apiserver), [controller-manager](https://gallery.ecr.aws/eks-distro/kubernetes/kube-controller-manager) and [kube-scheduler](https://gallery.ecr.aws/eks-distro/kubernetes/kube-scheduler) for the specific tags you might want to use. **Example using EKS v1.30:** ``` controlplane: distro: k8s: enabled: true apiserver: image: registry: "public.ecr.aws" repository: "eks-distro/kubernetes/kube-apiserver" tag: "v1.30.2-eks-1-30-10" controllerManager: image: registry: "public.ecr.aws" repository: "eks-distro/kubernetes/kube-controller-manager" tag: "v1.30.2-eks-1-30-10" scheduler: image: registry: "public.ecr.aws" repository: "eks-distro/kubernetes/kube-scheduler" tag: "v1.30.2-eks-1-30-10" ``` ## Upgrade your vCluster instance[​](#upgrade-your-vcluster-instance "Direct link to Upgrade your vCluster instance") Downgrading vCluster instances is NOT supported While downgrading to an older version of the vCluster CLI is supported, you cannot downgrade cluster instances to a prior version of vCluster (for example, from 0.20+ back to 0.19). Make sure you have tested the upgrade sufficiently before rolling out this change to critical systems. * vCluster CLI * Helm * Terraform * Argo CD * Cluster API vCluster CLI Version To keep the same version of your vCluster, your CLI needs to be the same version as your cluster. ``` vcluster create --upgrade -n -f vcluster.yaml ``` Replace: * `` with the name of the vCluster instance to update. * `` with the namespace where the vCluster instance is deployed. ``` helm upgrade --install vcluster \ --values vcluster.yaml \ --repo https://charts.loft.sh \ --namespace \ --repository-config='' ``` Replace: * `` with the name of the vCluster instance to update. * `` with the namespace where the vCluster instance is deployed. 1. The steps assume that you have the terraform file that you initially deployed your terraform resource from. Generate a new plan with your updated `vcluster.yaml`. ``` terraform plan ``` Verify that the provider can access your cluster and that the proposed changes are correct. 2. Apply your new changes. ``` terraform apply ``` 1) Commit and push your updated `vcluster.yaml` to your configured ArgoCD repository. 2) Synchronize your ArgoCD repository with your configured cluster. Learn more about [Cluster API Provider for vCluster](https://github.com/loft-sh/cluster-api-provider-vcluster). 1. Install the vCluster provider. ``` clusterctl init --infrastructure vcluster:v0.2.0 ``` 2. Export environment variables to be used by the cluster API provider to create an updated manifest. The manifest will be applied to your Kubernetes cluster, which will update your vCluster with the updated configuration options. ``` # Replace and with values of the vCluster that you are updating export CLUSTER_NAME= export CLUSTER_NAMESPACE= # Since the vcluster.yaml has changed, you'll need to re-export it to the updated vcluster.yaml export VCLUSTER_YAML=$(awk '{printf "%s\\n", $0}' vcluster.yaml) ``` Replace: * `` with the name of the vCluster instance to update. * `` with the namespace where the vCluster instance is deployed. 3. Regenerate the manifest and apply the updated manifest. ``` clusterctl generate cluster ${CLUSTER_NAME} \ --infrastructure vcluster \ --target-namespace ${CLUSTER_NAMESPACE} \ | kubectl apply -f - ``` Kubernetes Version The Kubernetes version for the vCluster is not set at the CAPI provider command. It is configured the `vcluster.yaml` file based on your Kubernetes distribution. 4. Wait for vCluster to be updated by watching for the vCluster custom resource to report a `ready` status. ``` kubectl wait --for=condition=ready vcluster -n $CLUSTER_NAMESPACE $CLUSTER_NAME --timeout=300s ``` ## Changes in the vCluster values between 0.19 and 0.20+[​](#changes-in-the-vcluster-values-between-019-and-020 "Direct link to Changes in the vCluster values between 0.19 and 0.20+") The following is a reference for how specific 0.19 fields map to (or have changed in) 0.20+. autoDeletePersistentVolumeClaims **0.19**: ``` autoDeletePersistentVolumeClaims: false ``` **0.20+**: ``` controlPlane: backingStore: etcd: deploy: statefulSet: persistence: volumeClaim: retentionPolicy: Delete ``` fallbackHostDns **0.19**: ``` fallbackHostDns: false ``` **0.20+**: ``` networking: advanced: fallbackHostCluster: true ``` init Section **0.19**: ``` init: helm: [] manifests: '---' manifestsTemplate: "" ``` **0.20+**: ``` experimental: deploy: vcluster: helm: [] manifests: '---' manifestsTemplate: "" ``` isolation.nodeProxyPermission.enabled **0.19**: ``` isolation: nodeProxyPermission: enabled: false ``` **0.20+**: This setting is removed. Instead, replicate the same RBAC permissions using `rbac.clusterRole.extraRules` or `rbac.clusterRole.overwriteRules`: ``` rbac: clusterRole: enabled: true extraRules: - apiGroups: [""] resources: ["nodes/proxy"] verbs: ["get", "watch", "list"] ``` isolation.podSecurityStandard **0.19**: ``` isolation: podSecurityStandard: baseline ``` **0.20+**: ``` policies: podSecurityStandard: Baseline ``` mapServices, monitoring.serviceMonitor, noopSyncer, openshift **0.19**: ``` mapServices: fromHost: [] fromVirtual: [] monitoring: serviceMonitor: enabled: false noopSyncer: enabled: false synck8sService: false openshift: enable: false ``` **0.20+**: mapServices → `networking.replicateServices`: ``` networking: replicateServices: fromHost: [] fromVirtual: [] ``` monitoring.serviceMonitor.enabled → `controlPlane.serviceMonitor.enabled`: ``` controlPlane: serviceMonitor: enabled: false ``` noopSyncer → `experimental.syncSettings`: ``` experimental: syncSettings: disableSync: false rewriteKubernetesService: false ``` openshift.enable → Replace with extra RBAC rules: ``` rbac: role: extraRules: - apiGroups: [""] resources: ["endpoints/restricted"] verbs: ["create"] ``` pro / proLicenseSecret **0.19**: ``` pro: false proLicenseSecret: "" ``` **0.20+**: ``` pro: false # proLicenseSecret: ${namespace}/${name} external: platform: apiKey: secretName: ${name} namespace: ${namespace} ``` proxy, securityContext, and service **0.19**: ``` proxy: metricsServer: nodes: enabled: false pods: enabled: false securityContext: allowPrivilegeEscalation: false runAsGroup: 0 runAsUser: 0 service: externalIPs: [] externalTrafficPolicy: "" loadBalancerAnnotations: {} loadBalancerClass: "" loadBalancerIP: "" loadBalancerSourceRanges: [] type: ClusterIP ``` **0.20+**: proxy.metricsServer → `integrations.metricsServer`: ``` integrations: metricsServer: enabled: true nodes: true pods: true ``` securityContext → `controlPlane.statefulSet.security.containerSecurityContext`: ``` controlPlane: statefulSet: security: containerSecurityContext: allowPrivilegeEscalation: false runAsGroup: 0 runAsUser: 0 ``` service → `controlPlane.service.spec`: ``` controlPlane: service: annotations: {} spec: externalIPs: [] externalTrafficPolicy: "" type: ClusterIP loadBalancerClass: "" loadBalancerIP: "" loadBalancerSourceRanges: [] loadBalancerAnnotations: {} ``` sync.fake-\* **0.19**: ``` sync: fake-nodes: enabled: true fake-persistentvolumes: enabled: true ``` **0.20+**: Disable is an inverse of enable and is acceptable in this context. `fake-nodes.enabled = true` → disable node sync from the host: ``` sync: fromHost: nodes: enabled: false ``` `fake-persistentvolumes.enabled = true` → disable PV sync to the host: ``` sync: toHost: persistentVolumes: enabled: false ``` ### Quick reference table[​](#quick-reference-table "Direct link to Quick reference table") | Old (0.19) | New (0.20+) | | ---------------------------------------- | --------------------------------------------------------------------------------------------------- | | `autoDeletePersistentVolumeClaims` | `controlPlane.backingStore.etcd.deploy.statefulSet.persistence.volumeClaim.retentionPolicy: Delete` | | `fallbackHostDns` | `networking.advanced.fallbackHostCluster` | | `init.helm/manifests/manifestsTemplate` | `experimental.deploy.vcluster.helm/manifests/manifestsTemplate` | | `isolation.nodeProxyPermission.enabled` | Replaced by extra RBAC rules in `rbac.clusterRole.extraRules` | | `isolation.podSecurityStandard` | `policies.podSecurityStandard` | | `mapServices` | `networking.replicateServices` | | `monitoring.serviceMonitor.enabled` | `controlPlane.serviceMonitor.enabled` | | `noopSyncer` | `experimental.syncSettings` | | `openshift.enable` | Add extra rules in `rbac.role.extraRules` | | `proLicenseSecret` | `external.platform.apiKey.secretName` / `namespace` | | `proxy.metricsServer.nodes/pods` | `integrations.metricsServer.enabled/nodes/pods` | | `securityContext` | `controlPlane.statefulSet.security.containerSecurityContext` | | `service.*` | `controlPlane.service.spec` | | `sync.fake-nodes/fake-persistentvolumes` | `sync.fromHost.nodes.enabled = false` / `sync.toHost.persistentVolumes.enabled = false` | ## Compare Helm values between versions[​](#compare-helm-values-between-versions "Direct link to Compare Helm values between versions") compare version values You can easily compare values files between versions to see the changes. You can spin up two clusters, one at older version, such as 0.19 and one at newer version, for example 0.20+. Retrieve their computed `values.yaml` files and compare them with a simple `diff`: ``` VERSION_OLD="0.19.7" VERSION_NEW="0.20.0" NAMESPACE_OLD="vcluster-old" NAMESPACE_NEW="vcluster-new" RELEASE_OLD="vcluster-${VERSION_OLD//./-}" RELEASE_NEW="vcluster-${VERSION_NEW//./-}" helm upgrade --install $RELEASE_OLD vcluster \ --repo https://charts.loft.sh \ --namespace $NAMESPACE_OLD \ --repository-config='' \ --version $VERSION_OLD \ --create-namespace helm upgrade --install $RELEASE_NEW vcluster \ --repo https://charts.loft.sh \ --namespace $NAMESPACE_NEW \ --repository-config='' \ --version $VERSION_NEW \ --create-namespace helm get values $RELEASE_OLD -n $NAMESPACE_OLD --all > ${VERSION_OLD}.yaml helm get values $RELEASE_NEW -n $NAMESPACE_NEW --all > ${VERSION_NEW}.yaml diff -u ${VERSION_OLD}.yaml ${VERSION_NEW}.yaml > v${VERSION_OLD}-to-v${VERSION_NEW}.diff ``` ## Related content[​](#related-content "Direct link to Related content") * [vCluster config reference](https://vcluster.com/docs/vcluster/configure/vcluster-yaml) for full configuration details. --- # Loft CLI to vCluster CLI Migration This section provides an overview of `loft` CLI commands that have migrated to the `vcluster` CLI. Most commands behave as they did previously. Changes are noted in the "Comments" column. ## Global --config flag[​](#global---config-flag "Direct link to Global --config flag") The `loft` CLI previously read its config file from `~/.loft/config.json` if you did not specify a `--config` flag. The `vcluster` CLI now defaults to `~/.vcluster/config.json`. The `vcluster` CLI also introduces a new config structure. Configuration related to the platform is now nested under the `platform` key. ``` CONFIG_PATH=/your/path/to/.loft/config.json jq -n --rawfile loft_config $CONFIG_PATH '{platform: ($loft_config | fromjson)}' -c > /tmp/config.json # Do this only if you want to use the new vCluster CLI default location mkdir -p ~/.vcluster CONFIG_PATH=~/.vcluster/config.json mv /tmp/config.json $CONFIG_PATH ``` Specify the path with `--config $CONFIG_PATH`, or allow `vcluster` CLI to default to `~/.vcluster/config.json`. ## Mappings of old `loft` CLI commands to `vcluster` CLI commands[​](#mappings-of-old-loft-cli-commands-to-vcluster-cli-commands "Direct link to mappings-of-old-loft-cli-commands-to-vcluster-cli-commands") ### vCluster commands[​](#vcluster-commands "Direct link to vCluster commands") | loft CLI | vCluster CLI | Alias | Comments | | ---------------------- | ------------------------------------ | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `loft create vcluster` | `vcluster create --driver=platform` | `vcluster platform create vcluster` | Following flags have been **removed**:
`--delete-after`
`--disable-direct-cluster-endpoint`
`--sleep-after`

Following flags have been **changed**:
`--version` was renamed to `--template-version` | | `loft delete vcluster` | `vcluster delete --driver=platform` | `vcluster platform delete vcluster` | Following flags have been **removed**:
`--cluster` (The platform automatically handles this in the background.)
`--namespace` (The platform automatically handles this in the background.)
`--delete-namespace` (Use `--delete-namespace`) | | `loft import vcluster` | `vcluster platform add vcluster` | | | | `loft list vclusters` | `vcluster list --driver=platform` | `vcluster platform list vclusters` | | | `loft share vcluster` | `vcluster platform share vcluster` | | | | `loft sleep vcluster` | `vcluster sleep --driver=platform` | `vcluster platform sleep vcluster`
`vcluster pause --driver=platform` | | | `loft use vcluster` | `vcluster connect --driver=platform` | `vcluster platform connect vcluster` | Following flags have been **removed**:
`--cluster` (The platform automatically handles this in the background.)
`--namespace` (The platform automatically handles this in the background.)
`--skip-wait`
`--disable-direct-cluster-endpoint`

To obtain the kube config for the given tenant cluster issue the following:
`vcluster connect myvcluster --print > /tmp/myvcluser-kube-config.yaml` | | `loft wakeup vcluster` | `vcluster wakeup --driver=platform` | `vcluster platform wakeup vcluster`
`vcluster resume --driver=platform` | | ### Namespace commands[​](#namespace-commands "Direct link to Namespace commands") | loft CLI | vCluster CLI | Comments | | ------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------- | | `loft create space` | `vcluster platform create namespace` | Following flags have been **changed**:
`--version` was renamed to `--template-version` | | `loft delete space` | `vcluster platform delete namespace` | | | `loft import space` | - | **Removed** as it was unused | | `loft list spaces` | `vcluster platform list namespaces` | | | `loft share space` | `vcluster platform share namespace` | | | `loft sleep space` | `vcluster platform sleep namespace` | | | `loft use space` | `vcluster platform connect namespace` | | | `loft wakeup space` | `vcluster platform wakeup namespace` | | ### Cluster commands[​](#cluster-commands "Direct link to Cluster commands") | loft CLI | vCluster CLI | Comments | | ---------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `loft connect cluster` | `vcluster platform add cluster` | Default agent namespace has changed to the `vcluster-platform`. To achieve same behavior as before, use `--namespace=loft` flag. | | `loft list clusters` | `vcluster platform list clusters` | | | `loft use cluster` | `vcluster platform connect cluster` | | | `loft vars cluster` | `vcluster platform get current-cluster -o name` | | ### Token, secret, and password commands[​](#token-secret-and-password-commands "Direct link to Token, secret, and password commands") | loft CLI | vCluster CLI | | ----------------------------- | ------------------------------------------ | | `loft get cluster-access-key` | `vcluster platform get cluster-access-key` | | `loft get secret` | `vcluster platform get secret` | | `loft set secret` | `vcluster platform set secret` | | `loft list secrets` | `vcluster platform list secrets` | | `loft reset password` | `vcluster platform reset password` | | `loft token` | `vcluster platform access-key` | ### User and team commands[​](#user-and-team-commands "Direct link to User and team commands") | loft CLI | vCluster CLI | | -------------------- | -------------------------------------------- | | `loft get user` | `vcluster platform get current-user` | | `loft list teams` | `vcluster platform list teams` | | `loft vars username` | `vcluster platform get current-user -o name` | ### Platform and management commands[​](#platform-and-management-commands "Direct link to Platform and management commands") | loft CLI | vCluster CLI | | --------------------- | -------------------------------------- | | `loft start` | `vcluster platform start` | | `loft use management` | `vcluster platform connect management` | ### Miscellaneous commands[​](#miscellaneous-commands "Direct link to Miscellaneous commands") | loft CLI | vCluster CLI | | -------------- | -------------------------- | | `loft backup` | `vcluster platform backup` | | `loft credits` | `vcluster credits` | ### Removed commands[​](#removed-commands "Direct link to Removed commands") | loft CLI | vCluster CLI | Comments | | --------------------------------- | ------------ | -------------------------------- | | `loft defaults get` | - | Removed as it is no longer used. | | `loft defaults set` | - | Removed as it is no longer used. | | `loft defaults view` | - | Removed as it is no longer used. | | `loft generate admin-kube-config` | - | Removed as it is no longer used. | | `loft completion` | - | Removed as it is no longer used. | | `loft ui` | - | Removed as it is no longer used. | --- # Search vCluster docs with Algolia MCP AI coding assistants that support the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) can query the vCluster documentation directly from your editor or chat interface. This uses Algolia's [hosted MCP server](https://www.algolia.com/doc/guides/model-context-protocol/public-mcp), which connects to the same search index that powers the vCluster docs site search. ## Available tools[​](#available-tools "Direct link to Available tools") The MCP server exposes two tools: * **algolia\_search\_vcluster** - Full-text search across vCluster documentation. * **algolia\_search\_for\_facet\_values** - Filter results by facets such as version or section. Access is read-only. No data is modified through this integration. ## Configure your AI tool[​](#configure-your-ai-tool "Direct link to Configure your AI tool") Add the vCluster docs MCP server to your AI assistant of choice. * Claude Code * Claude.ai * Cursor * VS Code Add to your project's `.mcp.json` or global `~/.claude.json`: ``` { "mcpServers": { "vcluster-docs": { "url": "https://mcp.eu.algolia.com/1/VR12oimCRMyH3BKzGU2jVw/mcp" } } } ``` 1. Go to **Settings** > **Integrations** > **Add MCP Server**. 2. Enter the server URL: ``` https://mcp.eu.algolia.com/1/VR12oimCRMyH3BKzGU2jVw/mcp ``` Add to `~/.cursor/mcp.json`: ``` { "mcpServers": { "vcluster-docs": { "url": "https://mcp.eu.algolia.com/1/VR12oimCRMyH3BKzGU2jVw/mcp" } } } ``` Add to `~/.vscode/mcp.json`: ``` { "servers": { "vcluster-docs": { "type": "http", "url": "https://mcp.eu.algolia.com/1/VR12oimCRMyH3BKzGU2jVw/mcp" } } } ``` No API keys or authentication tokens are needed. The server URL is the only configuration required. ## Usage tips[​](#usage-tips "Direct link to Usage tips") Once configured, your AI assistant can search the vCluster docs when answering questions. Most MCP-aware tools call the search automatically based on your prompt. You can also reference the tool directly, for example by asking your assistant to "search the vCluster docs for sleep mode configuration." To search within a specific docs version, ask your assistant to filter by version facet. For example: "Search vCluster docs for pod syncing in version 0.22." --- # Contribute to vCluster Docs This guide is for anyone who wants to improve the vCluster and vCluster Platform docs, and it's written for contributors outside vCluster Labs. Everything it describes uses public resources: this repository, the published docs site, and the public [loft-sh/vcluster](https://github.com/loft-sh/vcluster) and [loft-sh/api](https://github.com/loft-sh/api) repositories. A few steps, such as adding backport labels or requesting an AI review, need a maintainer. Ask for them in your PR description. vCluster Labs staff have an internal guide with more detail on tickets, branches, and private previews. This docs website is built using [Docusaurus](https://docusaurus.io/) v3, a modern static website generator. ## Getting started[​](#getting-started "Direct link to Getting started") Fork the [vCluster docs repository](https://github.com/loft-sh/vcluster-docs) and clone your fork locally ``` git clone https://github.com//vcluster-docs ``` then run the following command to install dependencies: ``` npm install ``` This repository requires Node.js 22 or later. The `.nvmrc` file pins the version that CI uses. ### Local development[​](#local-development "Direct link to Local development") Run the following command to start a local development server: ``` npm run start ``` This command opens `http://localhost:3000/docs` in your default browser. Most changes are reflected live without having to restart the server. Note that this command doesn't always catch build errors. ### Build[​](#build "Direct link to Build") After finishing your changes, run the following command to test production build locally: ``` npm run build ``` This command generates static content into the `build` directory and can be served using any static contents hosting service. You can run the following command to serve the built website locally: ``` npm run serve ``` Before making a pull request, it's recommended to run this command to fix any broken links that may have been introduced. A build doesn't catch every kind of broken link, so also see [Checking links before you push](#checking-links-before-you-push). ### Preview a stacked PR[​](#preview-a-stacked-pr "Direct link to Preview a stacked PR") Netlify builds a deploy preview only for PRs whose base branch is `main` or another branch Netlify deploys, such as `next`. A stacked PR, one that targets another PR's branch, gets no preview from Netlify. The `stacked-pr-preview` workflow fills the gap. For a PR whose base isn't `main`, `next`, or a `vcluster-v*` or `platform-v*` branch, it builds the site and deploys a Netlify draft at the alias `pr-`. It then posts the URL, `https://pr---vcluster-docs-site.netlify.app/docs/`, as a PR comment and updates both on every push. PRs from forks get no preview, because GitHub doesn't pass repository secrets to them. A native GitHub stack created with `gh stack` on top of `main` already gets Netlify Deploy Previews, so the workflow skips it. #### Deploy a preview by hand[​](#deploy-a-preview-by-hand "Direct link to Deploy a preview by hand") Use the manual steps when the workflow can't run, for example on a PR from a fork. If you have access to the docs project in Netlify, you can deploy a draft preview yourself with the Netlify CLI. Otherwise, ask someone who has access. The build needs about 11 GB of memory. ``` npx netlify-cli login npx netlify-cli link npx netlify-cli deploy --context deploy-preview --dir=public --alias=pr- ``` `netlify link` prompts you to choose the project. The `deploy` command runs the site build first, so you don't need a separate `netlify build`. The deploy is a draft, so it doesn't change the live site, and the CLI prints its URL when it finishes. Rerun the `deploy` command to update it after you push changes. Never use `main`, `next`, or another deployed branch name as the alias. Netlify serves an alias at the same subdomain as a branch deploy with that name. Omit `--prod`, which publishes to the live site. As with any preview, put `/next/` after the product path to see unreleased changes, for example `/docs/platform/next/...`. ## Where your change goes[​](#where-your-change-goes "Direct link to Where your change goes") Start by finding the page you want to change. Most changes update a page that already exists. Search the docs site, or search this repository for a config key, flag, or resource name. If a feature extends something that's already documented, update that page instead of adding a new one. The site is built from three folders, each with its own sidebar: | Folder | Covers | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `vcluster/` | The cluster vCluster creates: its control plane, worker nodes, `vcluster.yaml`, resource sync, and the `vcluster` CLI. | | `platform/` | vCluster Platform: tenants and projects, templates, users and permissions, apps, fleet operations, Platform install and administration, and the API. vMetal, the machine layer, lives in `platform/vmetal/`. | | `docs/` | Pages that span products, such as the services a provider can offer and end-to-end guides. | Within each folder, sections are organized by what the reader is doing, not by where the code lives. These are the placement questions that come up most: * **vCluster: Deploy, Configure, or Operate?** Deploy (`vcluster/deploy/`) is for creating a cluster or attaching its nodes. Configure (`vcluster/configure/`) is for changing a `vcluster.yaml` setting to turn on or tune a capability. Operate (`vcluster/operate/`) is for actions on a cluster that's already running, such as upgrades and backups. * **Platform: Deliver services to tenants, or Use the platform?** If an admin sets something up for others to consume, such as a template or a permission, it goes in `platform/tenants/`. If a project user works with what they were given, such as creating a cluster from a template, it goes in `platform/use-platform/`. * **vMetal: machine model or driver?** Behavior that's the same for every provisioning driver goes in `platform/vmetal/concepts/`. Anything specific to one driver, such as Metal3 or KubeVirt, goes with that driver in `platform/vmetal/drivers/`. * **A feature that spans products.** Document each part where its owner lives, and link between them, instead of writing the same procedure twice. If you're not sure where a page belongs, say so in the PR description. A maintainer will place it. ### Moving or renaming a page[​](#moving-or-renaming-a-page "Direct link to Moving or renaming a page") A move needs more than a `git mv`. Links into the page, imports, and redirects all have to follow it. The scripts in `hack/move-docs/` move the files, rewrite the links, and write the redirects. Their README lists what they don't catch and the checks to run before you push. Keep each move in its own PR. ### Generated pages[​](#generated-pages "Direct link to Generated pages") Some reference pages are generated from source on each release, so an edit to them is overwritten. Change the source instead, or describe the problem in an issue or PR. | Content | Folder | Generated from | | ----------------------------- | ---------------------------- | ---------------------------------------------------------------------------------------- | | `vcluster.yaml` key reference | `vcluster/_partials/config/` | The schema field descriptions in [loft-sh/vcluster](https://github.com/loft-sh/vcluster) | | vCluster CLI reference | `vcluster/cli/vcluster_*.md` | The command help text in [loft-sh/vcluster](https://github.com/loft-sh/vcluster) | | Platform API reference | `platform/api/` | The Go doc comments in [loft-sh/api](https://github.com/loft-sh/api) | To add prose next to a generated field, see [Adding prose to generated partials](#adding-prose-to-generated-partials). ## Versions and branches[​](#versions-and-branches "Direct link to Versions and branches") ### Versioned and unversioned folders[​](#versioned-and-unversioned-folders "Direct link to Versioned and unversioned folders") | Folder | What it is | Published at | | ---------------------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | `vcluster/`, `platform/` | The docs for the next, unreleased version. Make your changes here. | `/docs/vcluster/next/...` and `/docs/platform/next/...` | | `vcluster_versioned_docs/version-X.Y.Z/` | A frozen copy of each released vCluster version. The newest is the default. | `/docs/vcluster/...` for the newest, `/docs/vcluster/X.Y.Z/...` for older ones | | `platform_versioned_docs/version-X.Y.Z/` | A frozen copy of each released Platform version. | `/docs/platform/...` for the newest, `/docs/platform/X.Y.Z/...` for older ones | | `docs/` | The root docs. Not versioned. | `/docs/...` | At each release, the unversioned folders are copied into a new version folder. Don't edit the versioned folders directly. Change the page in `vcluster/` or `platform/`, and use backport labels to carry the fix into released versions. ### Which branch to target[​](#which-branch-to-target "Direct link to Which branch to target") Open your PR against `main`. Everything merged to `main` appears on the public site right away, under the unreleased `/next/` URLs. The repository also has a `next` branch. It isn't the same as the `/next/` URLs. Maintainers use the `next` branch only for major releases, to prepare docs that must not appear on the public site before the release, such as a large reorganization. Don't target it unless a maintainer asks you to. ### Backport changes to released versions[​](#backport-changes-to-released-versions "Direct link to Backport changes to released versions") To carry a fix into released versions, add a `backport-vX.XX` label to the PR, one per version. After the PR merges, a workflow copies each changed file under `vcluster/` or `platform/` to the same path in that version's folder and opens a backport PR. > \[!WARNING] The copy only works when the page has the same path in the older version. Platform 5.0 and vCluster 0.38 reorganized most of the docs, so a page's current path often doesn't exist in Platform 4.12 or earlier, or vCluster 0.37 or earlier. For those pages, don't add a backport label. Make the change by hand in each older version's folder, at that version's own path, and say so in the PR description. Pages that kept their path still backport normally. This applies until Platform 4.12 and vCluster 0.37 reach end of support. Until Platform 5.0 and vCluster 0.38 are released, the unversioned folders on `main` still use the Platform 4.12 and vCluster 0.37 layout, so backport labels on PRs into `main` work normally. vCluster labels use the minor version, such as `backport-v0.37`. Platform labels include the patch version, such as `backport-v4.12.0`. Only maintainers can add labels, so if you're contributing from outside vCluster Labs, list the versions the fix applies to in your PR description. ## After you open a PR[​](#after-you-open-a-pr "Direct link to After you open a PR") * **Checks.** If you opened the PR from a fork, GitHub waits for a maintainer to approve the workflow runs before any checks start. The checks include the Lint check (see [Style guide automation](#style-guide-automation)), link validators, glossary and feature table checks, and a redirect check when a page moves or is deleted. * **Preview.** Netlify builds a deploy preview and comments with its link. Your changes to `vcluster/` or `platform/` appear in the unreleased version, so add `/next/` after `/docs/vcluster` or `/docs/platform` in the preview URL. * **Review.** A maintainer reviews the PR and may suggest edits for style, placement, or accuracy. ## AI-assisted PR review[​](#ai-assisted-pr-review "Direct link to AI-assisted PR review") Maintainers can request an AI review from Claude by mentioning `@claude` in a PR comment. Only members of the `loft-sh` organization can trigger it, so if you're contributing from outside vCluster Labs, ask a maintainer. Example commands: * `@claude review this PR` - get a focused review * `@claude fact check this page` - check the technical claims against the code Claude reads the PR and posts review comments. It doesn't push changes. The review checks documentation style, validates vCluster YAML configurations, and identifies broken links. The AI review is meant to assist, not replace, human review. ## SEO and crawler policy[​](#seo-and-crawler-policy "Direct link to SEO and crawler policy") The docs site exposes multiple versions (current stable, prior supported, EOS/EOL, and the `next` preview). Only the current stable version should compete for search engine and AI crawler attention; older versions remain reachable for users who need them, but must not outrank or dilute the current docs. See DOC-1325 for background. ### What the current setup does[​](#what-the-current-setup-does "Direct link to What the current setup does") * **`lastVersion`** in `docusaurus.config.js` makes unversioned URLs (for example `/docs/vcluster/deploy/...`) serve the current stable release. Those unversioned URLs are the canonical, indexable entry points. * **noindex on non-current versions** is emitted by `src/theme/DocItem/Layout/index.js`. The swizzle reads `useActivePluginAndVersion()` and injects `` whenever `isLast` is false. That covers EOS, EOL, any prior stable version still in `onlyIncludeVersions`, and the `current` (next/main) preview. * **Sitemap** (`docusaurus.config.js`, sitemap plugin) lists only unversioned URLs. Any path matching `/vcluster//` or `/platform//` — plus the `next` previews — is stripped by both `ignorePatterns` and a post-filter in `createSitemapItems`. * **`llms.txt`** already excludes versioned docs via `includeVersionedDocs: false` on the `docusaurus-plugin-llms-txt` plugin. That stays as-is. * **`static/robots.txt`** is intentionally permissive. `Disallow` would block crawlers from fetching the page and therefore from ever seeing the `noindex` meta tag, which is the opposite of what we want. Note that the production host `vcluster.com/robots.txt` is served by the main site repo, not this one; this file applies to the Netlify preview domain. * **Canonical tags** are not emitted automatically. Only add a `` when the content at a versioned URL is effectively identical to the current version. Do not point version-specific content (release notes, deprecated APIs, migration guides) at the current docs. ### When adding or removing a version[​](#when-adding-or-removing-a-version "Direct link to When adding or removing a version") 1. Update `onlyIncludeVersions` in `docusaurus.config.js` for the affected docs plugin (vcluster or platform). No SEO code changes are required — the swizzle derives `isLast` from the plugin config. 2. When bumping `lastVersion`, the previous stable version automatically starts emitting `noindex,follow`. No per-version list to edit. 3. Regenerate the lifecycle JSON (`npm run generate-lifecycle-json`) so the supported-versions partial stays in sync with the config. ## Style guide[​](#style-guide "Direct link to Style guide") The full style guide is [style-guide.md](https://github.com/loft-sh/vcluster-docs/blob/main/.claude/skills/vcluster-docs-writer/references/style-guide.md), with product names and terminology in [vcluster-terms.md](https://github.com/loft-sh/vcluster-docs/blob/main/.claude/skills/vcluster-docs-writer/references/vcluster-terms.md). Both are part of the docs-writer skill in this repository, so AI tools that work in the repository follow the same rules. For a new feature, task, or troubleshooting page, start from the matching [page template](https://github.com/loft-sh/vcluster-docs/blob/main/.claude/skills/vcluster-docs-writer/references/templates/README.md). The sections below summarize the rules contributors run into most. The Vale linter enforces most of the rules. See the [style guide automation](#style-guide-automation) section for more information. The vCluster docs use the following style guides in this order: 1. [Google developer documentation style guide](https://developers.google.com/style) 2. Kubernetes [Documentation Style Guide](https://kubernetes.io/docs/contribute/style/style-guide/) for Kubernetes terms. Page titles follow **sentence** case. Capitalize Kubernetes objects [according to the K8s style guide](https://kubernetes.io/docs/contribute/style/style-guide/#documentation-formatting-standards). ### Core principles[​](#core-principles "Direct link to Core principles") Here are some core principles to keep in mind when writing documentation: * Active voice creates stronger documentation and helps with translations - write "deploy the configuration" instead of "the configuration was deployed". * Using sentence case in headings makes content more approachable - write "Getting started with providers" instead of "Getting Started with Providers". * Keep sentences under 25 words to improve readability and SEO performance - break down complex ideas into digestible chunks. * Write in present tense - say "this command installs" rather than "this command will install". Docs refer to actions that happen in the present as users read them. * Use contractions (don't, isn't, can't) for better clarity - "don't" stands out more than "do not" when expressing negatives. * Avoid Latin phrases (e.g., i.e., etc.) - choose plain English alternatives that all readers can understand easily. * Never use terms like "easy," "simple," or "obvious" as they can undermine reader confidence and create unnecessary pressure. * Create descriptive link text - instead of "click here," explain where the link leads, such as "view the installation guide". * Spell out acronyms on first use unless they're universally known in technical contexts. ### Consistency and document flow[​](#consistency-and-document-flow "Direct link to Consistency and document flow") When creating new documentation, review similar existing documents to maintain consistent flow and structure. For example, see how platform deployment documentation follows a standard pattern in the [Quick Start Guide](https://www.vcluster.com/docs/platform/install/quick-start-guide). This helps readers navigate documentation intuitively since they know what to expect. Documents of the same kind should use consistent header levels and structure. All installation guides should use the same header hierarchy and naming conventions to help users navigate between different installation options. Docusaurus admonitions should be used consistently throughout the documentation to highlight important information. ``` :::note[Additional context: The `vcluster create` command automatically creates] a new namespace if it doesn't exist. ::: :::tip[Use `vcluster` CLI to quickly deploy a cluster. :::] :::info[The default configuration uses minimal resources suitable for testing.] ::: :::warning[Ensure your Kubernetes cluster has sufficient available memory before] deploying vCluster. ::: :::danger[Do not delete the vCluster pod. :::] ``` ### Oxford comma[​](#oxford-comma "Direct link to Oxford comma") Use the Oxford comma (serial comma) before the last item when listing 3 or more items in a sentence. * **Do Not**: vCluster requires a, b and c. * **Do**: vCluster requires a, b, and c. See Scribbr's [article](https://www.scribbr.com/commas/oxford-comma/) on the Oxford comma for a detailed explanation and examples. ### Version numbers[​](#version-numbers "Direct link to Version numbers") Version numbers for vCluster Labs products carry a lowercase `v` prefix in prose. That covers vCluster, vCluster Platform, vNode, vMetal, and the vCluster CLI. * **Do Not**: Upgrade to vCluster 0.38 before you install Platform 5.0. * **Do**: Upgrade to vCluster v0.38 before you install Platform v5.0. Third-party versions, such as Kubernetes 1.34 or etcd 3.5, follow their upstream style. Code, commands, flags, YAML values, URLs, and paths don't change, so Helm's `--version 0.38.0` stays as it is. The `Loft.version-prefix` Vale rule flags a product name or "version" followed by an unprefixed number. See "Version Numbers" in the [style guide](https://github.com/loft-sh/vcluster-docs/blob/main/.claude/skills/vcluster-docs-writer/references/style-guide.md) for the full rule. ### Code blocks[​](#code-blocks "Direct link to Code blocks") Use `<>` to indicate placeholders in code blocks. For example: ``` kubectl get pods ``` #### Dynamic version tokens[​](#dynamic-version-tokens "Direct link to Dynamic version tokens") Instead of hardcoding version numbers that go stale, use these tokens: | Token | Renders as | | ---------------------------- | ------------------------------------------------ | | `__PLATFORM_VERSION__` | Latest stable Platform patch release, as `X.Y.Z` | | `__VCLUSTER_VERSION__` | Latest stable vCluster patch release, as `X.Y.Z` | | `__PLATFORM_VERSION_MINOR__` | Minor version of that Platform release, as `X.Y` | | `__VCLUSTER_VERSION_MINOR__` | Minor version of that vCluster release, as `X.Y` | ``` export PLATFORM_VERSION=__PLATFORM_VERSION__ helm install vcluster --version __VCLUSTER_VERSION__ ``` The values come from `src/data/latest-versions.json`, which a daily workflow (`sync-latest-versions.yml`) keeps at the latest stable patch of each product's `lastVersion`. Don't edit that file by hand. A remark plugin replaces tokens at build time in text, inline code, code blocks, and string JSX attributes, and `InterpolatedCodeBlock` replaces them in its `code` prop. Both read the same file, so they agree. In versioned docs, tokens resolve to that folder's version instead. A page in `platform_versioned_docs/version-4.11.0/` renders `__PLATFORM_VERSION__` as `4.11.0`, not the latest 4.11 patch. Use tokens only for "the current version". For a fixed minimum, such as "requires vCluster v0.34 or later", write the number. A token would render the current version on every page and in every versioned copy. Tokens render without the `v` prefix, because most of them sit in code. When one appears in prose, write `v__VCLUSTER_VERSION__`. #### Interactive code blocks[​](#interactive-code-blocks "Direct link to Interactive code blocks") For code blocks that contain values users need to customize, use the `InterpolatedCodeBlock` component instead of regular code blocks. This allows users to edit values directly in the documentation. **Using local variables** (user can edit each instance): ``` import InterpolatedCodeBlock from '@site/src/components/InterpolatedCodeBlock'; ``` **Using global variables** (define once, use everywhere on the page): ``` import PageVariables from '@site/src/components/PageVariables'; import InterpolatedCodeBlock from '@site/src/components/InterpolatedCodeBlock'; ``` This eliminates repeated export statements across multiple code blocks. Use `PageVariables` when the same values (like version numbers or registry URLs) appear in multiple code examples on a page. Notes: * Place `PageVariables` before any code blocks that use those variables * Can be placed anywhere on the page (doesn't have to be at the top) * Multiple `PageVariables` components will merge together **Adding titles to code blocks** (optional): The `InterpolatedCodeBlock` component supports an optional `title` attribute to display a descriptive title above the code block: ``` ``` The `title` attribute is completely optional and backwards compatible. Existing code blocks without a title will continue to work as before. #### Icon component[​](#icon-component "Direct link to Icon component") Use the `Icon` component to display checkmarks, crosses or warnings for feature availability, comparison tables, or inline indicators. **Usage:** ``` import Icon from '@site/src/components/Icon'; | Feature | Supported | |---------|-----------| | Connection pooling | | | SSL encryption | | | Legacy auth | | ``` **Inline usage:** ``` Connection pooling is available in all tiers. ``` **With custom tooltips:** ``` ``` **Icon types:** * `type="check"` or `type="checkmark"` - Orange checkmark (light mode), green (dark mode) * `type="cross"` or `type="x"` - Gray X symbol * `type="warning"` or `type="!"` - Yellow ! symbol The component automatically adapts to dark/light theme changes. #### Formatting and variables[​](#formatting-and-variables "Direct link to Formatting and variables") To make code blocks easier to work with, consider adding variables and use [here docs](https://en.wikipedia.org/wiki/Here_document) to make it easier to copy and paste code snippets. Using a variable in a code block: ``` $POD_NAME = pod-name kubectl get pods $POD_NAME ``` Creating a file using a here doc: ``` cat < vcluster.yaml controlPlane: # Deploy etcd instead of using the embedded SQLite backingStore: etcd: deploy: enabled: true statefulSet: highAvailability: replicas: 3 # Deploy vCluster with 3 replicas statefulSet: highAvailability: replicas: 3 EOF ``` Applying a Kubernetes manifest using docs: ``` cat << EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: example-pod labels: app: nginx spec: containers: - name: nginx image: nginx:1.21 ports: - containerPort: 80 EOF ``` > \[!NOTE] here docs support is available in most shells, including bash, zsh, and fish, however in shells other than bash, the syntax may vary. #### Highlight lines of code[​](#highlight-lines-of-code "Direct link to Highlight lines of code") Use [inline comments](https://docusaurus.io/docs/markdown-features/code-blocks#highlighting-with-comments) in the code to highlight lines. ### Links and imports[​](#links-and-imports "Direct link to Links and imports") Link to another page in the same docs section with a relative path to the source file, including the `.mdx` suffix. The suffix is what tells Docusaurus to rewrite the link at build time. Without it, the raw string survives into the compiled bundle and the browser resolves it against the current page URL when clicked, which produces a different path. The server-rendered HTML still looks correct, so this kind of break is invisible to `curl`, `git grep`, and static inspection. It only shows up on click. * Correct: `[Projects](../understand/what-are-projects.mdx)` * Correct, for a directory target: `[Networking](../networking/README.mdx)` * Broken: `[Projects](../understand/what-are-projects)` * Broken: `[Networking](../networking/)` Relative paths also work on GitHub, survive slug changes, and track file moves, so prefer them over `/docs/` URL paths inside a section. Link across sections, such as from Platform to vCluster, with a `/docs/` absolute path. The two sections are separate Docusaurus plugin instances, so relative file paths don't resolve between them. * `[vCluster docs](/docs/vcluster)` * `[Sleep mode](/docs/vcluster/configure/vcluster-yaml/sleep-mode)` #### Links inside partials and fragments[​](#links-inside-partials-and-fragments "Direct link to Links inside partials and fragments") Calculate the path from the partial or fragment file itself, not from a page that imports it. Reusable content gets imported from pages at different depths, so a path that works from one importer breaks from another. Markdown link syntax isn't parsed inside a JSX expression. When reusable content needs to link from within JSX, use the `VersionAwareDocLink` component so the link keeps the reader's docs version. ``` import VersionAwareDocLink from '@site/src/components/VersionAwareDocLink'; security baseline ``` A cross-product link targets the other product's stable docs, because Platform and vCluster version independently. #### Partials and fragments[​](#partials-and-fragments "Direct link to Partials and fragments") The two folder names mean different things: | Folder | Contains | Edit it? | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | `_partials/` | Generated content: the vcluster.yaml reference (`vcluster/_partials/config/`), the Platform API reference (`platform/api/_partials/resources/`), and the Platform CLI index | No. A generator overwrites it. Change the source, as described in [Generated pages](#generated-pages) | | `_fragments/` | Hand-written content that more than one page imports, such as admonitions, shared procedures, and prerequisites | Yes | Put a new reusable snippet in `_fragments/`, in the same product tree as the pages that import it, or in `docs/_fragments/` for root and global content. Don't add a hand-written file to `_partials/`. Many existing hand-written snippets still live in `_partials/` from before this rule. Import them where they are, and don't use them as a model for new files. The "Check partials" CI check enforces this. It fails a PR that adds a new file under `_partials/` outside the generated paths. Editing or moving an existing file passes. #### Import paths[​](#import-paths "Direct link to Import paths") Choose the import path based on what you're importing and whether that content should freeze with a docs version. | Importing | Use | Why | | ----------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | | React or theme components | `@site/src/` or `@theme/` | Not version-specific | | Same-product partials and fragments | a relative path | Versioning copies the docs tree, so each snapshot stays self-contained | | Intentionally global content | `@site/docs/_fragments/` for new content, `@site/docs/_partials/` for existing files | Updates every docs version at once | | Cross-product content | `@site//` | Tracks the other product's current content, which is the only option because the version lines don't match up | A same-product `@site/` import keeps resolving to the live tree even from a versioned page, so an archived page ends up showing current content. CI rejects those. #### Checking links before you push[​](#checking-links-before-you-push "Direct link to Checking links before you push") A production build catches links that point at nothing. It doesn't catch a link that resolves at build time but breaks on click, so run the link checks too. ``` npm run build # links to missing pages npm run validate-mdx-links # click-time breaks in live docs npm run validate-reusable-imports # same-product live imports ``` ### vCluster terms[​](#vcluster-terms "Direct link to vCluster terms") vCluster Labs is the company. Do not use "Loft", "LoftLabs", or "Loft Platform" to refer to vCluster products. "vCluster" is a trademark. There are strict legal frameworks around how to use a trademark, for example, it cannot be used in plural. **Do not use "vClusters"**. Never use vCluster or vClusters when talking about a cluster that vCluster creates. Use **clusters**. Both "virtual cluster" and "tenant cluster" are retired: the Tenant primitive claims the word "tenant" for the customer organization, so the cluster gives it back. ### The hierarchy[​](#the-hierarchy "Direct link to The hierarchy") 1. A **system admin** installs Platform onto a Kubernetes cluster, which becomes the **control plane cluster**. 2. A **platform admin** creates **Tenants**. 3. A **tenant admin** divides the Tenant boundary across **Projects**. 4. A **project user** creates a **cluster**. A **Tenant** is a customer organization, never a cluster. Capitalize it only when naming the API resource, the same way "Project" is. **Tenant isolation** is the term for separation at every layer, from the Tenant boundary in the management plane down to dedicated nodes. Don't coin "cluster isolation", and don't convert existing "tenant isolation" prose. When a sentence needs to say *how* separation happens, name the mechanism: private nodes, vNode, API-level and namespace separation. "Multi-Tenancy" stays only as the name of the licensed feature, on the Platform License page and in `loft-sh/plans`. As a descriptor, "multi-tenancy" and "multitenancy" are retired. Describe what the product does instead. ### Products[​](#products "Direct link to Products") * vCluster: open source project that provisions and manages clusters * vCluster Platform: the management platform and UI for managing clusters and Tenants across one or more control plane clusters * vMetal: infrastructure orchestrator for bare-metal and VM provisioning beneath clusters * vNode: tenant-isolation container runtime "vCluster Pro" is not a product and should not be used. License-gated features are "Enterprise" features, unlocked via a vCluster Platform Enterprise plan (Dev/Prod/Scale), not a separate "Pro" product or tier. The `vcluster-pro` container image, the `loft-sh/vcluster-pro` repository name, and the `sidebar_class_name: pro` frontmatter value are code/engineering identifiers and are unaffected by this rule — that sidebar class renders as an "ENTERPRISE" badge. Name vCluster Platform once per page, then call it "the platform". Keep the formal name on first mention and in install, licensing, support, release, and API contexts. Call the interface "the platform UI", not "vCluster Platform UI". The `Loft.no-vcluster-platform` Vale rule warns when a page repeats the formal name. Use `vcluster.yaml` for all cluster configuration. `values.yaml` only means the Helm values file for installing vCluster Platform. ### CLI[​](#cli "Direct link to CLI") The command line interface name is `vcluster`. ### Abbreviations for Kubernetes distros[​](#abbreviations-for-kubernetes-distros "Direct link to Abbreviations for Kubernetes distros") * [Lightweight Kubernetes](https://k3s.io/): K3s * [Kubernetes](https://kubernetes.io/): K8s * [Zero Friction Kubernetes](https://k0sproject.io/): k0s Note that k0s is the only Kubernetes distro to use a lower case 'k' * [AWS Elastic Kubernetes Service](https://aws.amazon.com/eks/): EKS ### Other product terms[​](#other-product-terms "Direct link to Other product terms") * AWS [EKS](https://aws.amazon.com/eks/) * [CoreDNS](https://coredns.io/) * [etcd](https://etcd.io/) ### Positioning shared nodes[​](#positioning-shared-nodes "Direct link to Positioning shared nodes") Shared nodes are a supported tenancy boundary for trusted tenants, not a security boundary for untrusted tenants with Kubernetes access or arbitrary workload execution. Frame the boundary by trust and tenant access, not by internal versus external. * Recommend shared nodes for trusted tenants such as development, testing, CI/CD, and internal engineering teams. * Route external, resale, regulated, or otherwise untrusted tenant offerings to private nodes, optionally with vNode for runtime isolation. * Don't soften this to "strongly not recommended". Shared-node mitigations such as NetworkPolicy don't replace data-plane isolation for untrusted tenants. * Recommend NetworkPolicy as an added isolation layer for all tenants, including trusted ones. vCluster can create it through `policies.networkPolicy`, and the CNI enforces it. * Frame the risk as an architectural property (shared kernel and nodes) plus a configuration responsibility, not as a vCluster defect. Don't reference specific customers or security incidents. * Reuse the `vcluster/_partials/admonitions/shared-nodes-suitability.mdx` partial rather than rewriting the caveat. There is one exception. A provider serving its own trusted models behind an API (no tenant Kubernetes access) is application-level multitenancy and can serve external customers. See the inference provider production guide. See the shared nodes positioning section in `CLAUDE.md` for the full guidance. ## Style guide automation[​](#style-guide-automation "Direct link to Style guide automation") To maintain quality and consistency in our technical documentation, we use [vale](https://vale.sh/) as a linter to automatically enforce style guidelines. > \[!NOTE] The Lint check runs Vale on pull requests into `main` that change `.md` or `.mdx` files under `platform/`, `vcluster/`, and the versioned docs folders. It posts its findings as review comments on your PR. Only Vale errors on lines your PR adds fail the check. Warnings appear as comments but don't block the merge. Root `docs/` pages and the generated CLI pages aren't checked. > > Fix what Vale flags in the prose. Don't change file paths, commands, or config keys to satisfy it. ### What is Vale?[​](#what-is-vale "Direct link to What is Vale?") [Vale](https://vale.sh/) is an open-source, command-line linter that helps you enforce style and grammar rules in written documentation. It’s highly configurable, allowing you to define custom rules that suit your project’s needs. By integrating it into our CI pipeline, contributors can receive real-time feedback on their documentation during pull requests. ### Running Vale locally[​](#running-vale-locally "Direct link to Running Vale locally") Vale offers multiple installation methods suitable for different operating systems: * **macOS**: `brew install vale` * **Windows**: `choco install vale` * **Linux**: `brew install vale` or use another [package manager](https://repology.org/project/vale/versions) * **Docker**: `docker pull jdkato/vale` Individual files or folders can be linted * To check all files (this may take some time): `vale .` * To check a specific file: `vale path/to/file.mdx` * To check a specific folder including subfolders: `vale /path/to/folder` Running `vale` locally allows us to check our documentation before submitting a pull request. This helps catch style issues early and streamlines the review process. ### Plugins[​](#plugins "Direct link to Plugins") VSCode and Neovim have `vale` plugins that can be installed to lint files as you write them. * VS Code [Vale plugin](https://github.com/errata-ai/vale-vscode). * Neovim setup: * Install [mason.nvim](https://github.com/williamboman/mason.nvim) and add vale\_ls LSP. Configure and use like any other LSP. * Install `vale_ls` on start with `lazy.nvim` ``` return { "williamboman/mason.nvim", optional = true, opts = function(_, opts) if type(opts.ensure_installed) == "table" then vim.list_extend(opts.ensure_installed, { "vale_ls" }) end end, } ``` Run `vale` from the repository root so it picks up the repository's `.vale.ini`. CI uses Vale 3.13.0, so a newer local version can report slightly differently. > \[!NOTE] The repository's `.vale.ini` sets `MinAlertLevel = warning`, so Vale hides suggestions. To see them too, run `vale --minAlertLevel=suggestion `. ### Adding terms to the vocabulary[​](#adding-terms-to-the-vocabulary "Direct link to Adding terms to the vocabulary") If Vale flags a legitimate product or technical term, add it to `.github/styles/config/vocabularies/Loft/accept.txt` in the same PR. Each line is a regular expression. Terms that Vale should always flag go in `reject.txt` in the same folder. Don't add a word only to silence one finding. ### Controlling Vale rules[​](#controlling-vale-rules "Direct link to Controlling Vale rules") Use HTML comments to turn Vale off around a block, in both `.md` and `.mdx` files. The repository's `.vale.ini` treats `.mdx` files as Markdown, so JSX comments such as `{/* vale off */}` don't suppress Vale. ``` This content won't be checked by Vale. ``` ### Suppressing the release drift check[​](#suppressing-the-release-drift-check "Direct link to Suppressing the release drift check") On every vCluster release, `hack/cli-drift` and `hack/config-drift` compare prose code blocks against the regenerated CLI reference and `vcluster.yaml` schema partials, and drift findings open an automated fix PR. Some pages show outdated config or removed commands on purpose: migration guides, before/after comparisons, changelog examples. Opt such a block out by placing a `drift-ignore` comment on the line directly above the opening fence: ```` {/* drift-ignore */} ```yaml # deliberately shows the pre-0.24 syntax exportKubeConfig: secret: name: my-secret ``` ```` In `.md` files use `` instead. The marker applies only to the single fenced block below it. Both scanners honor it, for YAML blocks and for CLI examples alike. Do not use it to silence a finding on current-state docs; fix the drift instead. `InterpolatedCodeBlock` components are scanned the same way as fences: `[[VAR:...]]` placeholders are resolved to their defaults before validation, and the marker on the line directly above the component tag opts it out. A `yaml` block or component titled `vcluster.yaml` is always validated, even when its top-level keys no longer match any current schema root (that is what a block looks like after a release removes the field it documents). ## Adding prose to generated partials[​](#adding-prose-to-generated-partials "Direct link to Adding prose to generated partials") The vCluster config partials under `vcluster/_partials/config/` are overwritten on every release by `hack/vcluster/partials/main.go`. Editing those files in place was the original workflow and the original failure mode: the next regeneration silently dropped the edit. Two routes exist now, both designed to survive regeneration. ### Use the Extras map when prose belongs with a schema field[​](#use-the-extras-map-when-prose-belongs-with-a-schema-field "Direct link to Use the Extras map when prose belongs with a schema field") If the prose is intrinsically tied to one schema path — a security warning that only makes sense alongside `controlPlane.standalone.joinNode`, a migration tip that belongs at the bottom of `controlPlane.distro` — add a `pathExtras` entry in `hack/vcluster/partials/main.go`: ``` var pathExtras = map[string]Extras{ "controlPlane/standalone/joinNode": { Before: "\n:::warning Security consideration\n...\n:::\n\n", }, } ``` `Before` lands above the rendered schema content, `After` below. Links in this prose follow the same rules as any other partial, so use a relative path with the `.mdx` suffix, calculated from where the generated file lands. That depth is the same for live and versioned output. The map keys are validated against the live schema before any output is written; a rename or removal upstream panics the generator with the full list of dangling keys, so the prose can be migrated explicitly instead of silently orphaned. ### Use a fragment when prose stands alone[​](#use-a-fragment-when-prose-stands-alone "Direct link to Use a fragment when prose stands alone") If the prose is conceptual (an overview page, a how-to, a comparison) and only happens to live near generated content, write it as a fragment under `vcluster/_fragments/` and import it where needed. The generator doesn't modify it. This is the right choice when: * The prose can be read on its own without a specific schema field next to it. * The prose references multiple schema paths. * The prose has its own headings, examples, and lifecycle (changes for reasons unrelated to schema bumps). ### Quick rule of thumb[​](#quick-rule-of-thumb "Direct link to Quick rule of thumb") * One schema path with one or two paragraphs of prose → `pathExtras`. * Multi-paragraph prose, multiple references, own structure → a fragment, imported manually. --- # Secure vCluster Deployments Plan before you deploy Several deployment decisions cannot be changed after initial deployment, including rootless mode, worker node type, backing store, and control plane distro. Review the [deployment overview](https://vcluster.com/docs/vcluster/deploy.md) and this page before deploying production tenant clusters. vCluster security depends on where a workload runs, which resources vCluster synchronizes, and who controls the configuration. A secure deployment starts by separating three boundaries: * **Inside the tenant cluster** * Kubernetes RBAC * Admission control * Resource limits * Tenant-defined network policies * **Between the tenant cluster and the control plane cluster** * vCluster sync configuration * **Outside the tenant cluster** * Platform RBAC and approved templates * Control plane pod hardening * Private nodes, vNode, and vMetal ## Baseline security configuration[​](#baseline-security-configuration "Direct link to Baseline security configuration") A hardened baseline combines four controls: * A [Pod Security Standard](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/pod-security-standard.md) applied to tenant namespaces blocks privileged containers, direct host path mounts, and other high-risk pod configurations inside the tenant cluster. It evaluates the Pod spec, not what a PersistentVolumeClaim's volume actually points at. On shared nodes, pair it with the storage guidance in the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#storage-breakout-prevention). * A [ResourceQuota](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/resource-quota.md) and [LimitRange](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/limit-range.md) in the control plane cluster namespace cap what a tenant workload consumes there. This prevents a runaway or malicious tenant from exhausting shared-node capacity. They don't cap what a tenant stores in its own tenant cluster backing store. See [shared-node hardening](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#shared-node-baseline-configuration) for that gap. * [NetworkPolicies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md) in the control plane cluster namespace restrict traffic between vCluster components and other workloads on the shared cluster. `policies.networkPolicy` defaults to `false` and requires CNI support to take effect. * Storing synced pod [service account tokens](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/pods.md) in Secrets rather than pod annotations scopes token access to Secret-level RBAC. Anyone who can read pod specs can read tokens stored in annotations. Enable all four together: vcluster.yaml ``` policies: podSecurityStandard: baseline resourceQuota: enabled: true limitRange: enabled: true networkPolicy: enabled: true sync: toHost: pods: useSecretsForSATokens: true ``` NetworkPolicy support NetworkPolicies only take effect when the control plane cluster's CNI implements Kubernetes NetworkPolicy. Creating NetworkPolicy resources without a controller that enforces them does not isolate traffic. ## Workload security inside the tenant cluster[​](#workload-security-inside-the-tenant-cluster "Direct link to Workload security inside the tenant cluster") ### Kubernetes RBAC[​](#kubernetes-rbac "Direct link to Kubernetes RBAC") Kubernetes RBAC inside the tenant cluster controls what users and service accounts can do in the tenant environment. This is separate from Platform RBAC, which controls who can create or manage tenant clusters. Key access to restrict at the tenant level: * **Pod and workload creation**: Any user who can create a Pod can run arbitrary container images and influence sync behavior. Limit `create` on Pods, Deployments, StatefulSets, and Jobs to subjects that need it. * **`exec` and `attach`**: `pods/exec` and `pods/attach` grant interactive shell access to running containers. Grant these verbs only to users who genuinely need them. * **Secret access**: Restrict `get`, `list`, and `watch` on Secrets to minimize exposure of credentials and service account tokens. * **Cluster-scoped resources**: ClusterRoles, ClusterRoleBindings, and CRDs affect the entire tenant cluster. Prefer namespace-scoped Roles and RoleBindings for tenant users over cluster-scoped grants. When Platform templates define tenant cluster configuration, review the initial RBAC grants created at cluster instantiation time. Tenants with `cluster-admin` inside the tenant cluster can still be constrained at the sync and Platform governance layers. ### NetworkPolicy[​](#networkpolicy "Direct link to NetworkPolicy") vCluster-managed NetworkPolicies (created by `policies.networkPolicy`) protect traffic in the control plane cluster namespace where vCluster itself runs. Tenant users can also create NetworkPolicies inside the tenant cluster for application-to-application traffic. On shared nodes, those tenant policies are enforced only when `sync.toHost.networkPolicies` is enabled, which syncs them to the control plane cluster CNI. With private nodes, tenant workloads run on dedicated nodes, so their CNI enforces tenant policies directly. Every layer requires CNI support for NetworkPolicy. Use namespace-level NetworkPolicies inside the tenant cluster to restrict lateral movement between applications: * Apply a default-deny ingress policy in namespaces that do not need cross-namespace traffic. * Restrict egress from sensitive workloads to the services they actually call. ### Admission control[​](#admission-control "Direct link to Admission control") Use `policies.podSecurityStandard` when the platform operator needs to enforce a Pod Security Standard uniformly across all tenant namespaces. Tenants cannot remove or loosen PSA labels that vCluster applies. For granular, platform-managed admission policies that tenants cannot override, use [central admission control](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/admission-control.md). Central admission control is an Enterprise feature. It configures webhooks in the tenant cluster. The webhooks point through the vCluster proxy to a service in the control plane cluster or to an external policy service. Tenants can install their own admission controllers, but they cannot mutate or delete the centrally configured hooks. Test admission webhooks with `failurePolicy: Ignore` before switching to `failurePolicy: Fail` in production. A misconfigured webhook with `failurePolicy: Fail` can block all workload creation in the tenant cluster. ## Control plane cluster boundary[​](#control-plane-cluster-boundary "Direct link to Control plane cluster boundary") vCluster synchronizes selected resources between the tenant cluster and the control plane cluster. Review sync settings before allowing non-admin users to create or modify vCluster configurations. ### Sync to host[​](#sync-to-host "Direct link to Sync to host") vCluster creates [`sync.toHost`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host.md) resources in the control plane cluster after translating names, labels, and namespaces. Normal workloads such as Pods, Services, ConfigMaps, Secrets, and PersistentVolumeClaims require this. It also means tenant-created objects can create real Kubernetes objects in the control plane cluster. Limit `sync.toHost` to the resources tenants actually need. Treat advanced options such as namespace sync, custom resource sync, service account sync, and host deployments as security-sensitive. ### Sync from host[​](#sync-from-host "Direct link to Sync from host") vCluster copies [`sync.fromHost`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host.md) resources from the control plane cluster into the tenant cluster. Tenants cannot modify these resources or write changes back to the control plane cluster. Use selectors and explicit mappings when syncing shared or sensitive resources such as StorageClasses, RuntimeClasses, Secrets, ConfigMaps, and custom resources. ### vCluster RBAC[​](#vcluster-rbac "Direct link to vCluster RBAC") vCluster automatically generates the control plane cluster RBAC it needs based on enabled features and synchronized resources. This governs what the vCluster syncer can do in the control plane cluster, not what tenant users can do inside the tenant cluster. For restricted environments, use [RBAC configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/rbac.md) to disable automatic RBAC generation, use an administrator-managed ServiceAccount, or overwrite the generated rules. ## Harden the vCluster control plane[​](#harden-the-vcluster-control-plane "Direct link to Harden the vCluster control plane") The vCluster control plane runs as a workload in the control plane cluster. Harden it like any other privileged control plane component: * Apply a [Pod Security Standard label](https://kubernetes.io/docs/concepts/security/pod-security-admission/#pod-security-levels) to the control plane cluster namespace where vCluster runs. This is independent of `policies.podSecurityStandard`, which labels namespaces inside the tenant cluster — the host namespace label is a backstop on the control plane cluster side. `baseline` enforcement works with the default vCluster deployment. `restricted` enforcement requires [rootless mode](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/rootless-mode.md) to be enabled first. * Run the vCluster pod as a non-root user with [rootless mode](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/rootless-mode.md). * Configure [`controlPlane.statefulSet.security.podSecurityContext` and `controlPlane.statefulSet.security.containerSecurityContext`](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/host-nodes/policies.md) for user IDs, file system groups, privilege escalation, capabilities, and seccomp settings. * Configure the [control plane ServiceAccount](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/rbac.md) explicitly when your organization manages RBAC outside of the vCluster Helm chart. * Use the [CIS hardening guide](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/overview.md) for API server audit logging, EventRateLimit, encryption at rest, admission plugin settings, and hardened Kubernetes component arguments. * Use [FIPS images](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/fips.md), [air-gapped deployment](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/air-gapped.md), or [corporate proxy configuration](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/corporate-proxy.md) when required by your environment. ## Platform governance[​](#platform-governance "Direct link to Platform governance") When users create tenant clusters through vCluster Platform, templates are the main security boundary for vCluster configuration. * Keep templates required for projects where non-admin users create tenant clusters. * Put approved security settings directly into [VirtualClusterTemplates](https://vcluster.com/docs/platform/administer/templates/create-templates.md). * Limit which templates a project can use with [allowed templates](https://vcluster.com/docs/platform/administer/projects/allowed/templates.md). * Limit which control plane clusters a project can use with allowed clusters. * Use [Platform RBAC](https://vcluster.com/docs/platform/administer/users-permissions/overview.md) to separate platform administrators from project users. * Use [Least Privilege Mode](https://vcluster.com/docs/platform/configure/agent-settings/least-privilege-mode.md) for Platform agents on external control plane clusters when your organization needs reduced agent permissions. * Configure [agent security contexts](https://vcluster.com/docs/platform/configure/agent-settings/security-context.md) for connected control plane clusters. Do not allow untrusted project users to create tenant clusters without an approved template. A user with full control over `vcluster.yaml` can loosen sync settings, expose control plane cluster resources, or deploy resources directly to the control plane cluster. ## Tenancy and isolation profiles[​](#tenancy-and-isolation-profiles "Direct link to Tenancy and isolation profiles") These controls apply to the baseline profile, where tenant workloads run on shared nodes of the control plane cluster. Shared nodes are a supported model for trusted tenants, not a security boundary for untrusted tenants with Kubernetes access or arbitrary workload execution. See the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) for the full threat model, responsibility matrix, and baseline configuration before deploying trusted tenants on shared nodes. Two additional profiles move the isolation boundary further out. ### vNode[​](#vnode "Direct link to vNode") vCluster defines the tenant cluster boundary. vNode strengthens workload isolation on shared nodes using a runtime sandbox. Use vNode when tenants share physical nodes but workloads need stronger runtime containment than standard containers provide. This is especially relevant for privileged workloads, Docker-in-Docker, `hostPID`, host path access, GPU workloads, or user-supplied model inference code. vNode does not replace vCluster policy guardrails. Continue to use Pod Security Standards, NetworkPolicies, quotas, reviewed sync settings, and approved templates. See the [vNode docs](https://www.vnode.com/docs/) for configuration details. ### Private nodes[​](#private-nodes "Direct link to Private nodes") Private nodes move tenant workloads off the control plane cluster's shared worker nodes. The vCluster control plane still runs in the control plane cluster, but workload execution moves to nodes dedicated to the tenant cluster. Use private nodes when tenants require dedicated node capacity, stronger workload isolation, or direct control over node-level Kubernetes behavior. See [Private Nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for deployment details. For GPU or PCIe device exclusivity, bare metal compliance, or sovereign deployments, private nodes can be provisioned as dedicated physical servers using vMetal. vMetal adds bare metal operational security requirements outside the scope of vCluster itself, including BMC credential handling, OS image integrity, PXE and provisioning network exposure, Metal3 and Ironic hardening, and BareMetalHost access control. See the [vMetal docs](https://www.vmetal.ai/docs/) for bare metal-specific controls. ## Production checklist[​](#production-checklist "Direct link to Production checklist") Before deploying production tenant clusters, verify: * [ ] The control plane cluster CNI enforces NetworkPolicy. * [ ] `policies.podSecurityStandard`, `policies.resourceQuota`, `policies.limitRange`, and `policies.networkPolicy` are configured intentionally. * [ ] `sync.toHost` and `sync.fromHost` are limited to required resources. * [ ] Tenant RBAC restricts Pod creation, `exec`, Secret access, and cluster-scoped role grants to intended subjects. * [ ] Central admission control is tested with `failurePolicy: Ignore` before using `failurePolicy: Fail`. * [ ] vCluster control plane pods run with the required security context, rootless mode, and ServiceAccount model. * [ ] API server audit logging, EventRateLimit, and encryption settings meet compliance requirements. * [ ] Non-admin Platform users can create tenant clusters only from approved templates. * [ ] Platform Agent permissions and security contexts meet the control plane cluster's trust requirements. * [ ] The isolation profile is explicit: shared nodes, shared nodes with vNode, private nodes, or private nodes with vMetal. * [ ] On shared nodes, PersistentVolume sync, PriorityClass sync, secondary networks (Multus), LoadBalancer address pools, KubeVirt, node placement, and real-node disclosure are reviewed against the [shared-node hardening guide](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md). * [ ] Host-side admission control covers translated or synced objects and controller-generated workloads that tenant-facing admission never sees. * [ ] vMetal deployments validate BMC credential storage and rotation, OS image checksums, provisioning network segmentation, BareMetalHost and NodeProvider RBAC, and control plane access to BMC networks. --- # Shared-node admission policy examples Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** These policies apply through your own policy engine, not vCluster, so the YAML itself doesn't depend on a vCluster edition. The first one's documented delivery path does. It needs [`policies.centralAdmission`](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#prerequisites), an Enterprise feature, for the tamper-resistant delivery this guide assumes. [Manually mapping a host policy service into the tenant cluster](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/admission-control.md), or installing a webhook directly inside it, reaches the tenant cluster too, without that dependency. You then have to protect that setup from tenant modification yourself. Where they run also differs. The first evaluates Pods inside each tenant cluster, reached through that centralAdmission webhook wiring, before vCluster ever syncs the Pod to the control plane cluster. The other two evaluate objects natively on the control plane cluster, after translation. Kyverno registers native admission webhooks for the control plane cluster automatically, for any installed `ClusterPolicy` that matches a resource kind. That happens regardless of which cluster you intend the policy to serve. Running this policy on a separate Kyverno deployment doesn't prevent that instance from also registering its own native webhook and evaluating the control plane cluster's own Pods. Add a precondition instead, requiring the [`request.options.vCluster.name`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/admission-control.md#access-tenant-identifiers-in-webhooks) field vCluster's proxy adds to a tenant-routed request. A control plane cluster's own native Pods never carry that field, so the precondition alone keeps this policy from evaluating anything but tenant-routed admission. `background: false` is still worth setting, since background scans have no admission request to check the precondition against, but the precondition is what actually isolates the rule. This page is a companion to [Shared-node security hardening](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md). It provides Kyverno examples for three admission controls from that guide, the two toleration controls from [Node scheduling and isolation](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#node-scheduling-and-isolation), and the NetworkPolicy peer restrictions from [Primary pod network](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#primary-pod-network). Kyverno version and policy kind These examples need Kyverno 1.13 or later, since they set enforcement through the per-rule `validate.failureAction` field. Earlier versions use `spec.validationFailureAction` instead and reject these manifests during validation. They also use the classic `ClusterPolicy` kind for consistency with the PV example in the main guide. Kyverno's own docs now list `ClusterPolicy` as deprecated in favor of the CEL-based `ValidatingPolicy` kind. `ClusterPolicy` still works, but check your Kyverno version's support window and adapt these examples if you're on the newer kind. Apply each of these through the same [Rollout sequence](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#rollout-sequence) as the rest of the guide: start in `Audit`, confirm no legitimate workload gets flagged, then switch to `Enforce`. ## Deny tolerations for a protected infrastructure taint in the tenant cluster[​](#deny-tolerations-for-a-protected-infrastructure-taint-in-the-tenant-cluster "Direct link to Deny tolerations for a protected infrastructure taint in the tenant cluster") This runs as [tenant-cluster admission](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md#node-scheduling-and-isolation), so it sees the tenant Pod before vCluster's `enforceTolerations` injects the tenant-pool toleration. Kubernetes admission may already have added its standard node-condition tolerations at this point, so a safe policy can't reject every `Exists` operator or empty value indiscriminately. This example protects the infrastructure taint `dedicated=platform-infrastructure:NoSchedule`. One rule catches `Exists` tolerations whose key and effect can match that taint. The other catches `Equal` tolerations that match its complete key, value, and effect. Replace that tuple with each infrastructure taint you protect, using another pair of rules for every tuple: deny-protected-infrastructure-tolerations.yaml ``` apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: deny-protected-infrastructure-tolerations spec: background: false rules: - name: deny-protected-key-exists match: any: - resources: kinds: - Pod preconditions: all: - key: "{{ request.options.vCluster.name || '' }}" operator: NotEquals value: "" validate: failureAction: Audit message: "Toleration can match the protected infrastructure taint dedicated=platform-infrastructure:NoSchedule." foreach: - list: "request.object.spec.tolerations" deny: conditions: all: - key: "{{ contains(['', 'dedicated'], element.key || '') }}" operator: Equals value: true - key: "{{ element.operator || '' }}" operator: Equals value: "Exists" - key: "{{ contains(['', 'NoSchedule'], element.effect || '') }}" operator: Equals value: true - name: deny-protected-key-value-equal match: any: - resources: kinds: - Pod preconditions: all: - key: "{{ request.options.vCluster.name || '' }}" operator: NotEquals value: "" validate: failureAction: Audit message: "Toleration can match the protected infrastructure taint dedicated=platform-infrastructure:NoSchedule." foreach: - list: "request.object.spec.tolerations" deny: conditions: all: - key: "{{ element.key || '' }}" operator: Equals value: "dedicated" - key: "{{ contains(['', 'Equal'], element.operator || '') }}" operator: Equals value: true - key: "{{ element.value || '' }}" operator: Equals value: "platform-infrastructure" - key: "{{ contains(['', 'NoSchedule'], element.effect || '') }}" operator: Equals value: true ``` ## Allowlist exact tolerations on the control plane cluster[​](#allowlist-exact-tolerations-on-the-control-plane-cluster "Direct link to Allowlist exact tolerations on the control plane cluster") This runs as host-side admission on the translated Pod, after vCluster has injected the tenant-pool toleration and Kubernetes has added any automatic tolerations. It can't distinguish a tenant-supplied toleration from an injected one, so it allowlists the exact set instead of denying specific patterns. The `vcluster.loft.sh/managed-by` selector keeps the policy from affecting unrelated control plane cluster Pods. This example allows the tenant-pool toleration, the two default `NoExecute` tolerations, and the `NoSchedule` tolerations Kubernetes can add to DaemonSet Pods. It denies anything else: allow-only-known-tolerations.yaml ``` apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: allow-only-known-tolerations spec: background: true rules: - name: allow-only-known-tolerations match: any: - resources: kinds: - Pod selector: matchExpressions: - key: vcluster.loft.sh/managed-by operator: Exists validate: failureAction: Audit message: "Toleration does not match an allowed entry." foreach: - list: "request.object.spec.tolerations" anyPattern: - key: dedicated operator: Equal value: tenant-workloads effect: NoSchedule - key: node.kubernetes.io/not-ready operator: Exists effect: NoExecute - key: node.kubernetes.io/unreachable operator: Exists effect: NoExecute - key: node.kubernetes.io/disk-pressure operator: Exists effect: NoSchedule - key: node.kubernetes.io/memory-pressure operator: Exists effect: NoSchedule - key: node.kubernetes.io/pid-pressure operator: Exists effect: NoSchedule - key: node.kubernetes.io/unschedulable operator: Exists effect: NoSchedule - key: node.kubernetes.io/network-unavailable operator: Exists effect: NoSchedule ``` Replace `dedicated`/`tenant-workloads` with your own tenant-pool taint key and value. Remove controller-added entries for workload kinds you prohibit, and add any other tolerations your approved controllers or admission plugins inject. Kyverno's pattern matching only asserts on the fields you list, so the node-condition entries match regardless of whatever `tolerationSeconds` value Kubernetes attaches. ## Restrict NetworkPolicy peers on the control plane cluster[​](#restrict-networkpolicy-peers-on-the-control-plane-cluster "Direct link to Restrict NetworkPolicy peers on the control plane cluster") This runs on a vCluster-synced NetworkPolicy. The `vcluster.loft.sh/managed-by` selector keeps it from affecting unrelated control plane cluster policies. It covers two separate bypasses, since a tenant doesn't need `ipBlock` at all to reopen the platform default-deny: * **Disallowed `ipBlock` values.** Denies the resource if any `ipBlock.cidr` in `spec.ingress[].from[]` or `spec.egress[].to[]` isn't an exact string in the allowed list. Peers using `podSelector` or `namespaceSelector` instead of `ipBlock` are skipped, since the per-entry precondition only evaluates peers that carry an `ipBlock.cidr`. * **Unrestricted rules.** Kubernetes treats a missing or empty `from`/`to` as matching every source or destination. vCluster's translator preserves that unchanged, so a bare `ingress: [{}]` or `egress: [{}]` reopens the boundary without ever touching `ipBlock`. Denies any ingress rule with no `from` entries, or egress rule with no `to` entries. The example uses documentation-only addresses that don't route to real services. Replace them with narrow approved destinations. Never allow the control plane cluster's Pod, Service, node, control-plane, storage, management, or metadata CIDRs here, since doing so would reopen the boundary this policy protects: restrict-networkpolicy-peers.yaml ``` apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: restrict-networkpolicy-peers spec: background: true rules: - name: restrict-ipblock-cidrs match: any: - resources: kinds: - NetworkPolicy selector: matchExpressions: - key: vcluster.loft.sh/managed-by operator: Exists validate: failureAction: Audit message: "ipBlock.cidr is not in the allowed CIDR list." foreach: - list: "request.object.spec.ingress[].from[]" preconditions: all: - key: "{{ element.ipBlock.cidr || '' }}" operator: NotEquals value: "" deny: conditions: any: - key: "{{ contains(['203.0.113.10/32', '2001:db8::10/128'], element.ipBlock.cidr) }}" operator: Equals value: false - list: "request.object.spec.egress[].to[]" preconditions: all: - key: "{{ element.ipBlock.cidr || '' }}" operator: NotEquals value: "" deny: conditions: any: - key: "{{ contains(['203.0.113.10/32', '2001:db8::10/128'], element.ipBlock.cidr) }}" operator: Equals value: false - name: deny-unrestricted-peers match: any: - resources: kinds: - NetworkPolicy selector: matchExpressions: - key: vcluster.loft.sh/managed-by operator: Exists validate: failureAction: Audit message: "Rule has no peer selector, which Kubernetes treats as matching every source or destination." foreach: - list: "request.object.spec.ingress" deny: conditions: any: - key: "{{ length(element.from || `[]`) }}" operator: Equals value: 0 - list: "request.object.spec.egress" deny: conditions: any: - key: "{{ length(element.to || `[]`) }}" operator: Equals value: 0 ``` This policy compares CIDR strings exactly. It doesn't treat an allowed supernet as implicitly allowing every subnet within it. List every CIDR form tenants may use, or use a policy engine with tested CIDR-containment functions if you need range containment. ## Related pages[​](#related-pages "Direct link to Related pages") * [Shared-node security hardening](https://vcluster.com/docs/vcluster/security/shared-nodes-hardening.md) for the full threat model, baseline, and checklist these examples support. * [Secure vCluster deployments](https://vcluster.com/docs/vcluster/security/.md) for the general security baseline and control plane hardening. --- # Shared-node security hardening Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** Most controls in this guide are available in open source vCluster. [Central admission control](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/admission-control.md), [KubeVirt integration](https://vcluster.com/docs/vcluster/integrations/kubevirt.md), and hybrid scheduling are Enterprise features, called out where they appear. [VirtualClusterTemplate](https://vcluster.com/docs/platform/administer/templates/create-templates.md) enforcement needs vCluster Platform, but not a paid Enterprise tier. This guide hardens the shared-node model for tenants you've already decided to trust with Kubernetes access. It doesn't replace that decision. If you haven't made it yet, start with [Architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes) and the [production guide](https://vcluster.com/docs/vcluster/production-guide/overview.md) to choose between shared and private nodes. Hardening doesn't change the trust boundary. Shared nodes give each tenant control-plane, API, and namespace isolation, but tenant workloads share the same kernel and physical nodes. That's a good fit for trusted tenants such as development, testing, CI/CD, and internal engineering teams. Even if you apply every control in this guide, shared nodes still aren't a substitute for [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) when tenants are external, resold, regulated, or otherwise untrusted. Application-level tenancy is out of scope A provider that exposes its own models or services behind an API, with no tenant Kubernetes access, isn't in scope here. That's application-level tenancy, not Kubernetes tenant isolation, so it doesn't need private nodes for that reason alone. Ordinary application and infrastructure hardening still applies. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Confirm these before applying this guide: * A control plane cluster CNI that enforces NetworkPolicy. If you plan to sync tenant NetworkPolicies, you also need either AdminNetworkPolicy or an equivalent CNI-specific tiered policy. A control plane cluster admission policy that constrains translated `ipBlock` peers and rejects rules with no peer list works too. * The ability to deploy your own admission control, such as Kyverno or OPA Gatekeeper, on the control plane cluster, outside vCluster. Several controls in this guide depend on it. * The ability to enforce admission inside every tenant cluster, for example through centrally configured admission webhooks locked into the VirtualClusterTemplate with [`policies.centralAdmission`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/admission-control.md), an Enterprise feature. The toleration controls in this guide need this specifically, since only tenant-cluster admission sees a Pod before `enforceTolerations` injection. * [VirtualClusterTemplate](https://vcluster.com/docs/platform/administer/templates/create-templates.md) enforcement through vCluster Platform, so tenants can't loosen the baseline. * vCluster v0.34.0 or later, if you enable `networking.advanced.proxyKubelets` for kubelet metrics. This guide's baseline disables it, so the version floor only applies if you deviate from that baseline. Earlier versions don't scope kubelet-proxy pod-listing and container-log responses to the tenant's own pods. ## Threat model and responsibility matrix[​](#threat-model-and-responsibility-matrix "Direct link to Threat model and responsibility matrix") Shared nodes give each tenant cluster its own API server, namespaces, and RBAC. They don't give each tenant its own kernel, kubelet, node filesystem, scheduler, host network, CNI, CSI, or device plugins. Every tenant workload that reaches the node shares those components with every other tenant on that node. That single fact drives the rest of this guide. A control that only inspects what a tenant submits to the tenant cluster's API server can't see what happens next. Once vCluster translates that object onto the control plane cluster, a different layer has to catch it. Several of the gaps below exist because one control was assumed to cover the whole path, but it only covers a single segment. Controls split across three layers, and no single layer can compensate for a gap in another: | Layer | Who configures it | What it covers | | --------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | Tenant cluster | Tenant users, or the Platform template applied at creation | RBAC, tenant-authored NetworkPolicies, tenant-authored admission objects. Enforced inside the tenant cluster's own API server. | | vCluster sync | Platform operator, through `vcluster.yaml` | Which resource kinds translate to the control plane cluster, and how. Determines what a tenant-cluster object can turn into on shared infrastructure. | | Control plane cluster | Platform operator, outside vCluster entirely | CNI policy enforcement, host-side admission, node labels and taints, kubelet configuration, and kernel and runtime patching. | A control in the tenant cluster layer protects the tenant cluster's own view of itself. It doesn't reach the node. Closing a shared-node gap almost always requires a control in the control plane cluster layer, because that's the only layer positioned to see the translated, host-side object. ## Shared-node baseline configuration[​](#shared-node-baseline-configuration "Direct link to Shared-node baseline configuration") This baseline is what the vCluster sync layer can enforce on its own. Every setting maps to a gap covered later in this guide, not to generic hardening advice. Several of them only close the gap fully when paired with the control plane cluster admission or CNI enforcement described in that section. Start every shared-node tenant cluster here, then adjust deliberately rather than by omission: vcluster.yaml ``` policies: podSecurityStandard: restricted resourceQuota: enabled: true limitRange: enabled: true networkPolicy: enabled: true workload: publicEgress: enabled: false # centralAdmission: configure with your policy engine's actual webhooks, # see the explanation below. There's no standalone enable flag. sync: toHost: namespaces: enabled: false pods: useSecretsForSATokens: true persistentVolumes: enabled: false priorityClasses: enabled: false networkPolicies: enabled: false fromHost: nodes: enabled: false clearImageStatus: true networking: advanced: proxyKubelets: byHostname: false byIP: false integrations: kubeVirt: enabled: false ``` This baseline sets the following foundation: * **`policies.podSecurityStandard: restricted`**. Default: unset vCluster enforces no Pod Security Standard unless you set one. `restricted` requires containers to run as non-root and blocks privilege escalation. It also denies `hostPID`, `hostIPC`, and `hostNetwork`, so a tenant Pod can't directly observe host processes, host IPC, or the node's network namespace. That's a direct answer to the shared-kernel exposure this guide's threat model opens with. `restricted` doesn't reliably stop reads or writes to a mounted host path reached through a PVC. Kubernetes permits PVC volumes under `restricted`, and file permissions, `fsGroup`, and supplemental groups still govern access. The control plane cluster admission described in [Storage breakout prevention](#storage-breakout-prevention) is the actual mitigation. Use `baseline` only where a workload genuinely can't run under `restricted`, and treat that as a documented compatibility tradeoff, not a default. * **`policies.resourceQuota`, `policies.limitRange`**. Default: `auto` Enabling one automatically enables the other. Both apply to the control plane cluster namespace where the tenant's synced objects land, capping CPU, memory, and object counts there. * **Backing store isn't covered.** These quotas don't cap what a tenant stores in the tenant cluster's own [backing store](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/backing-store.md). A tenant creating large numbers of ConfigMaps, Secrets, or custom resources that never sync to the host can still exhaust that backing store's capacity. Constrain that separately with quotas inside the tenant cluster itself and with backing-store-level storage and object-count limits. * **Defaults need review.** Enabling either setting also activates the chart's default values, for example a 10 CPU request cap, a 20-pod limit, and one allowed LoadBalancer Service. Review and set these explicitly for your tenant's actual sizing tier rather than accepting the defaults as a generic security switch. * **`policies.networkPolicy`**. Default: `false`. Creates NetworkPolicies for the vCluster control plane and workload traffic in the control plane cluster namespace. Requires an enforcing CNI, see [Primary pod network](#primary-pod-network). * **`policies.networkPolicy.workload.publicEgress.enabled: false`**. Default: `true` once `networkPolicy` is enabled. Enabling `networkPolicy` alone pulls in a permissive default egress rule allowing `0.0.0.0/0`, excepting only a handful of private ranges. That exception list doesn't include link-local addresses, so cloud metadata endpoints such as `169.254.169.254` stay reachable. Disable public egress by default, and treat enabling it as an explicit, scoped opt-in per tenant cluster, not per workload. `workload.publicEgress` generates one rule covering every workload this vCluster instance syncs. For narrower, per-workload exceptions, manage separate host-side NetworkPolicy objects with tighter selectors instead. * **`policies.centralAdmission`**. No default. [Enterprise feature](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/admission-control.md). There's no enable flag to begin with. Configure `mutatingWebhooks` or `validatingWebhooks` pointing at a reachable policy engine. The webhook plumbing alone enforces nothing, see [Admission and policy coverage](#admission-and-policy-coverage). * **`sync.toHost.namespaces.enabled: false`**. Default: `false`, same as this baseline. Documenting the default as a decision. If you enable namespace syncing, each tenant namespace maps to its own host namespace. `policies.resourceQuota`, `policies.limitRange`, and `policies.networkPolicy` still create exactly one set of objects each, permanently scoped to the vCluster release namespace. None of them extend to a namespace-sync-created mapped namespace. Provision equivalent ResourceQuota, LimitRange, and NetworkPolicy objects in every mapped namespace yourself, or those tenant workloads run with no quota or network enforcement at all. * **`sync.toHost.pods.useSecretsForSATokens: true`**. Default: `false`. Stores synced pod service account tokens in Secrets, scoped by Secret RBAC, rather than in a pod annotation any Pod-reader can see. * **`sync.toHost.persistentVolumes.enabled: false`**. Default: `false`, same as this baseline. This setting doesn't change anything by itself. It's spelled out so leaving PV sync off is a documented decision, not an oversight. See [Storage breakout prevention](#storage-breakout-prevention). Dynamic provisioning through PersistentVolumeClaims doesn't need PV sync. * **`sync.toHost.priorityClasses.enabled: false`**. Default: `false`, same as this baseline. Also just documenting the default as an intentional decision. See [Operational hardening and lifecycle](#operational-hardening-and-lifecycle) for what it protects against if you ever enable it. * **`sync.toHost.networkPolicies.enabled: false`**. Default: `false`, same as this baseline. A tenant's own NetworkPolicy objects exist only inside the tenant cluster and enforce nothing on shared nodes until this syncs them to the control plane cluster CNI. Kubernetes NetworkPolicies are additive, so a tenant's own permissive rule can reopen access a platform-level default-deny was meant to close. See [Primary pod network](#primary-pod-network) before enabling this. * **`sync.fromHost.nodes.enabled: false`**. Default: `false`, same as this baseline. See [node and kubelet disclosure](#operational-hardening-and-lifecycle). If you do enable node sync, always pair it with `clearImageStatus: true`, which defaults to `false`. * **`networking.advanced.proxyKubelets.byHostname` / `byIP: false`**. Default: `true` for both. See [node and kubelet disclosure](#operational-hardening-and-lifecycle). `kubectl logs` and `exec` don't depend on this setting. Disabling it does break `kubectl top node` and kubelet metrics scraping against synced nodes. Validate that need before disabling it in an environment that depends on it. * **`integrations.kubeVirt.enabled: false`**. Default: `false`, same as this baseline. Documenting the default so KubeVirt stays off until a deliberate opt-in. See [KubeVirt](#kubevirt) for the host-side review required before enabling it. Enforce this configuration through required [VirtualClusterTemplates](https://vcluster.com/docs/platform/administer/templates/create-templates.md) so tenants can't loosen it. A tenant with unrestricted `vcluster.yaml` access can undo every setting in this list. ## Rollout sequence[​](#rollout-sequence "Direct link to Rollout sequence") Apply this baseline in stages, not as a single cutover: 1. **Audit first.** Deploy admission policies in `Audit` mode, for example Kyverno's per-rule `failureAction: Audit`, where your policy engine supports it. Standard Kubernetes NetworkPolicy has no built-in audit mode. Staged or audit enforcement for network policy is CNI-specific, for example Calico's staged policies or Cilium's audit mode. Confirm what would be denied before anything is actually denied. 2. **Fix exceptions.** Work through genuine denials with the workload owner, not by loosening the policy. A workload that needs a wildcard toleration or a broad egress rule usually needs a narrower one, not an exemption. 3. **Enforce the policy.** Switch each policy's own enforcement setting, such as Kyverno's `failureAction`, from `Audit` to `Enforce`, once the audit period surfaces no unexpected denials. 4. **Then harden the webhook.** Separately, move the admission webhook's own `failurePolicy` from `Ignore` to `Fail` once you've validated the webhook is reliably reachable. `failurePolicy` governs what happens if the webhook itself is unavailable, a different concern from whether the policy denies a violation. A misconfigured fail-closed webhook can block all workload creation, so validate readiness and rollback behavior before this step. 5. **Re-verify after every change.** A CNI upgrade, a new admission policy version, or a vCluster upgrade can silently change enforcement behavior. Repeat the audit step after any of them. ## Storage breakout prevention[​](#storage-breakout-prevention "Direct link to Storage breakout prevention") [Pod Security Standards](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/pod-security-standard.md) evaluate the Pod spec. `restricted` rejects a Pod that mounts a `hostPath` volume directly. It has no way to inspect what a PersistentVolume's `spec.hostPath` points at. From the Pod's perspective, that volume is a `persistentVolumeClaim`, not a `hostPath` volume. If [`sync.toHost.persistentVolumes`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/storage/persistent-volumes.md) is enabled, a tenant can create an ordinary PersistentVolume with `spec.hostPath` set to any path on the node. They can bind it with a matching PersistentVolumeClaim and mount that PVC in a Pod. Pod Security Standards, at any profile, permit this. The Pod's volume is a PVC, and PVC mounts aren't a Pod Security Standards field. The same PVC can also back a KubeVirt VM disk. * Disable `sync.toHost.persistentVolumes` for tenants that only use dynamic provisioning through StorageClasses. Dynamic provisioning doesn't require PersistentVolume sync at all. * Where PV sync is genuinely required, add control plane cluster admission (for example, a Kyverno `ClusterPolicy`) that denies any PersistentVolume carrying `spec.hostPath`, or restricts it to an explicit safe-path allowlist. This has to run on the control plane cluster, because the synced PV is a host-side object, not a tenant-cluster object your tenant-facing admission ever inspects. * Treat `restricted` Pod Security Standards as defense in depth for this gap, not the primary control. It reduces the privileges available to the container process, but it doesn't constrain filesystem permissions on the mounted path. * Apply the same "can this create host-side capability indirectly" review to any other cluster-scoped or custom resource you sync to the control plane cluster. A resource that looks tenant-scoped from inside the tenant cluster can still resolve to a host-side object once synced. * The same admission should deny `hostPath` volumes targeting device nodes under `/dev`, such as `/dev/nvidia0`. See [GPU and device isolation](#gpu-and-device-isolation). * Disabling PV sync doesn't by itself restrict which host StorageClass a tenant's PersistentVolumeClaim can name. When StorageClass syncing is disabled or its selector is empty, vCluster can pass through a requested class that exists on the control plane cluster. To gate dynamic provisioning, enable [`sync.fromHost.storageClasses`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/storage-classes.md) with a nonempty selector matching only approved classes. vCluster then leaves a PVC that names a missing or nonmatching class unsynced and reports a `SyncWarning`. The selector should cover storage backend, encryption, cost tier, and reclaim policy. A PVC with no `storageClassName` bypasses this selector, so make the control plane cluster's default StorageClass tenant-safe or use admission to require an explicitly allowed class. Host admission can also provide an immediate denial instead of the native gate's unsynced-PVC behavior. A Kyverno `ClusterPolicy` for the PersistentVolume denial looks like this. It needs Kyverno 1.13 or later, since it sets enforcement through the per-rule `validate.failureAction` field. The `vcluster.loft.sh/managed-by` selector scopes it to PersistentVolumes vCluster translated from a tenant cluster. It doesn't affect unrelated hostPath or local PVs that already exist on the control plane cluster for other platform workloads. Start with `Audit`, confirm no legitimate PersistentVolume gets flagged, then switch to `Enforce` per the [Rollout sequence](#rollout-sequence): deny-hostpath-pv.yaml ``` apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: deny-hostpath-persistentvolumes spec: background: true rules: - name: deny-hostpath-pv match: any: - resources: kinds: - PersistentVolume selector: matchExpressions: - key: vcluster.loft.sh/managed-by operator: Exists validate: failureAction: Audit message: "PersistentVolumes with spec.hostPath are not permitted on shared nodes." pattern: spec: X(hostPath): "null" ``` This same field-absence pattern applies directly to the device-node `hostPath` case in [GPU and device isolation](#gpu-and-device-isolation), since it's the same kind of PersistentVolume object. The `ipBlock` allowlist and wildcard-toleration checks elsewhere in this guide need different Kyverno constructs. They check specific field values and traverse a list, rather than test whether one field is present. Write and test those separately. ## Network and endpoint isolation[​](#network-and-endpoint-isolation "Direct link to Network and endpoint isolation") ### Primary pod network[​](#primary-pod-network "Direct link to Primary pod network") NetworkPolicy enforcement depends entirely on the control plane cluster's CNI. Some CNIs accept NetworkPolicy resources without enforcing them, so confirm enforcement directly rather than assuming it from the presence of a CNI that supports the API. Deploy a platform-level default-deny so isolation doesn't depend on a tenant choosing to author their own NetworkPolicy. `policies.networkPolicy` and `sync.toHost.networkPolicies` only enforce policies that exist, not the absence of one. Kubernetes NetworkPolicies are additive. A pod governed by several policies gets the union of everything each one allows, not the narrowest one. A platform default-deny and a tenant's own permissive policy don't combine into a tighter boundary. They combine into whichever is more open. If [`sync.toHost.networkPolicies`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/network-policies.md) is enabled, vCluster's translator keeps a tenant-authored `ipBlock` peer and its ports unchanged. It preserves a rule with no peers at all the same way. Kubernetes treats a missing or empty `from` or `to` as matching every source or destination. A tenant can sync an egress or ingress rule, with or without `ipBlock`, that reopens traffic the platform default-deny was meant to close. A bare `egress: [{}]` rule needs no `ipBlock` field to do this. Label-based peers don't have this problem the same way. vCluster adds a namespace and vcluster marker label to a tenant's `podSelector` and label-based peers. A tenant can only select or target pods in their own translated namespace, never another tenant's. `ipBlock` peers carry no tenant identity for vCluster to constrain, so they pass through as written. A rule with no peers at all has even less for vCluster to scope, since there's no peer object there to begin with. **Only sync tenant NetworkPolicies where the platform boundary itself sits at a precedence a tenant's own NetworkPolicy can't override.** Otherwise, leave `sync.toHost.networkPolicies` disabled, or add control plane cluster admission that denies synced NetworkPolicies with an `ipBlock` peer outside an explicit allowlist, or with an ingress or egress rule that omits its peer list entirely. See [Restrict NetworkPolicy peers on the control plane cluster](https://vcluster.com/docs/vcluster/security/shared-nodes-admission-examples.md#restrict-networkpolicy-peers-on-the-control-plane-cluster) for a worked example. | Control | Qualifies? | Why | | --------------------------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Kubernetes `AdminNetworkPolicy` | Yes | A `Deny` action there takes effect regardless of any NetworkPolicy. | | Kubernetes `BaselineAdminNetworkPolicy` | No | Evaluated after NetworkPolicy, and designed to be overridden by it. | | Calico `GlobalNetworkPolicy` | Yes, if configured for it | Needs a `Deny` action and a lower `order` than tenant policies. | | Cilium `CiliumClusterwideNetworkPolicy` | No, unless using deny rules | Ordinary allow rules are additive with NetworkPolicy, same as `BaselineAdminNetworkPolicy`. Only an explicit `ingressDeny` or `egressDeny` rule takes precedence over any allow rule, regardless of source. | NetworkPolicy governs the primary pod network interface, including a VM's primary interface through its `virt-launcher` Pod. It doesn't cover Multus secondary interfaces or direct underlay network attachments. Host-network Pod traffic is undefined in the Kubernetes NetworkPolicy spec and CNI-dependent. Most CNIs ignore it, but confirm your CNI's actual behavior rather than assuming either way. ### Multus and secondary networks[​](#multus-and-secondary-networks "Direct link to Multus and secondary networks") A tenant workload attached to a Multus `NetworkAttachmentDefinition` bypasses Pod Security Standards, since attachment isn't a Pod Security field. It typically bypasses primary-interface NetworkPolicy enforcement too, because the secondary interface doesn't traverse the policy-enforced pod `veth`. Require the following at the control plane cluster level: * **Multus namespace isolation**, so a tenant namespace can only reference NetworkAttachmentDefinitions in its own namespace. NADs in the `default` namespace remain globally referenceable by design, and `globalNamespaces` can add more exceptions. Review and remove any privileged NAD from every globally reachable namespace. * **Per-tenant NetworkAttachmentDefinition ownership**, provisioned into each tenant's own host namespace rather than shared across tenants. * **Admission allowlisting for the `k8s.v1.cni.cncf.io/networks` Pod annotation and for KubeVirt `spec.networks`**, so a tenant can't name an infrastructure, management, or another tenant's network. * **Isolated VLANs or VPCs per tenant network**, so even an allowed attachment doesn't land tenants on a shared broadcast domain. Infrastructure and management networks must never be tenant-selectable. Management or control-plane endpoints must be unreachable from tenant workload networks regardless of which interface a tenant workload uses to reach them. ### KubeVirt[​](#kubevirt "Direct link to KubeVirt") [KubeVirt integration](https://vcluster.com/docs/vcluster/integrations/kubevirt.md) is an Enterprise feature. With it enabled, host-side KubeVirt creates the `virt-launcher` pod that actually runs the VM. Tenant-cluster Pod Security Standards never see the VM's tenant-side custom resource at all, since Pod Security Standards only evaluate Pods. This needs two separate host-side admission policies, since a policy that only targets Pods never sees the VM's original network or device fields: * Admission on the host-side VirtualMachine or VirtualMachineInstance object, which evaluates permitted networks and devices. Those fields live on the VM spec, not the generated Pod. * Admission or Pod Security Standards on the generated `virt-launcher` Pod, which covers ordinary Pod-level security context, separate from network and device selection. The VM's primary network interface is generally covered by NetworkPolicy like any other pod, since `virt-launcher` is a real Pod carrying the VMI's labels. Confirm your CNI enforces it the same way it does for other tenant pods. ### LoadBalancer address pools[​](#loadbalancer-address-pools "Direct link to LoadBalancer address pools") Scope LoadBalancer address pools (for example, MetalLB `IPAddressPool`) with namespace or Service selectors, or gate pool and `loadBalancerIP` selection with admission policy. Without scoping, a tenant Service can claim an address from any pool available on the cluster, including a management-network or public pool. Make public IP exposure an explicit, opt-in decision paired with the network default-deny above, not an automatic default from an `autoAssign` pool. ### Ingress and Gateway API hostname claims[​](#ingress-and-gateway-api-hostname-claims "Direct link to Ingress and Gateway API hostname claims") vCluster syncs a tenant's [Ingress](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/ingresses.md) `spec.rules[].host` unchanged. Nothing scopes a tenant's hostname claim to their own tenant cluster. A tenant can claim a hostname that belongs to another tenant or to platform infrastructure. If multiple tenant clusters share the control plane cluster's ingress controller, that controller decides which tenant's rule wins. That's the controller's own conflict-resolution behavior, not a Kubernetes-enforced boundary. A dedicated per-tenant IngressClass alone doesn't isolate tenants, since vCluster preserves a tenant's `spec.ingressClassName` unchanged and a tenant can still name another tenant's class. Isolate tenants one of two ways: * **Namespace-scoped controller.** Pair a per-tenant IngressClass with a controller instance that only watches that tenant's own host namespace. This closes the gap regardless of which class a tenant names. * **Enforced class selection.** Enable [`sync.fromHost.ingressClasses`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/ingress-classes.md) with a selector, and pair it with control plane cluster admission. The selector alone only helps partially. It skips the check when `spec.ingressClassName` is empty, and it doesn't inspect the legacy `kubernetes.io/ingress.class` annotation. Admission needs to cover both of those paths, plus any classless Ingress that would fall through to a default class. If tenants must share one controller, add control plane cluster admission that restricts which hostnames a given tenant namespace's Ingress can claim. Confirm your specific ingress controller's own behavior on cross-namespace hostname collisions rather than assuming isolation. [Gateway API sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md) is the safer default for new workloads. vCluster validates a tenant's HTTPRoute or TLSRoute against the imported Gateway's `allowedRoutes` hostname overrides before sync, when the platform operator configures one. Configure that allowlist. Enabling Gateway API sync alone doesn't create it. Tenant-created Gateways synced through `sync.toHost.gatewayApi.gateways` reintroduce the same hostname claim problem, since the tenant then owns the listener hostname directly. ## Node scheduling and isolation[​](#node-scheduling-and-isolation "Direct link to Node scheduling and isolation") vCluster's [node selector](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md) restricts which control plane cluster nodes a tenant cluster can schedule onto, but a selector is a placement mechanism, not a security boundary. It only isolates tenants from each other if the label it selects on is exclusive to tenant workload nodes. * Use a label and taint dedicated to tenant workloads, applied only to nodes meant to run tenant compute. Inject the exact tenant-pool toleration during sync with [`sync.toHost.pods.enforceTolerations`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/pods.md), so tenants don't need to include it themselves. * Don't reuse a general-purpose label (for example, a generic `worker` label) that's also present on ingress, metrics, storage, VPN, or other infrastructure nodes. A tenant that can target that label can schedule onto those nodes too. * Taint sensitive and infrastructure nodes separately from the tenant pool. A taint alone doesn't reject a tenant workload if the tenant can add a matching toleration, and vCluster preserves tenant-set tolerations by default. * Keep control plane, infrastructure, and tenant-workload node roles on distinct labels and taints, so a change to one role's scheduling doesn't silently open access to another. Blocking those tolerations has to run at two different layers, because vCluster injects the tenant-pool toleration during sync, after the tenant-authored Pod has already passed tenant-cluster admission: * **Tenant-cluster admission** sees the tenant-authored Pod before that injection happens. Deny any tenant-authored toleration capable of matching a protected infrastructure taint, without ever seeing the toleration `enforceTolerations` adds later. Match the full Kubernetes toleration semantics, not just an exact key comparison: * An empty key with `operator: Exists` matches every taint key. * `operator: Exists` alone matches every value for its key. * An empty effect matches every effect for its key. See [Deny tolerations for a protected infrastructure taint in the tenant cluster](https://vcluster.com/docs/vcluster/security/shared-nodes-admission-examples.md#deny-tolerations-for-a-protected-infrastructure-taint-in-the-tenant-cluster) for a worked example. * **Host-side admission** sees the translated Pod after injection, so it can't distinguish a tenant-supplied toleration from an injected one. Permit the exact tenant-pool toleration and the Kubernetes node-condition tolerations your workloads require, including the automatically added `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable` tolerations. Deny every other toleration capable of matching a protected infrastructure taint, including wildcard key, value, or effect combinations. See [Allowlist exact tolerations on the control plane cluster](https://vcluster.com/docs/vcluster/security/shared-nodes-admission-examples.md#allowlist-exact-tolerations-on-the-control-plane-cluster) for a worked example. vCluster also preserves a tenant Pod's requested `spec.runtimeClassName` unchanged by default. Unless you constrain it, a tenant picks its own container runtime handler. That changes the isolation model, resource overhead, and scheduling behavior for that Pod. Constrain it one of two ways: * **Force a handler.** Set [`sync.toHost.pods.runtimeClassName`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/pods.md) to apply the same RuntimeClass to every tenant Pod. * **Gate by selector.** Enable [`sync.fromHost.runtimeClasses`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/runtime-classes.md) with a nonempty selector. This doesn't rewrite or deny the request at admission. It blocks the Pod from syncing to the host at all when the requested class doesn't match. That shows up as a stuck tenant Pod with a `SyncWarning` event, not a clean rejection. ## GPU and device isolation[​](#gpu-and-device-isolation "Direct link to GPU and device isolation") vCluster syncs GPU and other extended device resource requests like any other Pod field. It adds no device-aware scoping beyond what device-plugin or DRA allocation already provides on the control plane cluster. With [`sync.fromHost.deviceClasses`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/device-classes.md), a selector controls which host `DeviceClass` resources a tenant cluster can see. vCluster rejects any tenant `ResourceClaim` or `ResourceClaimTemplate` referencing a class outside that selector. That gates which pool of devices a tenant can draw from. It doesn't control which specific device within an allowed class a tenant receives. That's determined by the `DeviceClass` and `ResourceSlice` design on the control plane cluster. The relationship mirrors a StorageClass and a PersistentVolumeClaim. A StorageClass controls which storage backend can be used, not which specific volume. [Node sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md) passes host GPU `capacity` and `allocatable` through to the tenant cluster unfiltered. These are total schedulable device counts, not another tenant's real-time usage. On a shared node with a small GPU pool, though, the unfiltered count alone discloses how many devices exist on that node. * **Default.** vCluster doesn't net any resource type, including CPU and memory, against other tenants' consumption on that node. * **With [virtual scheduling](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/virtual-scheduler.md) or [hybrid scheduling](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/pods/hybrid-scheduling.md) enabled.** Hybrid scheduling is an Enterprise feature; virtual scheduling isn't. CPU, memory, ephemeral storage, and pod count get netted out, but extended resources like GPU never are. The tenant cluster's own scheduler can then see more free GPU capacity than the node actually has left, since other tenants' extended-resource requests aren't accounted for. That produces optimistic scheduling decisions. A device node under `/dev`, for example `/dev/nvidia0`, can be mounted through a `hostPath` volume. Mounting the node alone doesn't grant permission to use it, since the container runtime still enforces device-cgroup rules independent of volume mounts. Combined with a privileged container or another permissive device-cgroup setting, though, this mount bypasses device-plugin accounting for whatever device it exposes. Treat it as the same class of gap as the PersistentVolume `hostPath` breakout in [Storage breakout prevention](#storage-breakout-prevention). Cover it with the same control plane cluster admission that denies `hostPath` PersistentVolumes. ## Admission and policy coverage[​](#admission-and-policy-coverage "Direct link to Admission and policy coverage") [Central admission control](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/admission-control.md) configures the webhook plumbing that lets a policy engine inspect tenant-cluster objects. Configuring it installs the plumbing only. It provides no protection on its own. Protection requires a reachable policy engine with active policies whose rules actually cover the resource kinds and operations you care about. Verify the engine has rules loaded and reachable, not just that the webhook configuration exists. Tenant-cluster admission and control plane cluster admission see different objects. Tenant admission never sees translated and synced objects. Tenant admission also doesn't see workloads that a host-side controller generates from a tenant custom resource, such as a `virt-launcher` pod from a VM. Host-side admission is required to cover those. Apply a consistent Pod Security or admission floor across every tenant namespace on the control plane cluster, not just the ones you remembered to configure. See [Rollout sequence](#rollout-sequence) before enabling `failurePolicy: Fail`. Scope any exemptions narrowly to vCluster's own system components, not broadly to entire namespaces. ## Operational hardening and lifecycle[​](#operational-hardening-and-lifecycle "Direct link to Operational hardening and lifecycle") * **PriorityClass sync.** [`sync.toHost.priorityClasses`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/priority-classes.md) lets a tenant's PriorityClass affect scheduling on shared nodes. Leave it disabled, or constrain synced values through control plane cluster admission, so one tenant can't preempt another tenant's workloads. * **Quotas and limits.** Apply `resourceQuota` and `limitRange` for Pods, and equivalent limits for storage, Services, and VM or VM-pool expansion where KubeVirt is in use. Without them, one tenant can exhaust capacity for every tenant on the same nodes. * **Node and kubelet disclosure.** Disable [`networking.advanced.proxyKubelets`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/networking/advanced.md) unless you need the metrics workflow it backs (`kubectl top node` and kubelet metrics scraping). * Node-level metric samples without a pod or PVC label aren't scoped to the tenant, on any vCluster version, whenever the proxy is enabled. * On vCluster v0.34.0 and later, its pod-listing and container-log paths are scoped to the tenant's own pods. Disabling `proxyKubelets` is a safeguard mainly for versions older than that. * `kubectl logs` and `exec` are unaffected either way, since they use an independent, ownership-validated path. * Regardless of these settings, require kubelet authentication and authorization, disable the kubelet read-only port on every shared node, and pair [`sync.fromHost.nodes`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md), if enabled, with `clearImageStatus: true`. * **Patching.** Keep the control plane cluster, node OS, node kernel, container runtime, CNI, CSI, KubeVirt, Multus, and vCluster itself on supported, security-patched versions. A container-escape-class kernel vulnerability turns any workload-level gap in this guide into full node compromise. * **Verify, don't just declare.** Periodically validate the effective controls on shared nodes, not only the `vcluster.yaml` and templates that are supposed to produce them. [Audit logs](https://vcluster.com/docs/vcluster/manage/logging.md#audit-logs) from the tenant cluster, [audit logging](https://vcluster.com/docs/platform/configure/platform-configs/audit.md) on the control plane cluster, and [monitoring](https://vcluster.com/docs/platform/maintenance/monitoring/overview.md) on both sides catch drift between declared and effective configuration. See [Validate the baseline](#validate-the-baseline) for the launch-gate tests to run before the first tenant, and repeat periodically. * **Tenant offboarding.** Deprovisioning a tenant is a separate control from onboarding it. Revoke the tenant's Platform and RBAC credentials first, then delete the tenant cluster. * Normal vCluster deletion removes the resources it synced or created on the control plane cluster, so most cleanup is automatic. Confirm no synced or controller-generated objects remain afterward, including `virt-launcher` Pods and other host-side objects a controller generated from the tenant's custom resources. * Imported host resources, such as a referenced DeviceClass or NetworkAttachmentDefinition, correctly remain, since the tenant never owned them. * Verify storage cleanup separately. A PersistentVolume with `reclaimPolicy: Retain` survives deletion by design, and that's the storage class's decision to preserve data, not a vCluster gap. ## Validate the baseline[​](#validate-the-baseline "Direct link to Validate the baseline") Test a representative sample of these controls before onboarding tenants, and extend the same pattern to every item in the checklist below. | Test | Attempt, as a tenant | Expected result | | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | Cross-tenant network reachability | Reach another tenant's Pod IP or Service directly | Denied by the platform default-deny NetworkPolicy | | Cloud metadata endpoint | Reach `169.254.169.254` from a workload Pod | Denied, once `workload.publicEgress` is scoped | | hostPath PV creation, default baseline | Create a PersistentVolume with `spec.hostPath` set | No host-side PV or `hostPath` mount is created, since `sync.toHost.persistentVolumes` is disabled | | hostPath PV creation, PV sync enabled | Create a PersistentVolume with `spec.hostPath` set | Denied by control plane cluster admission | | StorageClass selection | Create PersistentVolumeClaims naming a class reserved for another tenant and omitting `storageClassName` | The named class is left unsynced with a `SyncWarning` or denied by admission; the classless claim receives only a tenant-safe default or is denied | | Namespace-sync quota coverage, namespace sync enabled | Exceed CPU or memory limits, or create excess objects, in a namespace-sync-mapped namespace | Denied or capped by a ResourceQuota and LimitRange provisioned for that mapped namespace | | Unrestricted NetworkPolicy rule, NetworkPolicy sync enabled | Create an egress or ingress rule with no peer list, such as `egress: [{}]` | Denied by control plane cluster admission | | Wildcard toleration | Create a Pod with an empty-key, `Exists` toleration | Denied by tenant-cluster admission | | Cross-tenant Ingress claim | Create an Ingress naming another tenant's hostname or IngressClass | Denied, or served only by that tenant's own controller | Include a positive test alongside each denial. A tenant's own DNS resolution, API access, and approved workload-to-workload traffic should keep working. A baseline that blocks legitimate traffic is as much a rollout failure as one that permits an attack path. ## Shared-node hardening checklist[​](#shared-node-hardening-checklist "Direct link to Shared-node hardening checklist") Before onboarding trusted tenants onto shared nodes, confirm each group below. **Suitability and baseline** * [ ] The architecture decision is explicit and documented: shared nodes for trusted tenants, private nodes for external, resale, regulated, or otherwise untrusted tenants. * [ ] `policies.podSecurityStandard` is `restricted`, with any exception to `baseline` documented as a compatibility tradeoff. * [ ] `sync.toHost.pods.useSecretsForSATokens` is enabled. * [ ] `sync.toHost.namespaces` is disabled, or every namespace it maps to has its own equivalent ResourceQuota, LimitRange, and NetworkPolicy objects, since the baseline versions of those only cover the vCluster release namespace. **Storage** * [ ] `sync.toHost.persistentVolumes` is disabled, or backed by control plane cluster admission that denies or allowlists `hostPath` PersistentVolumes. * [ ] `sync.fromHost.storageClasses` is enabled with a nonempty selector that matches only approved classes, or control plane cluster admission validates a synced PersistentVolumeClaim's `spec.storageClassName` against an explicit allowlist. The control plane cluster's default StorageClass is tenant-safe, or admission rejects a PVC that omits `storageClassName`. **Network and endpoint isolation** * [ ] `policies.networkPolicy` is enabled, and the control plane cluster CNI is confirmed to enforce NetworkPolicy. * [ ] `policies.networkPolicy.workload.publicEgress` is disabled, or scoped to an explicit CIDR that excludes link-local addresses (`169.254.0.0/16`) and any other management or metadata endpoints. * [ ] `sync.toHost.networkPolicies` is disabled, or enabled only where a tenant's own NetworkPolicy can't widen the platform network boundary (a higher-precedence control, or host admission constraining synced `ipBlock` peers and rejecting rules with no peer list). * [ ] Multus namespace isolation, per-tenant NAD ownership, and admission allowlisting are in place if secondary networks are reachable. * [ ] LoadBalancer address pools are scoped per tenant, and public IP exposure is an explicit opt-in. * [ ] Tenant Ingress hostname claims can't collide across tenants, through namespace-scoped per-tenant controllers, or host admission that covers `spec.ingressClassName`, the legacy `kubernetes.io/ingress.class` annotation, and classless Ingresses on a shared controller. Gateway API sync, where used, has an `allowedRoutes` hostname allowlist configured on the imported Gateway. * [ ] KubeVirt, if enabled, has separate host-side admission for the VM or VMI object's networks and devices, and for the generated `virt-launcher` Pod. Tenant-cluster Pod Security Standards never see the VM custom resource. **Scheduling and admission** * [ ] Tenant workload nodes carry a dedicated label and taint, distinct from infrastructure, ingress, metrics, and control-plane node roles. The tenant-pool toleration is injected through `enforceTolerations`. Both admission layers deny explicit or wildcard tolerations capable of matching protected infrastructure taints. Host-side admission additionally permits the exact tenant-pool toleration and the required Kubernetes node-condition tolerations. * [ ] `policies.centralAdmission` has an active, reachable policy engine with rules covering the resource kinds and operations in scope, not just the webhook plumbing. * [ ] `spec.runtimeClassName` is forced through `sync.toHost.pods.runtimeClassName`, or constrained through `sync.fromHost.runtimeClasses` with a nonempty selector. * [ ] If GPU or other extended device resources are in use, `sync.fromHost.deviceClasses` is scoped with a selector. Control plane cluster admission denies `hostPath` volumes targeting device nodes. **Operational hardening and lifecycle** * [ ] `policies.resourceQuota` and `policies.limitRange` are enabled. * [ ] `sync.toHost.priorityClasses` is disabled, or synced values are constrained through admission. * [ ] `sync.fromHost.nodes` is disabled, or paired with `clearImageStatus: true` and `proxyKubelets.byHostname` / `byIP` set deliberately. * [ ] The control plane cluster, node OS/kernel, container runtime, CNI, CSI, KubeVirt, Multus, and vCluster are on supported, patched versions. **Enforcement** * [ ] All of the above is enforced through required Platform templates, not left to individual tenant-cluster configuration. * [ ] The [validation tests](#validate-the-baseline) above have been run against this tenant's actual deployment, not just declared in `vcluster.yaml`. ## Related pages[​](#related-pages "Direct link to Related pages") * [Shared-node admission policy examples](https://vcluster.com/docs/vcluster/security/shared-nodes-admission-examples.md) for worked Kyverno policies covering the toleration and NetworkPolicy peer controls above. * [Secure vCluster deployments](https://vcluster.com/docs/vcluster/security/.md) for the general security baseline and control plane hardening. * [GPU and accelerator support](https://vcluster.com/docs/vcluster/introduction/gpu-and-accelerator-support.md) for how vCluster syncs device resources between tenant and control plane clusters. * [CIS hardening guide](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/overview.md) for control plane component and API server benchmark hardening. * [Private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md) for the architecture required by untrusted tenants. * [vNode](https://www.vnode.com/docs/) for additional runtime isolation on shared nodes. --- # Deploy with Flux This guide shows how to manage tenant clusters using [Flux](https://fluxcd.io/), a GitOps tool for Kubernetes. Flux uses GitOps to keep your Kubernetes clusters in sync with the desired state defined in your Git repository. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Run the command `kubectl auth can-i create clusterrole -A` to verify that your current kube-context has administrative privileges. info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm`: Helm v3.10 is required for deploying the platform. Refer to the [Helm Installation Guide](https://Helm.sh/docs/intro/install/) if you need to install it. * `kubectl`: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. Additionally, you'll need: * A Kubernetes cluster with Flux controllers installed * The `flux` CLI tool installed on your machine (See [Flux Installation Guide](https://fluxcd.io/flux/installation/)) * The `vcluster` CLI tool installed on your machine * Basic understanding of GitOps principles ## Architecture options[​](#architecture-options "Direct link to Architecture options") When using Flux with tenant clusters, choose an architecture that aligns well with your GitOps workflow. You can configure Flux to work with tenant clusters in several ways, each offering different benefits and limitations. Unlike ArgoCD, which treats other clusters as first-class objects, Flux manages workloads on other clusters through kubeconfig references in `HelmRelease` and `Kustomization` resources. This difference in design influences how you'll structure your GitOps workflows when working with tenant clusters. The following are common approaches for integrating Flux with tenant clusters, each suited to different use cases: ### 1. Flux instance per control plane cluster[​](#flux-per-host "Direct link to 1. Flux instance per control plane cluster") When running Flux on each control plane cluster, it can manage the tenant clusters within that environment. This approach is recommended if you already use Flux for each traditional cluster and want to maintain a similar management pattern. Architecture characteristics: * One Flux instance per control plane cluster * Each Flux instance manages multiple tenant clusters on that host * Tenant cluster kubeconfig Secret management is simplified since Secrets are local to the cluster * Clear separation of responsibilities by control plane cluster * Recommended if you already use a Flux instance per traditional cluster * Provides better resource utilization since the Flux controllers are shared ### 2. Hub and spoke model[​](#hub-and-spoke "Direct link to 2. Hub and spoke model") With this approach, a central Flux instance manages multiple tenant clusters across different control plane clusters. This is a good option if you already use a single Flux instance with multiple Kubernetes clusters or if you want centralized control of all virtual environments. Architecture characteristics: * One central Flux instance manages multiple tenant clusters across different hosts * Works well with existing hub and spoke Flux setups * Requires secure kubeconfig Secret management between clusters * More efficient for large numbers of tenant clusters * Provides a single control point for all tenant cluster management * Can simplify GitOps workflows by having a single source of truth ### 3. Flux inside tenant clusters[​](#flux-inside-vclusters "Direct link to 3. Flux inside tenant clusters") While possible, running Flux inside every tenant cluster adds resource overhead and management complexity. This approach might be suitable when tenant clusters need complete isolation and independent GitOps workflows. Architecture characteristics: * Each tenant cluster runs its own Flux instance * Provides complete isolation between environments * Teams can manage their own GitOps workflows independently * Increased resource overhead (each vCluster needs its own Flux controllers) * More complex to manage at scale * Suitable for environments where strict isolation is required The standalone approach of deploying Flux in each tenant cluster might not scale effectively when managing large numbers of tenant clusters, particularly in ephemeral environments like pull request previews. In these cases, a hub-and-spoke model or a Flux instance per control plane cluster is typically more resource-efficient and can reduce the overhead of running multiple Flux controllers. ## Enable kubeconfig export[​](#enable-kubeconfig-export "Direct link to Enable kubeconfig export") To enable Flux to deploy to tenant clusters, you must create a kubeconfig Secret that Flux can reference. Configure kubeconfig export in vcluster.yaml ``` exportKubeConfig: # Set a meaningful context name context: default # Use a server URL that is accessible from the Flux controllers server: https://vcluster-name.vcluster-namespace.svc.cluster.local:443 # Skip TLS verification when Flux connects to the vCluster insecure: true # Export an additional kubeconfig secret for Flux to consume additionalSecrets: - name: vcluster-flux-kubeconfig context: default server: https://vcluster-name.vcluster-namespace.svc.cluster.local:443 insecure: true controlPlane: proxy: extraSANs: - "vcluster-demo.vcluster-demo.svc.cluster.local" ``` This configuration: * Exports the tenant cluster kubeconfig as a Secret in the host namespace * Makes the Secret available for Flux to use with the `spec.kubeConfig` field * Uses a server URL that is accessible from the Flux controllers (replace `vcluster-name` and `vcluster-namespace` with your actual values) * Sets `insecure: true` to automatically skip TLS certificate verification * Adds a TLS SAN (Subject Alternative Name) that matches the server URL, which helps prevent certificate verification errors The vCluster `exportKubeConfig` configuration creates a Secret with the kubeconfig data stored under the key `config`. When referring to this Secret in Flux resources, you must specify this key in the `secretRef.key` field, as shown in the examples below. ``` # In Flux HelmRelease spec: kubeConfig: secretRef: name: vcluster-flux-kubeconfig key: config # Must match the key used in the vCluster-generated Secret ``` Certificate verification considerations When using vCluster with Flux, proper TLS certificate configuration is essential: 1. Set `exportKubeConfig.insecure: true` in your vCluster configuration 2. Configure proper TLS SANs with the `--tls-san` flag in vCluster configuration 3. Ensure the server URL matches the certificate's SAN ``` # In your vCluster configuration controlPlane: proxy: extraSANs: - "vcluster-demo.vcluster-demo.svc.cluster.local" exportKubeConfig: server: https://vcluster-name.vcluster-namespace.svc.cluster.local:443 insecure: true ``` See the [Troubleshooting](#tls-certificate-verification-errors) section for solutions to certificate issues. ## Deploy applications to tenant clusters[​](#deploy-applications-to-tenant-clusters "Direct link to Deploy applications to tenant clusters") After the vCluster is up and running, use Flux to deploy applications directly into the tenant cluster. 1. Git Repository Create a Helm repository source vcluster-apps/sources/podinfo-repository.yaml ``` --- apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository metadata: name: podinfo namespace: vcluster-demo spec: interval: 1h url: https://stefanprodan.github.io/podinfo ``` 2. Git Repository Create a HelmRelease targeting the vCluster vcluster-apps/apps/podinfo-app.yaml ``` --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: podinfo namespace: vcluster-demo spec: chart: spec: chart: podinfo reconcileStrategy: ChartVersion sourceRef: kind: HelmRepository name: podinfo version: '*' interval: 30m kubeConfig: secretRef: name: vcluster-flux-kubeconfig key: config releaseName: podinfo targetNamespace: podinfo install: createNamespace: true values: ui: message: "Deployed by Flux to virtual cluster" ingress: enabled: true hosts: - host: podinfo.example.com paths: - path: / pathType: Prefix ``` Handling TLS certificate verification The `kubeConfig` section references the Secret created by the vCluster using the `exportKubeConfig` setting. To allow Flux to connect to the tenant cluster with a self-signed certificate, you must configure both Flux and vCluster to permit insecure TLS connections: * **Flux Helm Controller configuration**:
Add the `--insecure-kubeconfig-tls` flag to the [Helm Controller configuration](https://fluxcd.io/flux/components/helm/options/). This allows Flux to honor `insecure-skip-tls-verify: true` in the referenced kubeconfig and skip certificate verification when connecting to the tenant cluster. > **Warning:** This approach skips TLS certificate verification and is not recommended for production environments. * **vCluster configuration**:
Ensure your vCluster config includes both the TLS SAN and `insecure: true` settings, as shown in the previous example. This enables the vCluster API server to serve with a self-signed (or otherwise untrusted) certificate and accept insecure connections. * **If you still encounter certificate errors**: Use a modified Secret created with the solution in the [troubleshooting section](#tls-certificate-verification-errors): Use a modified kubeconfig Secret ``` kubeConfig: secretRef: name: vcluster-flux-kubeconfig-modified # Use the modified Secret key: config ``` 3. Git Repository Commit and push your changes Commit and push application definitions ``` git add vcluster-apps/ git commit -m "Add podinfo application for vCluster demo" git push ``` 4. Tenant Cluster Verify deployment After Flux reconciles the changes, you can connect to your vCluster and verify the application is deployed: Connect to vCluster and check deployment ``` vcluster connect vcluster-demo -n vcluster-demo # Check the deployment in the virtual cluster kubectl get namespace podinfo kubectl get pods -n podinfo kubectl get ingress -n podinfo ``` ## Manage multiple tenant clusters[​](#manage-multiple-vclusters "Direct link to Manage multiple tenant clusters") When managing multiple tenant clusters with Flux, you can use Kustomize to organize your configurations. clusters/kustomization.yaml ``` --- apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - sources/vcluster-repository.yaml - development/vcluster-dev.yaml - staging/vcluster-staging.yaml - production/vcluster-prod.yaml ``` ## Bootstrap review environments with pre-installed Flux[​](#bootstrapping-review-environments "Direct link to Bootstrap review environments with pre-installed Flux") A common scenario is having Flux already installed on the control plane cluster and wanting to use it for ephemeral review environments with tenant clusters. This approach follows GitOps principles while avoiding the need to install Flux separately for each environment. ### Use ResourceSet feature for review environments[​](#flux-resource-sets "Direct link to Use ResourceSet feature for review environments") For later versions of Flux, you can use the native Flux Operator `ResourceSet` capability to manage ephemeral environments efficiently. This is a feature for creating review environments that include tenant clusters. pr-resource-set.yaml ``` apiVersion: resourcesets.toolkit.fluxcd.io/v1alpha2 kind: ResourceSet metadata: name: pr-123 namespace: reviews spec: interval: 5m serviceAccountName: flux-reconciler resources: # First create the vCluster - apiVersion: v1 kind: Namespace metadata: name: review-123 - apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: vcluster-pr-123 namespace: review-123 spec: interval: 10m chart: spec: chart: vcluster version: "0.24.x" sourceRef: kind: HelmRepository name: vcluster namespace: flux-system values: exportKubeConfig: context: default server: https://vcluster-pr-123.review-123.svc.cluster.local:443 insecure: true additionalSecrets: - name: vcluster-pr-123-kubeconfig # Deploy the PR app code into the vCluster - apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: pr-app namespace: review-123 spec: interval: 5m chart: spec: chart: ./charts/app sourceRef: kind: GitRepository name: pull-request-123 namespace: flux-system kubeConfig: secretRef: name: vcluster-pr-123-kubeconfig key: config skipTLSVerify: true values: image: tag: pr-123 ``` This approach allows you to: * Create the vCluster and deploy applications in a single resource * Use the Flux Operator directly without having to create custom CI scripts * Simplify the cleanup when a PR is closed or merged * Manage the entire lifecycle of review environments through GitOps practices For more details, see the [Flux Operator `ResourceSet` documentation](https://fluxcd.control-plane.io/operator/resourcesets/github-pull-requests/) and an [example implementation](https://github.com/loft-demos/vcluster-platform-demo-app-template/blob/main/vcluster-use-cases/flux/pull-request-environments/pr-resource-set.yaml). ### Use existing Flux for review environments[​](#use-existing-flux-for-review-environments "Direct link to Use existing Flux for review environments") When Flux is already installed on your control plane cluster, you can create a GitOps workflow for review environments that: 1. Uses the existing Flux installation on the control plane cluster 2. Deploys tenant clusters for each review environment 3. Use Flux to deploy applications to these tenant clusters 4. Reduces overhead and speeds up environment bootstrapping 1) Git Repository Create a structure for review environments ``` ├── clusters/ │ ├── sources/ │ │ └── vcluster-repository.yaml # HelmRepository for vCluster │ └── reviews/ │ ├── review-env-template.yaml # Template for new review environments │ └── pr-123/ # Directory for a specific PR review │ └── vcluster.yaml # vCluster definition for PR-123 └── apps/ ├── sources/ │ └── app-repository.yaml # Application source repositories └── reviews/ └── pr-123/ # Apps for PR-123 environment └── deployment.yaml # Application deployment targeting PR-123 vCluster ``` 2) Git Repository Create a template for review environments clusters/reviews/review-env-template.yaml ``` --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: vcluster-${PR_NUMBER} namespace: review-${PR_NUMBER} spec: interval: 10m chart: spec: chart: vcluster version: "0.24.x" sourceRef: kind: HelmRepository name: vcluster namespace: flux-system values: sync: toHost: ingresses: enabled: true exportKubeConfig: context: default server: https://kubernetes.default.svc.cluster.local:443 additionalSecrets: - name: vcluster-${PR_NUMBER}-kubeconfig ``` 3) CI Pipeline Create a CI workflow that generates environments CI workflow (conceptual) ``` steps: - name: Checkout code uses: actions/checkout@v2 - name: Create PR-specific vCluster config run: | export PR_NUMBER=${GITHUB_REF#refs/pull/} export PR_NUMBER=${PR_NUMBER%/merge} mkdir -p clusters/reviews/pr-${PR_NUMBER} # Generate the vCluster config from template cat clusters/reviews/review-env-template.yaml | \ sed "s/\${PR_NUMBER}/$PR_NUMBER/g" > \ clusters/reviews/pr-${PR_NUMBER}/vcluster.yaml - name: Commit and push to GitOps repo run: | git add clusters/reviews/pr-${PR_NUMBER} git commit -m "Add review environment for PR #${PR_NUMBER}" git push ``` With this approach, your CI/CD pipeline creates the necessary configuration in your GitOps repository, and Flux (already running on the control plane cluster) automatically provisions the vCluster and deploys applications to it. 4) Control Plane Cluster Existing Flux detects and applies changes The Flux controllers already running on your control plane cluster is going to: * Detect the new vCluster configuration * Create the required namespace * Deploy the vCluster using the Helm chart * Create the kubeconfig Secret * Use the exported kubeconfig to deploy apps to the vCluster This entire process follows GitOps principles, with your Git repository as the source of truth, and Flux handling the reconciliation—all without requiring manual intervention or imperative commands. tip In production environments, implement automatic cleanup of preview environments when pull requests are closed or merged. You can do this by adding a CI workflow step that deletes the corresponding directory from your GitOps repository. This pattern allows you to use an existing Flux installation rather than deploying Flux separately for each review environment. This significantly reduces overhead and bootstrap time. Advanced management with vCluster Platform For organizations managing many tenant clusters, especially in dynamic, ephemeral environments, vCluster Platform offers advanced lifecycle management capabilities that integrate smoothly with GitOps workflows. It supports automatic creation of kubeconfig `Secrets`, access control management, and simplified bootstrapping of tenant clusters with Flux. ## Troubleshoot[​](#troubleshoot "Direct link to Troubleshoot") Control Plane Cluster * Verify the tenant cluster kubeconfig Secret exists with the correct format * Check Flux controller logs for errors * Ensure Flux has the necessary permissions to access the Secret Check Flux source controller logs ``` kubectl logs -n flux-system deployment/source-controller ``` Check Flux helm controller logs ``` kubectl logs -n flux-system deployment/helm-controller ``` Tenant Cluster * Verify that resources are being created in the tenant cluster * Check that the `exportKubeConfig` setting is properly configured * Ensure the server URL is reachable from the Flux controllers Check tenant cluster configuration ``` kubectl get configmap -n vcluster-namespace vcluster-flux-demo -o yaml ``` ### Common issues[​](#common-issues "Direct link to Common issues") #### TLS certificate verification errors[​](#tls-certificate-verification-errors "Direct link to TLS certificate verification errors") If you see TLS certificate verification errors in Flux controller logs like: ``` tls: failed to verify certificate: x509: certificate signed by unknown authority ``` This is a common issue when Flux attempts to connect to a vCluster, because the vCluster generates a self-signed certificate. Follow these solutions in order: ##### Solution 1: Properly configure vCluster certificate SANs[​](#solution-1-properly-configure-vcluster-certificate-sans "Direct link to Solution 1: Properly configure vCluster certificate SANs") The most reliable approach is to configure proper TLS SANs when deploying vCluster: vCluster configuration with proper TLS SANs ``` controlPlane: proxy: extraSANs: - "vcluster-demo.vcluster-demo.svc.cluster.local" exportKubeConfig: server: https://vcluster-name.vcluster-namespace.svc.cluster.local:443 insecure: true additionalSecrets: - name: vcluster-flux-kubeconfig server: https://vcluster-name.vcluster-namespace.svc.cluster.local:443 insecure: true ``` This ensures the certificate includes the correct SAN for the service DNS name. ##### Solution 2: Use a modified kubeconfig Secret[​](#solution-2-use-a-modified-kubeconfig-secret "Direct link to Solution 2: Use a modified kubeconfig Secret") If you're still encountering issues, create a modified kubeconfig Secret with TLS verification disabled: Create a modified kubeconfig Secret ``` # Set your environment variables NAMESPACE="vcluster-namespace" VCLUSTER_NAME="vcluster-name" KUBECONFIG_SECRET="vcluster-flux-kubeconfig" # Create a temporary directory TMPDIR=$(mktemp -d) cd $TMPDIR # Extract original kubeconfig kubectl get secret -n $NAMESPACE $KUBECONFIG_SECRET -o jsonpath='{.data.config}' | base64 -d > original-kubeconfig.yaml # Extract client certificates CLIENT_CERT=$(grep -A1 "client-certificate-data:" original-kubeconfig.yaml | tail -n1 | awk '{print $1}') CLIENT_KEY=$(grep -A1 "client-key-data:" original-kubeconfig.yaml | tail -n1 | awk '{print $1}') # Create kubeconfig without certificate-authority-data and with insecure-skip-tls-verify enabled cat > modified-kubeconfig.yaml << EOF apiVersion: v1 clusters: - cluster: insecure-skip-tls-verify: true server: https://${VCLUSTER_NAME}.${NAMESPACE}.svc.cluster.local:443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: ${CLIENT_CERT} client-key-data: ${CLIENT_KEY} EOF # Create Secret kubectl create secret generic ${KUBECONFIG_SECRET}-modified -n $NAMESPACE --from-file=config=modified-kubeconfig.yaml # Clean up rm -rf $TMPDIR ``` Update your Flux resource to use this Secret: Update HelmRelease or Kustomization ``` spec: kubeConfig: secretRef: name: vcluster-flux-kubeconfig-modified key: config ``` ##### Solution 3: Use Flux's built-in TLS verification options[​](#solution-3-use-fluxs-built-in-tls-verification-options "Direct link to Solution 3: Use Flux's built-in TLS verification options") To allow Flux (v2+) to connect to a tenant cluster with a self-signed certificate, you must configure the Flux Helm Controller to permit insecure TLS connections: * **Add `--insecure-kubeconfig-tls` to the Helm Controller**
Ensure the Helm Controller is started with the `--insecure-kubeconfig-tls` flag. This enables Flux to honor `insecure-skip-tls-verify: true` inside the referenced kubeconfig Secret and skip certificate verification for the target (virtual) cluster. Patch Helm Controller with --insecure-kubeconfig-tls ``` apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - gotk-components.yaml - gotk-sync.yaml patches: - patch: | - op: add path: /spec/template/spec/containers/0/args/- value: --insecure-kubeconfig-tls target: kind: Deployment name: "helm-controller" ``` #### Connection refused errors[​](#connection-refused-errors "Direct link to Connection refused errors") If you see "connection refused" errors in the Flux controller logs, it might indicate: 1. The tenant cluster's API server is not accessible from Flux 2. Network policies are blocking the communication 3. The tenant cluster is not running or healthy 4. The server URL in the kubeconfig is not correctly configured You might see errors in the Flux controller logs like: ``` connect: connection refused ``` To troubleshoot: 1. Check if the tenant cluster is running and ready: Verify vCluster status ``` kubectl get pods -n ``` 2. Verify the server URL in your `exportKubeConfig` setting: Check server URL in kubeconfig ``` kubectl get secret -n -o jsonpath='{.data.config}' | base64 -d | grep server ``` 3. Ensure the server URL is accessible from the Flux controllers. Using the service DNS name is generally more reliable: Recommended server URL format ``` server: https://vcluster-name.vcluster-namespace.svc.cluster.local:443 ``` #### Snapshots fail when HelmRelease is in a different namespace[​](#snapshot-namespace-mismatch "Direct link to Snapshots fail when HelmRelease is in a different namespace") When you deploy a tenant cluster through a `HelmRelease` resource that lives in a different namespace than the tenant cluster itself (for example, `HelmRelease` in `flux-system` and the tenant cluster in `mytestcluster`), Flux stores the Helm release secret in the `HelmRelease` namespace. The snapshot controller searches for the Helm release secret in the tenant cluster's namespace and fails to find it, which causes snapshots to fail. To fix this, set `spec.storageNamespace` in the `HelmRelease` to the tenant cluster's namespace: Set storageNamespace to match the tenant cluster namespace ``` apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: my-vcluster namespace: flux-system spec: storageNamespace: mytestcluster # Must match the tenant cluster namespace targetNamespace: mytestcluster interval: 10m chart: spec: chart: vcluster version: "0.26.x" sourceRef: kind: HelmRepository name: vcluster namespace: flux-system ``` With `storageNamespace` set, Flux stores the `helm.sh/release.v1` secret in `mytestcluster`, where the snapshot controller can find it. --- # Preview environments Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** This guide shows how to use vCluster and GitHub Actions to deploy preview environments on every pull request. The example creates a new tenant cluster for each pull request and installs Argo CD into it as a stand-in workload. It uses the sleep mode feature of vCluster Platform so that idle environments consume no resources. ## Prepare the platform[​](#prepare-the-platform "Direct link to Prepare the platform") ### Install an ingress controller[​](#install-an-ingress-controller "Direct link to Install an ingress controller") Install an ingress controller into the control plane cluster where the preview environments run. The tenant cluster template later reuses this controller to expose each preview environment through `sync.toHost.ingresses`. ### Configure a preview environments domain[​](#configure-a-preview-environments-domain "Direct link to Configure a preview environments domain") Configure a wildcard domain under which the preview environments are reachable, such as `*.my-preview-environments.com`. Point a wildcard DNS record at your ingress controller's external IP or hostname. If you don't have access to a custom domain, vCluster Platform can provide a wildcard domain for a connected cluster, which you configure in the cluster settings. ### Create a tenant cluster template[​](#create-a-tenant-cluster-template "Direct link to Create a tenant cluster template") Create a tenant cluster template for the preview environments and name it `preview-template`. In vCluster Platform, select **Templates**, then **Tenant Clusters**, and click **Add Tenant Cluster Template**. For the full flow, see [Create a template](https://vcluster.com/docs/platform/administer/templates/create-templates.md). Use the following `vcluster.yaml` values so the template reuses the ingress controller, isolates each environment, and puts idle environments to sleep: Tenant cluster template values ``` # Reuse the control plane cluster ingress controller for tenant cluster ingresses sync: toHost: ingresses: enabled: true # Enforce resource limits and pod security to keep preview environments isolated policies: resourceQuota: enabled: true limitRange: enabled: true podSecurityStandard: baseline # Put idle preview environments to sleep so they consume no resources sleep: auto: afterInactivity: 30m ``` Optional: ingress authentication Add the `loft.sh/require-ingress-authentication: "true"` annotation to the template, or enable ingress authentication in the template's advanced options, to allow only users with access to your platform instance to reach the preview environments. ### Create a preview environments project[​](#create-a-preview-environments-project "Direct link to Create a preview environments project") Create a new project named `preview` to host the preview environments. Add the template you just created as the only allowed template, and assign yourself and any other users or teams that need access to the preview environments. ## Create a GitHub deploy action[​](#create-a-github-deploy-action "Direct link to Create a GitHub deploy action") With the project configured, write the GitHub workflow that creates the preview environments. On every pull request, this workflow creates a tenant cluster, installs Argo CD, and exposes it through an ingress. The workflow logs in with `PLATFORM_URL` and `ACCESS_KEY`, populated from [GitHub secrets](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions) of the same name. Add both secrets to the repository before running the workflow. See [Access Keys](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) for how to generate a platform access key. The GitHub workflow looks like this: .github/workflows/preview-create.yaml ``` name: Create Preview Environment # Run for pull requests to the main branch on: pull_request: branches: - main # Only run one deployment at a time per pull request concurrency: group: preview-${{ github.head_ref || github.ref_name }} cancel-in-progress: true # Deploy the preview environment jobs: deploy: runs-on: ubuntu-latest # Allow the comment step to create and update the pull request comment permissions: pull-requests: write steps: - name: Checkout uses: actions/checkout@v4 - name: Install vCluster CLI uses: loft-sh/setup-vcluster@main - name: Log in to vCluster Platform env: PLATFORM_URL: ${{ secrets.PLATFORM_URL }} ACCESS_KEY: ${{ secrets.ACCESS_KEY }} run: vcluster platform login $PLATFORM_URL --access-key $ACCESS_KEY - name: Deploy preview environment run: | INGRESS_HOST=preview-${{ github.event.pull_request.number }}.my-preview-environments.com # Create the tenant cluster, or update it on later pushes to the same pull request vcluster create preview-${{ github.event.pull_request.number }} \ --project preview \ --template preview-template \ --upgrade # Deploy the example application with an ingress helm repo add argo https://argoproj.github.io/argo-helm helm upgrade --install argocd argo/argo-cd \ --create-namespace \ --namespace argocd \ --set global.domain=$INGRESS_HOST \ --set server.ingress.enabled=true - name: Comment preview URL on the pull request uses: marocchino/sticky-pull-request-comment@v2 with: number: ${{ github.event.pull_request.number }} header: preview message: | Preview environment deployed at: https://preview-${{ github.event.pull_request.number }}.my-preview-environments.com ``` Deploy your own application This workflow installs bare Argo CD as a stand-in workload. To deploy your own application through it, add an Argo CD `Application` manifest that points at your repository. See the [Argo CD documentation](https://argo-cd.readthedocs.io/en/stable/user-guide/application-specification/) for the manifest format. ## Create a destroy action[​](#create-a-destroy-action "Direct link to Create a destroy action") Environments must be cleaned up when they are no longer needed. Create a separate workflow that deletes the tenant cluster as soon as the pull request is closed or merged: .github/workflows/preview-destroy.yaml ``` name: Destroy Preview Environment on: pull_request: branches: - main types: - closed jobs: destroy: runs-on: ubuntu-latest steps: - name: Install vCluster CLI uses: loft-sh/setup-vcluster@main - name: Log in to vCluster Platform env: PLATFORM_URL: ${{ secrets.PLATFORM_URL }} ACCESS_KEY: ${{ secrets.ACCESS_KEY }} run: vcluster platform login $PLATFORM_URL --access-key $ACCESS_KEY - name: Destroy preview environment run: vcluster delete preview-${{ github.event.pull_request.number }} --project preview ``` ## Conclusion[​](#conclusion "Direct link to Conclusion") This guide showed how to build a complete flow for preview environments with vCluster Platform and GitHub Actions. After creation, a preview environment sleeps when it is not used and consumes no compute resources in the cluster. To access a preview environment, click the link in the pull request comment and the platform wakes the environment automatically. This gives you preview environments that are fast to create, isolated like separate Kubernetes clusters, and free of cost while unused. --- # GitHub actions on pull requests Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** GitHub Actions can be created for the following workflows: * [Setup vCluster](https://github.com/loft-sh/setup-vcluster): Installs the `vCluster` CLI * [Setup DevSpace](https://github.com/loft-sh/setup-devspace): Installs the `DevSpace` CLI * [Setup vind](https://github.com/loft-sh/setup-vind): Provisions a Kubernetes cluster via the [Docker driver (vind)](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/docker.md) for [CI testing](https://vcluster.com/docs/vcluster/integrations/setup-vind.md) ## Deploy tenant clusters on pull requests[​](#deploy-tenant-clusters-on-pull-requests "Direct link to Deploy tenant clusters on pull requests") Tenant clusters provide isolated Kubernetes environments for each pull request, enabling: * Parallel testing without resource conflicts between PRs * Clean environments that don't accumulate state from previous test runs * Fast creation and deletion compared to provisioning full clusters * Cost efficiency by sharing underlying infrastructure while maintaining isolation ### Deploy with host nodes[​](#deploy-with-host-nodes "Direct link to Deploy with host nodes") When using [host nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/host-nodes.md), tenant cluster workloads run on the existing control plane cluster nodes. This is the simplest setup for CI/CD pipelines. * Basic * Automatic Cleanup * Reuse This example shows how to create and delete a virtual cluster for testing an application named `the-app` on pull requests. ``` # .github/workflows/vclusters.yaml name: Pull Request Checks on: pull_request: branches: - "main" jobs: e2e: runs-on: ubuntu-latest steps: - name: Install vCluster CLI uses: loft-sh/setup-vcluster@main - name: Login to vCluster Platform instance env: PLATFORM_URL: ${{ secrets.PLATFORM_URL }} ACCESS_KEY: ${{ secrets.ACCESS_KEY }} run: vcluster platform login $PLATFORM_URL --access-key $ACCESS_KEY - name: Create PR Virtual Cluster env: NAME: repo-${{ github.event.repository.name }}-pr-${{ github.event.pull_request.number }} run: vcluster create $NAME --project default - name: Deploy Application run: kubectl apply -Rf ./kubernetes - name: Wait for Deployment run: kubectl rollout status deployments/the-app - name: Run Tests run: make e2e - name: Delete PR Virtual Cluster env: NAME: repo-${{ github.event.repository.name }}-pr-${{ github.event.pull_request.number }} run: vcluster delete $NAME --project default ``` **Explanation:** 1. The [Setup vCluster](https://github.com/loft-sh/setup-vcluster) action is used to install the vCluster CLI. 2. The `vcluster platform login` command is used to log in to the organization's platform instance. Environment variables `PLATFORM_URL` and `ACCESS_KEY` are populated using [GitHub secrets](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions). See [Access Keys](https://www.vcluster.com/docs/platform/administer/authentication/access-keys) for help generating a platform access key. 3. The `vcluster create` command is used to create a unique virtual cluster using information about the pull request in the `default` project. This automatically configures the kube context for the next steps. 4. The next step deploys the application using the runner provided `kubectl` and manifests located under `./kubernetes`. 5. Before running tests, use `kubectl` to wait for the `the-app` deployment to become ready. 6. Now, run the end-to-end tests. The example uses `make` to run tests, but the command should be customized for your testing framework. 7. Finally, the `vcluster delete` command is used to delete the virtual cluster. This example shows how to create and reuse a virtual cluster for testing an application named `the-app` on pull requests. ``` # .github/workflows/vclusters.yaml name: Pull Request Checks on: pull_request: branches: - "main" jobs: e2e: runs-on: ubuntu-latest steps: - name: Install vCluster CLI uses: loft-sh/setup-vcluster@main - name: Login to vCluster Platform instance env: PLATFORM_URL: ${{ secrets.PLATFORM_URL }} ACCESS_KEY: ${{ secrets.ACCESS_KEY }} run: vcluster platform login $PLATFORM_URL --access-key $ACCESS_KEY - name: Create PR Virtual Cluster env: NAME: repo-${{ github.event.repository.name }}-pr-${{ github.event.pull_request.number }} run: vcluster create $NAME --project default --upgrade - name: Deploy Application run: kubectl apply -Rf ./kubernetes - name: Wait for Deployment run: kubectl rollout status deployments/the-app - name: Run Tests run: make e2e ``` **Explanation:** 1. The [Setup vCluster](https://github.com/loft-sh/setup-vcluster) action is used to install the vCluster CLI. 2. The `vcluster platform login` command is used to log in to the organization's platform instance. Environment variables `PLATFORM_URL` and `ACCESS_KEY` are populated using [GitHub secrets](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions). 3. The `vcluster create` command is used to create a unique virtual cluster using information about the pull request in the `default` project. This automatically configures the kube context for the next steps. The `--upgrade` flag has been added to reuse the existing virtual cluster and upgrade it to the latest version. Additional flags may be used to control the desired virtual cluster version. 4. The next step deploys the application using the runner provided `kubectl` and manifests located under `./kubernetes`. 5. Before running tests, use `kubectl` to wait for the `the-app` deployment to become ready. 6. Now, run the end-to-end tests. The example uses `make` to run tests, but the command should be customized for your testing framework. This example shows how to automatically delete a Virtual Cluster after testing an application named `the-app` for pull requests. ``` # .github/workflows/vclusters.yaml name: Pull Request Checks on: pull_request: branches: - 'main' jobs: e2e: runs-on: ubuntu-latest steps: - name: Install vCluster CLI uses: loft-sh/setup-loft@main with: url: ${{ secrets.PLATFORM_URL }} access-key: ${{ secrets.PLATFORM_ACCESS_KEY }} - name: Create Virtual Cluster for PR uses: loft-sh/create-vcluster@main with: name: repo-${{ github.event.repository.name }}-pr-${{ github.event.pull_request.number }} auto-cleanup: true - name: Deploy Application run: kubectl apply -Rf ./kubernetes - name: Wait for Deployment run: kubectl rollout status deployments/the-app - name: Run Tests run: make e2e ``` **Explanation:** 1. The [Setup Loft](https://github.com/loft-sh/setup-loft) action is used to install the vCluster CLI and login using the provided `url` and `access-key`. 2. The [Create Virtual Cluster](https://github.com/loft-sh/create-vcluster) action is used to create a unique virtual cluster using information about the pull request. This automatically configures the kube context for the following steps. Additionally, the `auto-cleanup` option is enabled, which deletes the virtual cluster after the job completes. 3. The next step deploys the application using the runner provided `kubectl` and manifests located under `./kubernetes`. 4. Before running tests, use `kubectl` to wait for the `the-app` deployment to become ready. 5. Finally, run the end-to-end tests. This example uses `make` to run tests, but the command should be customized for your testing framework. There's no need to delete the virtual cluster since the `auto-cleanup` option was used when creating the virtual cluster. ### Deploy with private nodes[​](#deploy-with-private-nodes "Direct link to Deploy with private nodes") When using [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md), virtual clusters have dedicated worker nodes that are separate from the host cluster. For CI/CD workflows, combine private nodes with [auto-nodes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/private-nodes/auto-nodes.md) so that worker nodes are automatically provisioned when the virtual cluster is created and cleaned up when deleted. ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before using private nodes in your GitHub Actions workflow: 1. **vCluster Platform** must be installed and running 2. A **[node provider](https://vcluster.com/docs/platform/administer/node-providers/overview.md)** must be configured in vCluster Platform 3. A **virtual cluster template** with private nodes and auto-nodes enabled must be created ### Create a template with private nodes[​](#create-a-template-with-private-nodes "Direct link to Create a template with private nodes") In vCluster Platform, create a virtual cluster template with private nodes configuration: Private nodes template configuration ``` # Enable private nodes with auto-provisioning privateNodes: enabled: true autoNodes: # Reference the node provider configured in vCluster Platform - provider: my-node-provider dynamic: - name: ci-node-pool # Limit the number of nodes for CI workloads limits: nodes: "3" # Control plane configuration controlPlane: service: spec: # LoadBalancer required for private nodes connectivity type: LoadBalancer # Network configuration required for private nodes networking: podCIDR: 10.64.0.0/16 serviceCIDR: 10.128.0.0/16 ``` ### Configure GitHub Actions workflow[​](#configure-github-actions-workflow "Direct link to Configure GitHub Actions workflow") This example creates a virtual cluster with private nodes using a Platform template: ``` # .github/workflows/vclusters-private-nodes.yaml name: Pull Request Checks (Private Nodes) on: pull_request: branches: - "main" jobs: e2e: runs-on: ubuntu-latest steps: - name: Install vCluster CLI uses: loft-sh/setup-vcluster@main - name: Login to vCluster Platform instance env: PLATFORM_URL: ${{ secrets.PLATFORM_URL }} ACCESS_KEY: ${{ secrets.ACCESS_KEY }} run: vcluster platform login $PLATFORM_URL --access-key $ACCESS_KEY - name: Create PR Virtual Cluster with Private Nodes env: NAME: repo-${{ github.event.repository.name }}-pr-${{ github.event.pull_request.number }} # Use --template to reference the private nodes template run: vcluster create $NAME --project default --template private-nodes-ci - name: Wait for nodes to be ready # Auto-nodes provisions worker nodes; wait for them to join run: | echo "Waiting for private nodes to join the cluster..." kubectl wait --for=condition=Ready nodes --all --timeout=300s - name: Deploy Application run: kubectl apply -Rf ./kubernetes - name: Wait for Deployment run: kubectl rollout status deployments/the-app - name: Run Tests run: make e2e - name: Delete PR Virtual Cluster env: NAME: repo-${{ github.event.repository.name }}-pr-${{ github.event.pull_request.number }} # Deleting the vCluster also cleans up the auto-provisioned nodes run: vcluster delete $NAME --project default ``` **Explanation:** 1. The [Setup vCluster](https://github.com/loft-sh/setup-vcluster) action installs the vCluster CLI. 2. The `vcluster platform login` command authenticates to vCluster Platform using [GitHub secrets](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions). 3. The `vcluster create` command with `--template private-nodes-ci` creates a virtual cluster using the private nodes template. This triggers auto-nodes to provision dedicated worker nodes. 4. Wait for the auto-provisioned nodes to join and become ready before deploying applications. 5. Deploy and test the application on the dedicated private nodes. 6. The `vcluster delete` command removes the virtual cluster and auto-nodes automatically cleans up the provisioned worker nodes. Node provisioning time Auto-nodes provisioning typically takes 1-3 minutes depending on your cloud provider. Factor this into your CI/CD pipeline timeouts. ## DevSpace for running tests on pull requests[​](#devspace-for-running-tests-on-pull-requests "Direct link to DevSpace for running tests on pull requests") This example shows how to use the [Setup DevSpace](https://github.com/loft-sh/setup-devspace) GitHub Action to install the DevSpace CLI and [DevSpace commands](https://devspace.sh/cli/docs/configuration/commands/basics) to run tests. ``` # .github/workflows/vclusters.yaml name: Pull Request Checks on: pull_request: branches: - "main" jobs: e2e: runs-on: ubuntu-latest steps: - name: Install DevSpace CLI uses: loft-sh/setup-devspace@main - name: Install vCluster CLI uses: loft-sh/setup-vcluster@main - name: Login to platform instance env: PLATFORM_URL: ${{ secrets.PLATFORM_URL }} ACCESS_KEY: ${{ secrets.ACCESS_KEY }} run: vcluster platform login $PLATFORM_URL --access-key $ACCESS_KEY - name: Create PR Virtual Cluster env: NAME: repo-${{ github.event.repository.name }}-pr-${{ github.event.pull_request.number }} run: vcluster create $NAME --project default - name: Run Tests run: devspace run e2e ``` **Explanation:** 1. The [Setup DevSpace](https://github.com/loft-sh/setup-devspace) action installs the DevSpace CLI. 2. The [Setup vCluster](https://github.com/loft-sh/setup-vcluster) action is used to install the vCluster CLI. 3. The `vcluster platform login` command is used to log in to the organization's platform instance. Environment variables `PLATFORM_URL` and `ACCESS_KEY` are populated using [GitHub secrets](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions). 4. The `vcluster create` command is used to create a unique virtual cluster using information about the pull request in the `default` project. This automatically configures the kube context for the next steps. 5. Finally, use `devspace run e2e` to perform the needed steps to deploy and test `my-app`. --- # Netris Supported Configurations Running the control plane as a container with: * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. The Netris integration enables vCluster to use [Netris](https://www.netris.io/) for network automation for the vCluster control plane and [private nodes](https://vcluster.com/docs/vcluster/deploy/worker-nodes/private-nodes.md). ## Features[​](#features "Direct link to Features") For more information on the features capable of integrating with Netris, see: * [Automatically configure kube-vip](#automatically-configure-kube-vip) * [BCM Node Provider with Netris](https://vcluster.com/docs/platform/administer/node-providers/bcm.md) * [KubeVirt Node Provider with Netris](https://vcluster.com/docs/platform/administer/node-providers/kubevirt.md) ## Configure Netris integration[​](#configure-netris-integration "Direct link to Configure Netris integration") 1. Create a Secret in the control plane cluster in the namespace where vCluster Platform is installed to provide authentication and endpoint information: ``` apiVersion: v1 kind: Secret metadata: name: netris-credentials namespace: vcluster-platform stringData: url: "https://netris.example.com/api" username: "admin" password: "password" ``` 2. Reference the Secret in your `vcluster.yaml`: ``` integrations: netris: enabled: true connector: netris-credentials ``` ## Automatically configure kube-vip[​](#automatically-configure-kube-vip "Direct link to Automatically configure kube-vip") info This requires [Multus](https://github.com/k8snetworkplumbingwg/multus-cni) to be installed in the control plane cluster. When the Netris integration is enabled, you can configure it to automatically configure kube-vip for your vCluster control plane. The platform looks up the server cluster in Netris and uses the information to create the necessary Multus NetworkAttachmentDefinition and configure kube-vip. Add the `kubeVip` configuration to your Netris integration: ``` integrations: netris: enabled: true connector: netris-credentials kubeVip: # Netris server cluster to use for VIP allocation serverCluster: vcluster-control-plane # Optional: IP ranges to use for allocation instead of CIDR-based allocation ipRange: "10.0.0.10-10.0.0.20" ``` ### How it works[​](#how-it-works "Direct link to How it works") When configured, vCluster Platform will: 1. Query the specified Netris server cluster to get subnet, gateway & vxlan id information 2. Allocate a VIP from the subnet (or from the specified IP range if `ipRange` is set) 3. Create a NetworkAttachmentDefinition with the specified bridge 4. Transparently configure kube-vip for the vCluster instance to announce the VIP You don't need to configure any of this yourself. The equivalent manual configuration would look like this: ``` # transparently configured controlPlane: # Set the endpoint to the allocated VIP endpoint: 10.0.0.11:8443 statefulSet: annotations: # Reference the auto-created NetworkAttachmentDefinition k8s.v1.cni.cncf.io/networks: netris-kubevip- advanced: kubeVip: # enable kube-vip enabled: true # set to Multus interface interface: net1 # set gateway and subnet prefix for policy-based routing gateway: 10.0.0.1/24 ``` For more information on manual kube-vip configuration, see [Kube-vip](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/kube-vip.md). ## Additional resources[​](#additional-resources "Direct link to Additional resources") * [Netris website](https://www.netris.io/) --- # AKS Workload Identity Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** [Workload Identity](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview) allows Kubernetes workloads to securely access Azure resources without storing credentials in the cluster. It federates a Kubernetes Service Account with a Microsoft Entra identity, which improves security and simplifies identity management. To configure Workload Identity in a tenant cluster, you must link the Kubernetes Service Account (KSA) used by your workload with a Microsoft Entra Workload ID. This KSA must exist in the control plane cluster where the tenant cluster runs. Use the [`sync.toHost`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host.md) feature to expose the KSA to the control plane cluster. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Ensure you have the following: * `kubectl` installed * `az` CLI installed and configured * A running AKS cluster with Workload Identity Federation enabled. For more information, see the [Deploy and configure Workload Identity on an Azure Kubernetes Service (AKS) cluster documentation](https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster). ## Integrate AKS Workload Identity with vCluster[​](#integrate-aks-workload-identity-with-vcluster "Direct link to Integrate AKS Workload Identity with vCluster") 1. To integrate your workloads with [Azure Kubernetes Service (AKS) Workload Identity](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview), you must have a running instance of the vCluster Platform. If you have not set one up yet, follow the [vCluster Platform installation guide](https://vcluster.com/docs/platform/install/quick-start-guide.md). After deploying the platform, [create a new access key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md) to authenticate with the vCluster Platform API. This key is required to retrieve the synced Kubernetes Service Account (KSA) name and complete the Workload Identity setup. 2. Set up the necessary environment variables. These variables include information about your Azure resource group, vCluster details, and authentication keys. Modify the following with your specific values to generate a copyable command: RESOURCE\_GROUPvcluster-demo-rg SERVICE\_ACCOUNT\_NAMEvcluster-demo SERVICE\_ACCOUNT\_NAMESPACEdefault VCLUSTER\_NAMEvcluster-demo VCLUSTER\_NAMESPACEteam-x AKS\_CLUSTER\_NAMEmy-aks-cluster PLATFORM\_HOSTYOUR\_PLATFORM\_HOST ACCESS\_KEYYOUR\_ACCESS\_KEY REGIONwesteurope FEDERATED\_IDENTITY\_CREDENTIAL\_NAMEvcluster-demo USER\_ASSIGNED\_IDENTITY\_NAMEvcluster-demo-workload-identity ``` export RESOURCE_GROUP_NAME=vcluster-demo-rg export SERVICE_ACCOUNT_NAME=vcluster-demo export SERVICE_ACCOUNT_NAMESPACE=default export VCLUSTER_NAME=vcluster-demo export VCLUSTER_NAMESPACE=team-x export AKS_CLUSTER_NAME=my-aks-cluster export HOST=YOUR_PLATFORM_HOST export ACCESS_KEY=YOUR_ACCESS_KEY export REGION=westeurope export FEDERATED_IDENTITY_CREDENTIAL_NAME=vcluster-demo export USER_ASSIGNED_IDENTITY_NAME=vcluster-demo-workload-identity ``` Ensure you specify the correct Azure Resource Group name, vCluster Platform host, and auth key. 3. Create a `vcluster.yaml` file that includes the following configuration: ``` sync: toHost: serviceAccounts: enabled: true ``` 4. Run the following command to deploy the tenant cluster: ``` vcluster create $VCLUSTER_NAME --namespace $VCLUSTER_NAMESPACE --values vcluster.yaml ``` 5. Create a Kubernetes `ServiceAccount` that can be used to interact with the tenant cluster: ``` cat <-v-` - vCluster without the Platform - Platform-managed vCluster For vCluster deployments not connected with the platform (i.e. deployed with `vcluster create` or Helm without the platform): ``` az identity federated-credential create \ --name ${FEDERATED_IDENTITY_CREDENTIAL_NAME} \ --identity-name "${USER_ASSIGNED_IDENTITY_NAME}" \ --resource-group "${RESOURCE_GROUP_NAME}" \ --issuer "${AKS_OIDC_ISSUER}" \ --subject system:serviceaccount:"${VCLUSTER_NAMESPACE}":"${KSA_NAME}" \ --audience api://AzureADTokenExchange ``` For vCluster deployments managed by the platform with a project: ``` # Set project name and construct the namespace export PROJECT_NAME="my-project" # Replace with your actual project name export PLATFORM_NAMESPACE="loft-${PROJECT_NAME}-v-${VCLUSTER_NAME}" az identity federated-credential create \ --name ${FEDERATED_IDENTITY_CREDENTIAL_NAME} \ --identity-name "${USER_ASSIGNED_IDENTITY_NAME}" \ --resource-group "${RESOURCE_GROUP_NAME}" \ --issuer "${AKS_OIDC_ISSUER}" \ --subject system:serviceaccount:"${PLATFORM_NAMESPACE}":"${KSA_NAME}" \ --audience api://AzureADTokenExchange ``` 8. Assign the necessary RBAC roles to the managed identity so it can access Azure resources. For this example, you'll assign the `Reader` role at the subscription level: ``` # Get the principal ID of the managed identity export PRINCIPAL_ID="$(az identity show \ --resource-group "${RESOURCE_GROUP_NAME}" \ --name "${USER_ASSIGNED_IDENTITY_NAME}" \ --query 'principalId' \ --output tsv)" # Assign Reader role to the managed identity at subscription level az role assignment create \ --assignee "${PRINCIPAL_ID}" \ --role "Reader" \ --scope "/subscriptions/${SUBSCRIPTION}" # Wait for role assignment to propagate echo "Waiting for role assignment to propagate..." sleep 30 # Verify the role assignment az role assignment list --assignee "${PRINCIPAL_ID}" --output table ``` note You can assign more specific roles and scopes based on your requirements. For example, if you need the identity to manage specific resource groups or resources, assign appropriate roles such as "Contributor" or custom roles with specific permissions. 9. Annotate the service account with the user-assigned identity client ID and tenant ID: ``` kubectl annotate serviceaccount \ "${SERVICE_ACCOUNT_NAME}" \ -n "${SERVICE_ACCOUNT_NAMESPACE}" \ azure.workload.identity/use="true" \ azure.workload.identity/client-id="${USER_ASSIGNED_CLIENT_ID}" \ azure.workload.identity/tenant-id="${TENANT_ID}" ``` 10. Deploy an application. Replace placeholders with your actual values: ``` cat < * vCluster CLI * Helm * Terraform * ArgoCD * Flux * Cluster API The vCluster CLI provides the most straightforward way to deploy and manage virtual clusters. 1. Install the vCluster CLI: * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` 2. Deploy vCluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` vcluster create my-vcluster --namespace team-x --values vcluster.yaml ``` note After installation, vCluster automatically switches your Kubernetes context to the new virtual cluster. You can now run `kubectl` commands against the virtual cluster. Helm provides fine-grained control over the deployment process and integrates well with existing Helm-based workflows. 1. Deploy vCluster using the `helm upgrade` command: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CONFIG FILEvcluster.yaml NAMESPACEteam-x ``` helm upgrade --install my-vcluster vcluster \ --values vcluster.yaml \ --repo https://charts.loft.sh \ --namespace team-x \ --repository-config='' \ --create-namespace ``` You can use Terraform to deploy vCluster as code with version control and state management. 1. Create a `main.tf` file to define your vCluster deployment using the Terraform Helm provider: ``` provider "helm" { kubernetes = { config_path = "~/.kube/config" } } resource "helm_release" "my_vcluster" { name = "my-vcluster" namespace = "team-x" create_namespace = true repository = "https://charts.loft.sh" chart = "vcluster" # If you didn't create a vcluster.yaml, remove the values section. values = [ file("${path.module}/vcluster.yaml") ] } ``` Helm Provider Version This configuration uses the Terraform Helm provider v3.x syntax where `kubernetes` is defined as an argument (`kubernetes = {`). If you're using Helm provider v2.x, use the block syntax instead (`kubernetes {`). To use v3.x, ensure your provider version is at least v3.0.0. 2. Install the required [Helm provider](https://registry.terraform.io/providers/hashicorp/helm/latest) and initialize Terraform: ``` terraform init ``` 3. Generate a plan to preview the changes: ``` terraform plan ```
Review the plan output to verify connectivity and proposed changes. 4. Deploy vCluster: ``` terraform apply ``` ArgoCD deployment enables GitOps workflows for vCluster management, and provides automated deployment, drift detection, and declarative configuration management through Git repositories. To deploy vCluster using ArgoCD, you need the following files: * `vcluster.yaml` for your vCluster configuration options. * `-app.yaml` for your ArgoCD `Application` definition. Replace `` with your actual cluster name.
1. Create the ArgoCD `Application` file `-app.yaml`, which references the vCluster Helm chart: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x ``` --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-vcluster namespace: argocd spec: project: default source: chart: vcluster repoURL: https://charts.loft.sh helm: releaseName: my-vcluster valueFiles: - vcluster.yaml destination: server: https://kubernetes.default.svc namespace: team-x ``` 2. Commit and push these files to your configured ArgoCD repository. 3. Sync your ArgoCD repository with your configured cluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster ``` argocd app sync my-vcluster ``` 1) Create a `HelmRepository` source in the Git repository that Flux monitors, so Flux can fetch the vCluster Helm charts automatically. In this example, the related files are stored under the `clusters` folder. Save the following file as `clusters/sources/vcluster-repository.yaml`: Modify the following with your specific values to generate a copyable command: REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system ``` --- apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository metadata: name: vcluster namespace: flux-system spec: interval: 1h url: https://charts.loft.sh ``` 2) The vCluster will be deployed into the namespace `team-x` of the host cluster. If this namespace doesn’t already exist, create it with: Modify the following with your specific values to generate a copyable command: NAMESPACEteam-x ``` kubectl create namespace team-x ``` 3) Create a vCluster `HelmRelease` file in your Git repository. This `HelmRelease` tells Flux how to deploy a vCluster in your Kubernetes cluster using the configured Helm charts in Step 1. Save the following file as `clusters/production/vcluster-demo.yaml`: Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system SECRET NAMEvcluster-flux-kubeconfig ``` --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: vcluster-demo namespace: team-x spec: interval: 10m chart: spec: chart: vcluster version: "0.28.x" sourceRef: kind: HelmRepository name: vcluster namespace: flux-system values: # Configure TLS SAN for the certificate controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" coredns: enabled: true exportKubeConfig: # Set a meaningful context name context: default # Use a server URL that matches the TLS SAN server: https://vcluster-demo.team-x.svc.cluster.local:443 # Skip TLS verification when Flux connects to the vCluster insecure: true # Specify the secret where the KubeConfig is stored secret: name: vcluster-flux-kubeconfig sync: toHost: ingresses: enabled: true ``` The content in `vcluster.yaml` should be placed under the `values`. Unlike ArgoCD, Flux requires the KubeConfig secret to be able to access the vCluster's API server and deploy to vCluster. The `exportKubeConfig` configuration under `values`: * Exports the virtual cluster KubeConfig as a Secret in the host namespace * Makes the Secret available for Flux to use with the `spec.kubeConfig` field * Uses a server URL that is accessible from the Flux controllers (replace `vcluster-name` and `team-x` with your actual values) * Sets `insecure: true` to automatically skip TLS certificate verification * Adds a TLS SAN (Subject Alternative Name) that matches the server URL, which helps prevent certificate verification errors KubeConfig Secret key The vCluster `exportKubeConfig` configuration creates a Secret with the KubeConfig data stored under the key `config`. When referring to this Secret in Flux resources, you must specify this key in the `secretRef.key` field, as shown in the examples below. Modify the following with your specific values to generate a copyable command: SECRET NAMEvcluster-flux-kubeconfig ``` # In Flux HelmRelease spec: kubeConfig: secretRef: name: vcluster-flux-kubeconfig key: config # Must match the key used in the vCluster-generated Secret ``` Certificate verification considerations When using vCluster with Flux, proper TLS certificate configuration is essential: 1. Set `exportKubeConfig.insecure: true` in your vCluster configuration 2. Configure proper TLS SANs with the `--tls-san` flag in vCluster configuration 3. Ensure the server URL matches the certificate's SAN Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x ``` # In your vCluster configuration controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" exportKubeConfig: server: https://vcluster-demo.team-x.svc.cluster.local:443 insecure: true ``` 4) After adding the `HelmRelease` and any supporting files to your Git repository, commit and push them: Modify the following with your specific values to generate a copyable command: FOLDER NAMEclusters ``` git add clusters/ git commit -m "Add vCluster demo configuration" git push ``` Once the changes are pushed, Flux will automatically detect them and deploy the vCluster according to the configuration in your repository. Cluster API (CAPI) provides lifecycle management for Kubernetes clusters. The vCluster CAPI provider enables you to manage virtual clusters using the same declarative APIs and tooling used for physical clusters. For more details, see the [Cluster API Provider for vCluster documentation](https://github.com/loft-sh/cluster-api-provider-vcluster). 1. Install the [`clusterctl` CLI](https://cluster-api.sigs.k8s.io/user/quick-start.html#install-clusterctl). 2. Install the vCluster provider: ``` clusterctl init --infrastructure vcluster:v0.2.0 ``` 3. Export environment variables for the Cluster API provider to create the manifest. The manifest is applied to your Kubernetes cluster, which deploys a vCluster. Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` export CLUSTER_NAME=my-vcluster export CLUSTER_NAMESPACE=team-x export VCLUSTER_YAML=$(awk '{printf "%s\n", $0}' vcluster.yaml) ``` 4. Create the namespace for the vCluster using the exported variable: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x ``` kubectl create namespace team-x ``` 5. Generate the required manifests and apply them using the exported variables: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CLUSTER NAMESPACEteam-x ``` clusterctl generate cluster my-vcluster \ --infrastructure vcluster \ --target-namespace team-x \ | kubectl apply -f - ``` Kubernetes version The Kubernetes version for the vCluster is not set at the CAPI provider command. Configure it in the `vcluster.yaml` file based on your Kubernetes distribution. 6. Wait for vCluster to become ready by monitoring the vCluster custom resource status: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x CLUSTER NAMEmy-vcluster ``` kubectl wait --for=condition=ready vcluster -n team-x my-vcluster --timeout=300s ``` #### 5. Connect to vCluster[​](#5-connect-to-vcluster "Direct link to 5. Connect to vCluster") Establish a connection to your vCluster instance: ``` vcluster connect ${VCLUSTER_NAME} ``` #### 6. Create example workload[​](#6-create-example-workload "Direct link to 6. Create example workload") Create an example workload to list S3 buckets. ``` # Create example-workload.yaml content dynamically cat < example-workload.yaml apiVersion: v1 kind: ServiceAccount metadata: name: demo-sa namespace: default --- apiVersion: apps/v1 kind: Deployment metadata: name: s3-list-buckets namespace: default spec: replicas: 1 selector: matchLabels: app: s3-list-buckets template: metadata: labels: app: s3-list-buckets spec: serviceAccountName: demo-sa containers: - image: public.ecr.aws/aws-cli/aws-cli command: - "aws" - "s3" - "ls" name: aws-pod EOF kubectl apply -f example-workload.yaml ``` #### 7. Read updated name from platform API[​](#7-read-updated-name-from-platform-api "Direct link to 7. Read updated name from platform API") vCluster rewrites the ServiceAccount name when it syncs the ServiceAccount to the control plane cluster. The synced name follows the pattern `NAME-x-NAMESPACE-x-VCLUSTER_NAME`. Cloud IAM trust policies must reference that synced name, not the name used inside the tenant cluster. vCluster caps the rewritten name at 63 characters. This cap is vCluster's own, not a Kubernetes limit, since Kubernetes accepts ServiceAccount names up to 253 characters. When the rewritten name is longer, vCluster keeps its first 52 characters and appends `-` plus the first 10 characters of the hex-encoded SHA-256 digest of the full rewritten name. A trust policy built from the untruncated name doesn't match, and the failure surfaces as a permissions error rather than a naming error. Resolve the name through the API Don't hard-code synced ServiceAccount names or reimplement the truncation rule in your own automation. Query the vCluster Platform API as shown below so the name stays correct when namespace or tenant cluster names change. This applies to ServiceAccounts that sync into the vCluster control plane's namespace, which is the default. If you enable namespace syncing and the ServiceAccount's namespace matches a mapping, vCluster keeps the name unchanged and the API returns a name that doesn't exist. In that case, reference the ServiceAccount by its original name in the mapped namespace on the control plane cluster. * bash * Terraform Define a function to fetch the KSA name using curl and use it to export `KSA_NAME` environment variable. ``` # Define the function to get the KSA name using curl get_ksa_name() { local vcluster_ksa_name=$1 local vcluster_ksa_namespace=$2 local vcluster_name=$3 local host=$4 local access_key=$5 local resource_path="/kubernetes/management/apis/management.loft.sh/v1/translatevclusterresourcenames" local host_with_scheme=$([[ $host =~ ^(http|https):// ]] && echo "$host" || echo "https://$host") local sanitized_host="${host_with_scheme%/}" local full_url="${sanitized_host}${resource_path}" local response=$(curl -s -k -X POST "$full_url" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${access_key}" \ -d @- <my-policy.json <trust-relationship.json <-v-` - vCluster without the Platform - Platform-managed vCluster For vCluster deployments not connected with the platform (i.e. deployed with `vcluster create` or Helm without the platform): ``` # Set the namespace where vCluster is deployed export VCLUSTER_NAMESPACE="team-x" # Replace with your actual namespace aws eks create-pod-identity-association \ --cluster-name ${CLUSTER_NAME} \ --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/my-role \ --namespace ${VCLUSTER_NAMESPACE} \ --service-account ${KSA_NAME} ``` For vCluster deployments managed by the platform with a project: ``` # Set project name and construct the namespace export PROJECT_NAME="my-project" # Replace with your actual project name export PLATFORM_NAMESPACE="loft-${PROJECT_NAME}-v-${VCLUSTER_NAME}" aws eks create-pod-identity-association \ --cluster-name ${CLUSTER_NAME} \ --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/my-role \ --namespace ${PLATFORM_NAMESPACE} \ --service-account ${KSA_NAME} ``` ``` data "aws_iam_policy_document" "assume_role" { statement { effect = "Allow" principals { type = "Service" identifiers = ["pods.eks.amazonaws.com"] } actions = [ "sts:AssumeRole", "sts:TagSession" ] } } resource "aws_iam_role" "example" { name = "eks-pod-identity-example" assume_role_policy = data.aws_iam_policy_document.assume_role.json } resource "aws_iam_role_policy_attachment" "example_s3" { policy_arn = "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess" role = aws_iam_role.example.name } ``` #### 9. Verify the setup[​](#9-verify-the-setup "Direct link to 9. Verify the setup") Verify the setup by checking the logs. ``` kubectl logs -l app=s3-list-buckets -n default ``` --- Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** # Integrating GCP Workload Identity with vCluster This tutorial guides you through the process of integrating GCP Service Accounts with your vCluster using Workload Identity. Setting up Workload Identity requires you to link a GCP Service Account with the Kubernetes Service Account (KSA) used by your workloads. This KSA needs to be available in the control plane cluster in which your vCluster instance runs. To achieve this setup, we'll need to use the [sync.toHost feature](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host.md) to expose the KSA in the control plane cluster together with the vCluster Platform API to retrieve the updated name of the KSA in the control plane cluster. ### Prerequisites[​](#prerequisites "Direct link to Prerequisites") This guide assumes you have the following prerequisites: * `kubectl` installed * `gcloud` CLI installed and configured * A running GKE cluster with Workload Identity Federation enabled ([GCP docs](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#enable_on_clusters_and_node_pools)) ### Step-by-Step Guide[​](#step-by-step-guide "Direct link to Step-by-Step Guide") #### 1. Start vCluster Platform and create access key[​](#1-start-vcluster-platform-and-create-access-key "Direct link to 1. Start vCluster Platform and create access key") In order to integrate your workloads with GKE Workload Identity, you'll need a vCluster Platform instance running. If you don't have one already, follow the [vCluster Platform installation guide](https://vcluster.com/docs/platform/install/quick-start-guide.md). Once you're done, you'll need to create a new access key. This will allow you to use the vCluster Platform API. Please follow this [guide to create a new access key](https://vcluster.com/docs/platform/administer/authentication/access-keys.md). #### 2. Set Up Environment Variables[​](#2-set-up-environment-variables "Direct link to 2. Set Up Environment Variables") Next, you need to set up the necessary environment variables. These variables include information about your GCP project, vCluster details, and authentication keys. ``` # Set up environment variables export GSA_NAME=my-gke-sa export VCLUSTER_KSA_NAME=demo-sa export VCLUSTER_KSA_NAMESPACE=default export GCP_PROJECT_ID=my-gcp-project-id-12345 # Replace with your actual GCP project ID export VCLUSTER_NAME=my-vcluster export HOST=https://my.loft.host # Replace with your actual host export ACCESS_KEY=abcd1234 # Replace with your actual vCluster Platform access key ``` Before proceeding adjust all values to your specific case, make sure to specify correct GCP project ID, vCluster Platform host and auth key. #### 3. Read Updated Name From vCluster Platform API[​](#3-read-updated-name-from-vcluster-platform-api "Direct link to 3. Read Updated Name From vCluster Platform API") vCluster rewrites the ServiceAccount name when it syncs the ServiceAccount to the control plane cluster. The synced name follows the pattern `NAME-x-NAMESPACE-x-VCLUSTER_NAME`. Cloud IAM trust policies must reference that synced name, not the name used inside the tenant cluster. vCluster caps the rewritten name at 63 characters. This cap is vCluster's own, not a Kubernetes limit, since Kubernetes accepts ServiceAccount names up to 253 characters. When the rewritten name is longer, vCluster keeps its first 52 characters and appends `-` plus the first 10 characters of the hex-encoded SHA-256 digest of the full rewritten name. A trust policy built from the untruncated name doesn't match, and the failure surfaces as a permissions error rather than a naming error. Resolve the name through the API Don't hard-code synced ServiceAccount names or reimplement the truncation rule in your own automation. Query the vCluster Platform API as shown below so the name stays correct when namespace or tenant cluster names change. This applies to ServiceAccounts that sync into the vCluster control plane's namespace, which is the default. If you enable namespace syncing and the ServiceAccount's namespace matches a mapping, vCluster keeps the name unchanged and the API returns a name that doesn't exist. In that case, reference the ServiceAccount by its original name in the mapped namespace on the control plane cluster. * bash * Terraform Define a function to fetch the KSA name using curl and use it to export `KSA_NAME` environment variable. ``` # Define the function to get the KSA name using curl get_ksa_name() { local vcluster_ksa_name=$1 local vcluster_ksa_namespace=$2 local vcluster_name=$3 local host=$4 local access_key=$5 local resource_path="/kubernetes/management/apis/management.loft.sh/v1/translatevclusterresourcenames" local host_with_scheme=$([[ $host =~ ^(http|https):// ]] && echo "$host" || echo "https://$host") local sanitized_host="${host_with_scheme%/}" local full_url="${sanitized_host}${resource_path}" local response=$(curl -s -k -X POST "$full_url" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${access_key}" \ -d @- < * vCluster CLI * Helm * Terraform * ArgoCD * Flux * Cluster API The vCluster CLI provides the most straightforward way to deploy and manage virtual clusters. 1. Install the vCluster CLI: * Homebrew * Mac (Intel/AMD) * Mac (Silicon/ARM) * Linux (AMD) * Linux (ARM) * Download Binary * Windows PowerShell ``` brew install loft-sh/tap/vcluster ``` The binaries in the tap are signed using the [Sigstore](https://docs.sigstore.dev/) framework for enhanced security. ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` ``` curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster ``` Download the binary for your platform from the [GitHub Releases](https://github.com/loft-sh/vcluster/releases) page and add this binary to your $PATH. ``` md -Force "$Env:APPDATA\vcluster"; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'; Invoke-WebRequest -URI "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-windows-amd64.exe" -o $Env:APPDATA\vcluster\vcluster.exe; $env:Path += ";" + $Env:APPDATA + "\vcluster"; [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::User); ``` Reboot Required You may need to reboot your computer to use the CLI due to changes to the PATH variable (see below). Check Environment Variable $PATH Line 4 of this install script adds the install directory `%APPDATA%\vcluster` to the `$PATH`environment variable. This is only effective for the current PowerShell session, i.e. when opening a new terminal window,`vcluster` may not be found. **Make sure to add the folder `%APPDATA%\vcluster` to the `PATH` environment variable after installing vcluster CLI via PowerShell. Afterward, a reboot might be necessary.** Verify the CLI installed successfully. ``` vcluster --version ``` Output is similar to: ``` vCluster version 0.x.x ``` 2. Deploy vCluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` vcluster create my-vcluster --namespace team-x --values vcluster.yaml ``` note After installation, vCluster automatically switches your Kubernetes context to the new virtual cluster. You can now run `kubectl` commands against the virtual cluster. Helm provides fine-grained control over the deployment process and integrates well with existing Helm-based workflows. 1. Deploy vCluster using the `helm upgrade` command: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CONFIG FILEvcluster.yaml NAMESPACEteam-x ``` helm upgrade --install my-vcluster vcluster \ --values vcluster.yaml \ --repo https://charts.loft.sh \ --namespace team-x \ --repository-config='' \ --create-namespace ``` You can use Terraform to deploy vCluster as code with version control and state management. 1. Create a `main.tf` file to define your vCluster deployment using the Terraform Helm provider: ``` provider "helm" { kubernetes = { config_path = "~/.kube/config" } } resource "helm_release" "my_vcluster" { name = "my-vcluster" namespace = "team-x" create_namespace = true repository = "https://charts.loft.sh" chart = "vcluster" # If you didn't create a vcluster.yaml, remove the values section. values = [ file("${path.module}/vcluster.yaml") ] } ``` Helm Provider Version This configuration uses the Terraform Helm provider v3.x syntax where `kubernetes` is defined as an argument (`kubernetes = {`). If you're using Helm provider v2.x, use the block syntax instead (`kubernetes {`). To use v3.x, ensure your provider version is at least v3.0.0. 2. Install the required [Helm provider](https://registry.terraform.io/providers/hashicorp/helm/latest) and initialize Terraform: ``` terraform init ``` 3. Generate a plan to preview the changes: ``` terraform plan ```
Review the plan output to verify connectivity and proposed changes. 4. Deploy vCluster: ``` terraform apply ``` ArgoCD deployment enables GitOps workflows for vCluster management, and provides automated deployment, drift detection, and declarative configuration management through Git repositories. To deploy vCluster using ArgoCD, you need the following files: * `vcluster.yaml` for your vCluster configuration options. * `-app.yaml` for your ArgoCD `Application` definition. Replace `` with your actual cluster name.
1. Create the ArgoCD `Application` file `-app.yaml`, which references the vCluster Helm chart: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x ``` --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-vcluster namespace: argocd spec: project: default source: chart: vcluster repoURL: https://charts.loft.sh helm: releaseName: my-vcluster valueFiles: - vcluster.yaml destination: server: https://kubernetes.default.svc namespace: team-x ``` 2. Commit and push these files to your configured ArgoCD repository. 3. Sync your ArgoCD repository with your configured cluster: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster ``` argocd app sync my-vcluster ``` 1) Create a `HelmRepository` source in the Git repository that Flux monitors, so Flux can fetch the vCluster Helm charts automatically. In this example, the related files are stored under the `clusters` folder. Save the following file as `clusters/sources/vcluster-repository.yaml`: Modify the following with your specific values to generate a copyable command: REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system ``` --- apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository metadata: name: vcluster namespace: flux-system spec: interval: 1h url: https://charts.loft.sh ``` 2) The vCluster will be deployed into the namespace `team-x` of the host cluster. If this namespace doesn’t already exist, create it with: Modify the following with your specific values to generate a copyable command: NAMESPACEteam-x ``` kubectl create namespace team-x ``` 3) Create a vCluster `HelmRelease` file in your Git repository. This `HelmRelease` tells Flux how to deploy a vCluster in your Kubernetes cluster using the configured Helm charts in Step 1. Save the following file as `clusters/production/vcluster-demo.yaml`: Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x REPOSITORY NAMEvcluster FLUX NAMESPACEflux-system SECRET NAMEvcluster-flux-kubeconfig ``` --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: vcluster-demo namespace: team-x spec: interval: 10m chart: spec: chart: vcluster version: "0.28.x" sourceRef: kind: HelmRepository name: vcluster namespace: flux-system values: # Configure TLS SAN for the certificate controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" coredns: enabled: true exportKubeConfig: # Set a meaningful context name context: default # Use a server URL that matches the TLS SAN server: https://vcluster-demo.team-x.svc.cluster.local:443 # Skip TLS verification when Flux connects to the vCluster insecure: true # Specify the secret where the KubeConfig is stored secret: name: vcluster-flux-kubeconfig sync: toHost: ingresses: enabled: true ``` The content in `vcluster.yaml` should be placed under the `values`. Unlike ArgoCD, Flux requires the KubeConfig secret to be able to access the vCluster's API server and deploy to vCluster. The `exportKubeConfig` configuration under `values`: * Exports the virtual cluster KubeConfig as a Secret in the host namespace * Makes the Secret available for Flux to use with the `spec.kubeConfig` field * Uses a server URL that is accessible from the Flux controllers (replace `vcluster-name` and `team-x` with your actual values) * Sets `insecure: true` to automatically skip TLS certificate verification * Adds a TLS SAN (Subject Alternative Name) that matches the server URL, which helps prevent certificate verification errors KubeConfig Secret key The vCluster `exportKubeConfig` configuration creates a Secret with the KubeConfig data stored under the key `config`. When referring to this Secret in Flux resources, you must specify this key in the `secretRef.key` field, as shown in the examples below. Modify the following with your specific values to generate a copyable command: SECRET NAMEvcluster-flux-kubeconfig ``` # In Flux HelmRelease spec: kubeConfig: secretRef: name: vcluster-flux-kubeconfig key: config # Must match the key used in the vCluster-generated Secret ``` Certificate verification considerations When using vCluster with Flux, proper TLS certificate configuration is essential: 1. Set `exportKubeConfig.insecure: true` in your vCluster configuration 2. Configure proper TLS SANs with the `--tls-san` flag in vCluster configuration 3. Ensure the server URL matches the certificate's SAN Modify the following with your specific values to generate a copyable command: RELEASE NAMEvcluster-demo NAMESPACEteam-x ``` # In your vCluster configuration controlPlane: proxy: extraSANs: - "vcluster-demo.team-x.svc.cluster.local" exportKubeConfig: server: https://vcluster-demo.team-x.svc.cluster.local:443 insecure: true ``` 4) After adding the `HelmRelease` and any supporting files to your Git repository, commit and push them: Modify the following with your specific values to generate a copyable command: FOLDER NAMEclusters ``` git add clusters/ git commit -m "Add vCluster demo configuration" git push ``` Once the changes are pushed, Flux will automatically detect them and deploy the vCluster according to the configuration in your repository. Cluster API (CAPI) provides lifecycle management for Kubernetes clusters. The vCluster CAPI provider enables you to manage virtual clusters using the same declarative APIs and tooling used for physical clusters. For more details, see the [Cluster API Provider for vCluster documentation](https://github.com/loft-sh/cluster-api-provider-vcluster). 1. Install the [`clusterctl` CLI](https://cluster-api.sigs.k8s.io/user/quick-start.html#install-clusterctl). 2. Install the vCluster provider: ``` clusterctl init --infrastructure vcluster:v0.2.0 ``` 3. Export environment variables for the Cluster API provider to create the manifest. The manifest is applied to your Kubernetes cluster, which deploys a vCluster. Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster NAMESPACEteam-x CONFIG FILEvcluster.yaml ``` export CLUSTER_NAME=my-vcluster export CLUSTER_NAMESPACE=team-x export VCLUSTER_YAML=$(awk '{printf "%s\n", $0}' vcluster.yaml) ``` 4. Create the namespace for the vCluster using the exported variable: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x ``` kubectl create namespace team-x ``` 5. Generate the required manifests and apply them using the exported variables: Modify the following with your specific values to generate a copyable command: CLUSTER NAMEmy-vcluster CLUSTER NAMESPACEteam-x ``` clusterctl generate cluster my-vcluster \ --infrastructure vcluster \ --target-namespace team-x \ | kubectl apply -f - ``` Kubernetes version The Kubernetes version for the vCluster is not set at the CAPI provider command. Configure it in the `vcluster.yaml` file based on your Kubernetes distribution. 6. Wait for vCluster to become ready by monitoring the vCluster custom resource status: Modify the following with your specific values to generate a copyable command: CLUSTER NAMESPACEteam-x CLUSTER NAMEmy-vcluster ``` kubectl wait --for=condition=ready vcluster -n team-x my-vcluster --timeout=300s ``` #### 6. Connect to vCluster[​](#6-connect-to-vcluster "Direct link to 6. Connect to vCluster") Establish a connection to your vCluster instance: ``` vcluster connect ${VCLUSTER_NAME} ``` #### 7. Create Example Workload[​](#7-create-example-workload "Direct link to 7. Create Example Workload") Create a file named `gcs-list-buckets-deployment.yaml` with the following content: ``` --- apiVersion: v1 kind: ServiceAccount metadata: name: demo-sa namespace: default --- apiVersion: apps/v1 kind: Deployment metadata: name: gcs-list-buckets namespace: default spec: replicas: 1 selector: matchLabels: app: gcs-list-buckets template: metadata: labels: app: gcs-list-buckets spec: serviceAccountName: demo-sa containers: - image: google/cloud-sdk:slim command: - "gcloud" - "storage" - "buckets" - "list" name: gcs-pod ``` Apply the deployment: ``` kubectl apply -f gcs-list-buckets-deployment.yaml ``` #### 8. Create a GCP Service Account[​](#8-create-a-gcp-service-account "Direct link to 8. Create a GCP Service Account") Create a new GCP Service Account (SA) that will be used for workload identity. ``` # Create a new GCP Service Account gcloud iam service-accounts create ${GSA_NAME} \ --display-name "Workload Identity experiment SA" ``` #### 9. Bind IAM Policies to the GCP Service Account[​](#9-bind-iam-policies-to-the-gcp-service-account "Direct link to 9. Bind IAM Policies to the GCP Service Account") Bind the necessary IAM policies to the GCP Service Account, allowing it to be used by the Kubernetes Service Account and to list GCS buckets. Namespace configuration The namespace parameter depends on your vCluster deployment type: * **vCluster** (not using the platform): Use the namespace where vCluster is deployed * **Platform-managed vCluster**: The namespace follows the pattern `loft--v-` - vCluster without the Platform - Platform-managed vCluster For vCluster deployments not connected with the platform (i.e. deployed with `vcluster create` or Helm without the platform): ``` # Set the namespace where vCluster is deployed export VCLUSTER_NAMESPACE="team-x" # Replace with your actual namespace # Bind IAM policy for Workload Identity User gcloud iam service-accounts add-iam-policy-binding ${GSA_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com \ --member "serviceAccount:${GCP_PROJECT_ID}.svc.id.goog[${VCLUSTER_NAMESPACE}/${KSA_NAME}]" \ --role "roles/iam.workloadIdentityUser" # Bind IAM policy for Storage Object Viewer gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \ --member "serviceAccount:${GSA_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com" \ --role "roles/storage.objectViewer" # Add IAM policy binding to the GCP project gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \ --member "serviceAccount:${GSA_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com" \ --role "roles/editor" ``` For vCluster deployments managed by the platform with a project: ``` # Set project name and construct the namespace export PROJECT_NAME="my-project" # Replace with your actual project name export PLATFORM_NAMESPACE="loft-${PROJECT_NAME}-v-${VCLUSTER_NAME}" # Bind IAM policy for Workload Identity User gcloud iam service-accounts add-iam-policy-binding ${GSA_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com \ --member "serviceAccount:${GCP_PROJECT_ID}.svc.id.goog[${PLATFORM_NAMESPACE}/${KSA_NAME}]" \ --role "roles/iam.workloadIdentityUser" # Bind IAM policy for Storage Object Viewer gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \ --member "serviceAccount:${GSA_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com" \ --role "roles/storage.objectViewer" # Add IAM policy binding to the GCP project gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \ --member "serviceAccount:${GSA_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com" \ --role "roles/editor" ``` #### 10. Annotate the Kubernetes Service Account[​](#10-annotate-the-kubernetes-service-account "Direct link to 10. Annotate the Kubernetes Service Account") Annotate the Kubernetes Service Account with the GCP Service Account email. ``` # Annotate the Kubernetes Service Account kubectl annotate serviceaccount \ --namespace ${VCLUSTER_KSA_NAMESPACE} \ ${VCLUSTER_KSA_NAME} \ iam.gke.io/gcp-service-account=${GSA_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com ``` #### 11. Verify the Setup[​](#11-verify-the-setup "Direct link to 11. Verify the Setup") Verify that the setup is complete and the pod is able to list the GCS buckets using the IAM role. ``` kubectl logs -l app=gcs-list-buckets -n default ``` Following these steps will integrate your GCP Service Account with your vCluster, allowing you to manage resources securely and efficiently. --- # Install the Rancher integration in vCluster The vCluster Rancher integration uses an operator to connect tenant clusters to Rancher by creating a corresponding Rancher cluster resource for each tenant cluster. The operator runs on the Rancher local cluster and handles registration automatically. Rancher users don't need cluster management permissions to use this integration. The operator assigns Rancher cluster owner access to the tenant cluster creator, owners of the tenant cluster's project, and owners of the cluster where the tenant cluster is deployed. This setup allows tenant clusters to appear and behave like managed clusters in Rancher. This guide explains how to install the vCluster Rancher Operator. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") Ensure you have the following before installing the vCluster Rancher Operator: * A Rancher instance installed in a cluster. Rancher refers to this cluster as the **local cluster**. Note that the Docker installation method is not supported for production use. * Admin permissions for the Rancher local cluster. You must be able to add Helm repositories and install charts there. * At least one connected downstream cluster where tenant clusters are deployed. ### Network policies[​](#network-policies "Direct link to Network policies") If you are using [network policies](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/policies/network-policy.md), Rancher Operator traffic into the tenant cluster control plane must be allowed. vcluster.yaml ``` policies: networkPolicy: controlPlane: ingress: # Allow ingress traffic from the rancher-cluster-agent-install job pods. - from: - podSelector: matchExpressions: - key: batch.kubernetes.io/job-name operator: Exists workload: egress: # Allow workload egress traffic to the rancher server. # This example is allowing outgoing traffic to any address. # Depending on your use case, a more restrictive policy may be used. - to: - ipBlock: cidr: 0.0.0.0/0 ``` ## Install the vCluster Rancher operator[​](#install-the-vcluster-rancher-operator "Direct link to Install the vCluster Rancher operator") 1. In the Rancher UI, select the **local cluster** from the cluster overview. 2. In the sidebar, navigate to **Apps → Repositories** and click **Create**. 3. Set a repository name and use the following Index URL: `https://charts.loft.sh` 4. **\[Optional]** If pre-release charts are needed, go to **User Preferences**, scroll to **Charts**, and enable **Include Prerelease Versions**. 5. In the sidebar, navigate to **Apps → Charts** and search for **vCluster Rancher Operator**. 6. Select the chart and follow the prompts to install it in the **local cluster**. 7. After installation, go to **Workloads → Deployments** and confirm the operator deployment is active. After it is installed, the operator automatically registers any vCluster in Rancher. It creates a corresponding Rancher cluster, sets up the connection, and grants cluster owner permissions to relevant users. To update the tenant cluster or change the vcluster.yaml, follow the same method used for installation. For example, if you used the Apps interface described above, update the application in the Rancher UI. ## Additional operator configuration[​](#additional-operator-configuration "Direct link to Additional operator configuration") ### Sync labels[​](#sync-labels "Direct link to Sync labels") Defining `syncLabels` in the operator's Helm values configures the syncing of labels from the vCluster's Service (which has the label `app=vcluster`) to the Rancher management cluster. For example: ``` syncLabels: excludeKeys: - no-sync/* includeKeys: - some-specific-label - labels-starting-with* ``` #### Fleet workspaces[​](#fleet-workspaces "Direct link to Fleet workspaces") The default fleet workspace is `fleet-default`, but you can override it by setting `fleet.defaultWorkspace`. For example: ``` fleet: defaultWorkspace: some-other-ws ``` You can also map projects to workspaces using the Kubernetes UID for the Rancher project: ``` fleet: projectUIDToWorkspaceMappings: a8732c55-e618-42dc-8f75-0964b0180a79: ws-1 98abcc55-8341-c4dc-5531-902348b80321: ws-2 ``` ## Use the vCluster Rancher extension UI[​](#use-the-vcluster-rancher-extension-ui "Direct link to Use the vCluster Rancher extension UI") Optionally, you can install the vCluster Rancher Extension UI to enable a more tailored user experience in Rancher. The vCluster Rancher Extension UI allows you to deploy tenant clusters directly from the Rancher user interface. It provides a separate UI for managing tenant clusters and labels them in the Rancher Cluster Dashboard so you can distinguish them from physical clusters. Requirements Ensure your environment meets the following version requirements: * Rancher version ≥ 2.10 * Rancher UI Extensions version ≥ 3.0 and < 4.0 * Node.js version ≥ 20 1. Open Rancher and go to **Extensions** in the left navigation. Click the ellipsis menu in the top-right corner and select **Manage Repositories**. 2. Click **Create**, enter a name for the repository, and set the **Index URL** to: ``` https://charts.loft.sh/ ``` Click **Create** to save the repository. 3. Navigate to the **Extensions** page and click to the **Available** tab. Find **vCluster Rancher Extension UI** and click **Install**. 4. There are multiple entry points to create a tenant cluster: * From the Rancher home page: click **Create vCluster** * From the **Cluster Management** view: click **Create Cluster** * From the **vCluster List** (left menu): click **Create vCluster** 5. Use the dropdown list to select the downstream cluster where the tenant cluster is deployed. If the vCluster chart from the loft.sh repository is not available in the target cluster, Rancher prompts you to add it: * Select **Automatic** to let Rancher configure it. * Or select **Manual**, click **Create**, and enter: ``` Index URL: https://charts.loft.sh/ ``` 6. Choose a vCluster chart version. Set the **namespace** and **name** for the new tenant cluster. Then click **Create vCluster** to deploy. 7. Navigate to **vCluster List** in the side menu. The new tenant cluster is displayed as a downstream cluster in Rancher. 8. Click the tenant cluster in the list to access it in the **Cluster Explorer**. From here, you can manage workloads, namespaces, and configuration as you would with any other cluster. ### Upgrade the vCluster Rancher extension UI[​](#upgrade-the-vcluster-rancher-extension-ui "Direct link to Upgrade the vCluster Rancher extension UI") To upgrade the extension to a newer version, follow the [Rancher extension update guide](https://ranchermanager.docs.rancher.com/integrations-in-rancher/rancher-extensions#updating-and-upgrading-extensions). ## Remove virtual node resources[​](#remove-virtual-node-resources "Direct link to Remove virtual node resources") Since tenant clusters share physical nodes with the downstream cluster they run on, Rancher may count tenant cluster nodes toward your resource usage. To prevent this, use sync patches to zero out the CPU capacity reported by the tenant cluster's nodes. Add the following to your `vcluster.yaml` to zero out node CPU capacity: vcluster.yaml ``` sync: fromHost: nodes: enabled: true patches: - path: status.capacity.cpu expression: '"0"' ``` To also hide CPU resources for the cluster in the Rancher cluster overview, patch allocatable CPU as well: vcluster.yaml ``` sync: fromHost: nodes: enabled: true patches: - path: status.capacity.cpu expression: '"0"' - path: status.allocatable.cpu expression: '"0"' ``` note This approach requires `sync.fromHost.nodes.enabled: true` and is not compatible with the [virtual scheduler](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/virtual-scheduler.md) or [hybrid scheduling](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/core/pods/hybrid-scheduling.md), both of which rely on accurate node capacity information for scheduling decisions. ## Uninstall the vCluster Rancher operator[​](#uninstall-the-vcluster-rancher-operator "Direct link to Uninstall the vCluster Rancher operator") To remove the Rancher integration, uninstall the vCluster Rancher Operator from the local cluster. Deleting the operator stops new tenant clusters from appearing in Rancher, but does not remove any existing tenant clusters. 1. In the Rancher UI, select the **local cluster**. 2. In the sidebar, navigate to **Apps → Installed Apps**. 3. Locate the vCluster Rancher Operator chart and delete it. --- Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** # Configure NVIDIA KAI scheduler with vCluster This guide explains how to configure vCluster to work with the [NVIDIA KAI scheduler](https://github.com/NVIDIA/KAI-Scheduler). KAI is a Kubernetes scheduler that optimizes resource allocation for both GPU and CPU workloads, with advanced queuing, fractional GPU allocation, and topology awareness. NVIDIA KAI scheduler background NVIDIA KAI scheduler is the open source version of NVIDIA Run:ai's scheduler technology, which is why it uses `podgroups.scheduling.run.ai` CRDs. KAI was open sourced by NVIDIA in 2025 under the Apache 2.0 license. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Administrator access to a Kubernetes cluster: See [Accessing Clusters with kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for more information. Run the command `kubectl auth can-i create clusterrole -A` to verify that your current kube-context has administrative privileges. info To obtain a kube-context with admin access, ensure you have the necessary credentials and permissions for your Kubernetes cluster. This typically involves using `kubectl config` commands or authenticating through your cloud provider's CLI tools. * `helm`: Helm v3.10 is required for deploying the platform. Refer to the [Helm Installation Guide](https://Helm.sh/docs/intro/install/) if you need to install it. * `kubectl`: Kubernetes command-line tool for interacting with the cluster. See [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) for installation instructions. - KAI scheduler installed in the Control Plane Cluster ## Understand the challenge[​](#understand-the-challenge "Direct link to Understand the challenge") By default, when syncing workload pods from a vCluster to the control plane cluster, vCluster sets ownership references on the synced pods. When using custom schedulers like KAI, this causes issues because: * KAI's pod-grouper controller watches for pods with `schedulerName: kai-scheduler` * The pod-grouper controller traverses the owner references to group related pods * For vCluster workloads, the owner reference is set to the vCluster service in the host namespace * The pod-grouper may not have sufficient permissions to access these Service resources ## Solution: Configure vCluster for KAI scheduler[​](#solution-configure-vCluster-for-kai "Direct link to Solution: Configure vCluster for KAI scheduler") To properly integrate vCluster with KAI scheduler, you need to configure one thing: **Turn off owner references** to allow KAI's pod-grouper to process pods properly. Add the following to your vCluster configuration (`values.yaml` for Helm or vCluster config): vCluster configuration for KAI scheduler ``` experimental: syncSettings: setOwner: false ``` This configuration prevents owner reference traversal issues by not setting them on synced pods. The KAI scheduler's pod-grouper controller automatically creates PodGroup resources in the control plane cluster. Apply this configuration when creating or updating your vCluster: Create vCluster with KAI scheduler configuration ``` vcluster create my-vcluster --values kai-scheduler-values.yaml ``` Or for an existing vCluster, update the configuration: Update existing vCluster configuration ``` vcluster create --upgrade my-vcluster --values kai-scheduler-values.yaml ``` Where `kai-scheduler-values.yaml` contains the configuration shown previously. ## Use KAI scheduler for vCluster workloads[​](#use-kai-scheduler-for-workloads "Direct link to Use KAI scheduler for vCluster workloads") After applying this configuration, you can use the KAI scheduler for your vCluster workloads by specifying `schedulerName: kai-scheduler` in the pod specification. ### Example: CPU-only workload[​](#example-cpu-only-workload "Direct link to Example: CPU-only workload") Here's an example of a CPU-only pod using the KAI scheduler: CPU-only pod with KAI scheduler ``` apiVersion: v1 kind: Pod metadata: name: cpu-only-pod spec: schedulerName: kai-scheduler containers: - name: main image: ubuntu args: - sleep - infinity resources: requests: cpu: 100m memory: 250M ``` ### Example: GPU workload[​](#example-gpu-workload "Direct link to Example: GPU workload") Here's an example of a GPU pod using the KAI scheduler: GPU pod with KAI scheduler ``` apiVersion: v1 kind: Pod metadata: name: gpu-pod spec: schedulerName: kai-scheduler containers: - name: main image: ubuntu command: - /bin/bash - -c - nvidia-smi && sleep infinity resources: requests: nvidia.com/gpu: '1' limits: nvidia.com/gpu: '1' ``` ### Example: Fractional GPU workload[​](#example-fractional-gpu-workload "Direct link to Example: Fractional GPU workload") KAI scheduler supports GPU sharing through fractional allocation. Here's an example using half a GPU: Fractional GPU pod with KAI scheduler ``` apiVersion: v1 kind: Pod metadata: name: gpu-sharing-pod labels: kai.scheduler/queue: test # Optional: specify queue annotations: gpu-fraction: "0.5" # Request half of a GPU spec: schedulerName: kai-scheduler containers: - name: main image: ubuntu args: ["sleep", "infinity"] ``` GPU sharing notes * GPU sharing must be enabled during KAI installation with `--set "global.gpuSharing=true"` * KAI doesn't enforce memory isolation - applications should respect their allocation * You can also use `gpu-memory: "2000"` annotation to request specific MiB ## Verify configuration[​](#verify-configuration "Direct link to Verify configuration") To verify that the KAI scheduler is properly configured with vCluster: * Confirm vCluster is using `experimental.syncSettings.setOwner=false` in its configuration * Check that pods are being scheduled by KAI scheduler: Verify pod is using KAI scheduler ``` kubectl get pod -n -o jsonpath='{.spec.schedulerName}' ``` * Ensure the KAI scheduler components are running: Check KAI scheduler components ``` kubectl get pods -n kai-scheduler ``` Scheduler behavior When using the KAI scheduler with vCluster and `setOwner: false`, you may observe: 1. The pod-grouper adds a podgroup annotation to your pod: ``` Annotations: pod-group-name: pg-pod-name-[uuid] ``` 2. You might see a non-blocking message in the logs: ``` Detected pod with no owner but with podgroup annotation ``` This is expected and doesn't affect scheduling. The `setOwner: false` configuration successfully resolves owner reference issues and allows the KAI scheduler to work properly with vCluster workloads in production environments. ## How it works[​](#how-it-works "Direct link to How it works") The KAI scheduler's pod-grouper component is a controller that: * Watches pods with `schedulerName: kai-scheduler` * Traverses the owner references to find the topmost owner * Groups related pods for optimal scheduling decisions * Applies custom scheduling logic based on workload type When vCluster syncs pods to the control plane cluster, it sets an owner reference to the vCluster service by default. By disabling this with `setOwner: false`, the pod-grouper can process the pods normally without needing to follow service references. ## Limitations and considerations[​](#limitations-and-considerations "Direct link to Limitations and considerations") * The `experimental.syncSettings.setOwner: false` configuration is marked as experimental and may change in future vCluster releases * If you have other features that rely on pod ownership in vCluster, disabling owner references may affect those features * The KAI scheduler might require additional configuration for advanced features like GPU sharing and queue management * For GPU workloads, ensure that the control plane cluster has the necessary GPU drivers and device plugins installed ## Troubleshoot common issues[​](#troubleshooting "Direct link to Troubleshoot common issues") ### Common problems[​](#common-problems "Direct link to Common problems") 1. **GPU sharing pods stuck in pending**: * Verify GPU sharing is enabled: `--set "global.gpuSharing=true"` during KAI installation * Check if reservation pods are created in `kai-resource-reservation` namespace 2. **Pod-grouper permission errors**: * With `setOwner: false`, these should not occur * If they persist, check KAI scheduler logs: `kubectl logs -n kai-scheduler -l component=scheduler` --- Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** # Clean obsolete nodes after changing the node selector ## Issue[​](#issue "Direct link to Issue") After configuring a node selector in vCluster that previously had node syncing enabled without selectors, outdated nodes remain visible in the cluster. You might encounter this issue if you observe the following after configuring a node selector in vCluster: * Nodes that no longer match your selector criteria still appear in vCluster. * These nodes persist even after vCluster restarts. * You see more nodes than expected when running `kubectl get nodes`. * Node count doesn't decrease after changing the node selector configuration. * Events and logs don't show any errors related to node syncing. * The output of `kubectl get nodes` shows nodes that shouldn't be included based on your current node selector settings. * Pods might still be scheduled to these obsolete nodes until they're manually removed. * Node labels show the original state before your node selector configuration was updated. * Resource utilization metrics continue to be reported for nodes that should no longer be visible. ## Cause[​](#cause "Direct link to Cause") When node syncing is initially enabled without a specified node selector, vCluster syncs nodes based on where pods are scheduled. If you later configure a node selector, previously synced nodes aren't automatically removed when vCluster restarts. ## Solution[​](#solution "Direct link to Solution") 1. **Switch to the vCluster context**. Ensure you're connected to your vCluster: ``` vcluster connect my-vcluster -n my-namespace ``` 2. **Identify nodes to remove**. List all nodes in your vCluster: ``` kubectl get nodes ``` Identify which nodes no longer match your current node selector configuration. 3. **Delete the obsolete nodes**. Remove each unnecessary node using the following command: ``` kubectl delete node ``` If any pods are running on these nodes, they are rescheduled to the remaining nodes. ## Prevention[​](#prevention "Direct link to Prevention") To avoid this issue in the future: * Configure node selectors before enabling node syncing if possible. * Document your node selector configuration to avoid unexpected changes. * After changing node selectors, check and clean up any remaining nodes. ## Additional resources[​](#additional-resources "Direct link to Additional resources") For more information about configuring node syncing in vCluster, see to the [node syncing documentation](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/nodes.md). --- # Resolve CoreDNS DNS timeouts and SERVFAIL errors ## Symptoms[​](#symptoms "Direct link to Symptoms") Tenant workloads see one or more of: * DNS resolution timeouts, or slower pod startup when init containers or sidecars wait on DNS lookups. * Intermittent `SERVFAIL` responses from CoreDNS. * Sustained `NXDOMAIN` for a Service the tenant API server can still resolve. Occasional `NXDOMAIN` alone isn't this: `ndots:5` search-path expansion produces it as a normal byproduct of everyday lookups. It's only a signal when it's sustained for a name that actually exists. ## Confirm it's CoreDNS, not something else[​](#confirm-its-coredns-not-something-else "Direct link to Confirm it's CoreDNS, not something else") Test DNS resolution directly from a pod in the tenant cluster: Test DNS resolution from inside the tenant cluster ``` kubectl run dns-test --rm -it --restart=Never --image=busybox:1.36 -- nslookup kubernetes.default.svc.cluster.local ``` Repeat it a few times in quick succession. Intermittent failures or slow responses point at CoreDNS capacity. Consistent, immediate failures point somewhere else, a NetworkPolicy blocking egress to CoreDNS, a misconfigured Corefile, or the CoreDNS Service having no endpoints. Check CoreDNS's own resource usage: Check CoreDNS pod resource usage ``` kubectl top pod -n kube-system | grep ^coredns ``` If `kubectl top` isn't available (no metrics-server), or you already have Prometheus scraping CoreDNS's `:9153` metrics endpoint, use the [PromQL queries](https://vcluster.com/docs/vcluster/deploy/control-plane/coredns-scaling.md#what-to-watch) instead. They're more reliable for catching CPU or memory saturation than a point-in-time snapshot. Also check for restarts, which can mean CoreDNS is being OOMKilled rather than CPU-saturated. Both produce the same symptoms, but adding replicas only helps with the CPU case: Check for OOMKilled restarts ``` kubectl get pods -n kube-system | grep ^coredns kubectl describe pod -n kube-system | grep -A3 "Last State" ``` ## Cause[​](#cause "Direct link to Cause") A single CoreDNS replica is CPU-bound at vCluster's default 1 CPU limit, and has a 170Mi memory limit regardless of replica count. High DNS query volume, most often from AI and ML workloads with bursty pod scheduling, or from search-domain expansion (`ndots:5`) multiplying the number of queries per lookup, can exceed either ceiling. See [Why does CoreDNS become a DNS bottleneck?](https://vcluster.com/docs/vcluster/understand/coredns-bottlenecks.md) for the full explanation. ## Fix[​](#fix "Direct link to Fix") ### Separate CoreDNS deployment[​](#separate-coredns-deployment "Direct link to Separate CoreDNS deployment") **If you're running the default separate CoreDNS deployment** (`controlPlane.coredns.embedded: false`), scale it. See [Scale CoreDNS for large node counts](https://vcluster.com/docs/vcluster/deploy/control-plane/coredns-scaling.md) for raising the replica count or running an autoscaler. ### Check a node-proportional autoscaler[​](#check-a-node-proportional-autoscaler "Direct link to Check a node-proportional autoscaler") **If you already run a node-proportional autoscaler**, confirm it's actually acting instead of assuming it is: Watch the autoscaler adjust replica count ``` kubectl -n kube-system get deployment coredns -w kubectl -n kube-system logs deployment/coredns-autoscaler ``` ### CoreDNS embedded[​](#coredns-embedded "Direct link to CoreDNS embedded") **If you're running embedded CoreDNS** (`controlPlane.coredns.embedded: true`), neither of those options applies. There's no `coredns` Deployment to scale. See [Why embedded CoreDNS can't scale the same way](https://vcluster.com/docs/vcluster/understand/coredns-bottlenecks.md#why-embedded-coredns-cant-scale-the-same-way) for why. Two options add capacity instead, both ordinary `vcluster.yaml` changes applied through a normal upgrade on your live tenant cluster, no redeploy needed: * Raise `controlPlane.statefulSet.resources`. It resizes the whole control-plane pod rather than CoreDNS specifically. * Raise `controlPlane.statefulSet.highAvailability.replicas`. More syncer replicas means more active CoreDNS instances, since the `kube-dns` Service points at the syncer pods (`TCP`/`UDP` 1053) when embedded is on. Switching CoreDNS from embedded to a separate Deployment can be done on a live tenant cluster without redeploying it. You may see a brief DNS gap during the switch. ## Related resources[​](#related-resources "Direct link to Related resources") * [Why does CoreDNS become a DNS bottleneck?](https://vcluster.com/docs/vcluster/understand/coredns-bottlenecks.md) * [Scale CoreDNS for large node counts](https://vcluster.com/docs/vcluster/deploy/control-plane/coredns-scaling.md) * [CoreDNS configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/coredns.md) --- # Resolve etcd NOSPACE alarm in vCluster The [etcd](https://etcd.io/) `NOSPACE` alarm signals that the etcd database inside the vCluster has used its available disk space. When this occurs, etcd fails its health checks, which causes the control plane to become unresponsive. As a result, all cluster operations—such as deploying workloads, updating resources, or managing cluster components—are blocked, and the vCluster is unusable until the issue is resolved. ## Error message[​](#error-message "Direct link to Error message") You might find the following error in the logs of your etcd pods, if the etcd has run out of storage space: etcd NOSPACE alarm ``` etcdhttp/metrics.go:86 /health error ALARM NOSPACE status-code 503 ``` Identifying an etcd `NOSPACE` alarm in vCluster When interacting with the affected vCluster using `kubectl`, API requests fail with timeout errors: ``` Error from server: etcdserver: request timed out ``` Additionally, the etcd health metrics endpoint returns a `503` status code and the following error: ``` etcdhttp/metrics.go:86 /health error ALARM NOSPACE status-code 503 ``` To verify the `NOSPACE` alarm, run the following command against the etcd instance: ``` etcdctl alarm list --endpoints=https://$ETCD_SRVNAME:2379 [...] ``` The output displays the triggered alarm: ``` memberID:XXXXX alarm:NOSPACE ``` ## Causes[​](#causes "Direct link to Causes") The `NOSPACE` alarm occurs due to two common conditions: * **Excessive etcd data growth:** A large number of objects—such as Deployments, ConfigMaps, and Secrets—can fill etcd’s storage if regular compaction is not performed. * **Synchronization conflicts:** Conflicting objects between the vCluster and Control Plane Cluster can trigger continuous sync loops. For example, a custom resource modified by the Control Plane Cluster might sync back to the vCluster repeatedly. This behavior quickly fills etcd’s backend storage. ## Solution[​](#solution "Direct link to Solution") To resolve the issue, compact, and defragment the etcd database to free up space. Then, reconfigure etcd with automatic compaction and increase its storage quota to prevent recurrence. Backing store differences The compaction process differs between deployed and embedded etcd. Choose the appropriate tab below based on your vCluster configuration. 1. **Identify if there's a syncing conflict**. Check for objects that might be caught in a sync loop: ``` kubectl -n logs | grep -i "sync" | grep -i "error" ``` If you find a problematic object, pause syncing for it in your vCluster config. 2. **Compact and defragment etcd**. * Deployed etcd * Embedded etcd - Connect to each etcd pod. Access the etcd pod using the following command: ``` kubectl -n exec -it -- sh ``` - Set environment variables. Export the etcd service name as an environment variable: ``` export ETCD_SRVNAME= ``` - Get the current revision number. Retrieve the current revision number of etcd using the following command: ``` etcdctl endpoint status --write-out json \ --endpoints=https://$ETCD_SRVNAME:2379 \ --cacert=/run/config/pki/etcd-ca.crt \ --key=/run/config/pki/etcd-peer.key \ --cert=/run/config/pki/etcd-peer.crt ``` - Compact the etcd database. Compact etcd to remove old data and free up disk space: ``` etcdctl --command-timeout=600s compact \ --endpoints=https://$ETCD_SRVNAME:2379 \ --cacert=/run/config/pki/etcd-ca.crt \ --key=/run/config/pki/etcd-peer.key \ --cert=/run/config/pki/etcd-peer.crt ``` Replace `` with the value retrieved from the previous command. - Defragment etcd. Defragment etcd to optimize disk usage and improve performance: ``` etcdctl --command-timeout=600s defrag \ --endpoints=https://$ETCD_SRVNAME:2379 \ --cacert=/run/config/pki/etcd-ca.crt \ --key=/run/config/pki/etcd-peer.key \ --cert=/run/config/pki/etcd-peer.crt ``` - Repeat for all etcd pods in your cluster. * Connect to the vCluster pod. Access the vCluster pod using the following command: ``` kubectl -n exec -it -- sh ``` * Get the current revision number. Retrieve the current revision number of embedded etcd: ``` etcdctl endpoint status --write-out json \ --endpoints=https://127.0.0.1:2379 \ --cacert=/data/pki/etcd/ca.crt \ --key=/data/pki/etcd/tls.key \ --cert=/data/pki/etcd/tls.crt ``` * Compact the etcd database. Compact etcd to remove old data and free up disk space: ``` etcdctl --command-timeout=600s compact \ --endpoints=https://127.0.0.1:2379 \ --cacert=/data/pki/etcd/ca.crt \ --key=/data/pki/etcd/tls.key \ --cert=/data/pki/etcd/tls.crt ``` Replace `` with the value retrieved from the previous command. * Defragment etcd. Defragment etcd to optimize disk usage and improve performance: ``` etcdctl --command-timeout=600s defrag \ --endpoints=https://127.0.0.1:2379 \ --cacert=/data/pki/etcd/ca.crt \ --key=/data/pki/etcd/tls.key \ --cert=/data/pki/etcd/tls.crt ``` * For HA setups with multiple replicas, repeat for each vCluster pod (e.g., `-0`, `-1`, etc.). 3. **Verify disk usage reduction**. * Deployed etcd * Embedded etcd Check that the operation freed up space: ``` etcdctl endpoint status -w table \ --endpoints=https://$ETCD_SRVNAME:2379 \ --cacert=/run/config/pki/etcd-ca.crt \ --key=/run/config/pki/etcd-peer.key \ --cert=/run/config/pki/etcd-peer.crt ``` Check that the operation freed up space: ``` etcdctl endpoint status -w table \ --endpoints=https://127.0.0.1:2379 \ --cacert=/data/pki/etcd/ca.crt \ --key=/data/pki/etcd/tls.key \ --cert=/data/pki/etcd/tls.crt ``` 4. **Disarm the NOSPACE alarm**. * Deployed etcd * Embedded etcd Remove the alarm to restore normal operation: ``` etcdctl alarm disarm \ --endpoints=https://$ETCD_SRVNAME:2379 \ --cacert=/run/config/pki/etcd-ca.crt \ --key=/run/config/pki/etcd-peer.key \ --cert=/run/config/pki/etcd-peer.crt ``` Remove the alarm to restore normal operation: ``` etcdctl alarm disarm \ --endpoints=https://127.0.0.1:2379 \ --cacert=/data/pki/etcd/ca.crt \ --key=/data/pki/etcd/tls.key \ --cert=/data/pki/etcd/tls.crt ``` ## Prevention[​](#prevention "Direct link to Prevention") Update your vCluster configuration to prevent future occurrences. Use the following recommended settings to enable automatic maintenance of your etcd database: * Deployed etcd * Embedded etcd vcluster.yaml ``` controlPlane: backingStore: etcd: deploy: enabled: true statefulSet: enabled: true extraArgs: - '--auto-compaction-mode=periodic' - '--auto-compaction-retention=30m' - '--quota-backend-bytes=8589934592' # 8 GB ``` vcluster.yaml ``` controlPlane: backingStore: etcd: embedded: enabled: true extraArgs: - '--auto-compaction-mode=periodic' - '--auto-compaction-retention=30m' - '--quota-backend-bytes=8589934592' # 8GB ``` This configuration enables periodic compaction every 30 minutes and sets etcd quota to 8 GB. You can adjust parameters based on your needs. ## Verification[​](#verification "Direct link to Verification") After completing the solution steps: 1. Check that etcd pods are healthy: ``` kubectl -n get pods | grep etcd ``` 2. Verify that vCluster is functioning properly: ``` kubectl -n get pods kubectl -n logs | grep -i "alarm" ``` ## Best practices[​](#best-practices "Direct link to Best practices") To ensure optimal etcd performance in vCluster: * **Monitor etcd disk usage**: Use metrics tools to track disk usage and set up alerts for high usage levels. * **Enable automated compaction**: Configure compaction with `--auto-compaction-mode=periodic` and `--auto-compaction-retention=30m` to clean up old data. * **Size etcd storage appropriately**: Set `--quota-backend-bytes` based on usage, with a buffer for growth. * **Defragment etcd regularly**: Optimize disk usage by defragmenting etcd periodically. * **Resolve syncing conflicts**: Identify and fix syncing issues to prevent unnecessary data growth. * **Consider backing store choice**: Embedded etcd simplifies management but deployed etcd provides more control for complex scenarios. ## Certificate and path reference[​](#certificate-and-path-reference "Direct link to Certificate and path reference") | Configuration | Deployed etcd | Embedded etcd | | ------------------ | ------------------------------- | ------------------------ | | Endpoint | `https://:2379` | `https://127.0.0.1:2379` | | CA certificate | `/run/config/pki/etcd-ca.crt` | `/data/pki/etcd/ca.crt` | | Client key | `/run/config/pki/etcd-peer.key` | `/data/pki/etcd/tls.key` | | Client certificate | `/run/config/pki/etcd-peer.crt` | `/data/pki/etcd/tls.crt` | | Data directory | Separate PVC | `/data` in vCluster PVC | | Access method | Using etcd pod | Using vCluster pod | --- # Resolve Gateway API sync errors vCluster validates Gateway API references in the tenant cluster before syncing them to the control plane cluster. When a `Gateway`, `HTTPRoute`, `TLSRoute`, `ReferenceGrant`, or `BackendTLSPolicy` is misconfigured or depends on an object that has no synced host counterpart, vCluster does not sync the resource and records a Warning event on the tenant object. This page covers the most common events. For the feature overview and full examples, see [Gateway API sync](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md). ## `RefNotPermitted` on a route[​](#refnotpermitted-route "Direct link to refnotpermitted-route") ### Symptoms[​](#symptoms "Direct link to Symptoms") A tenant `HTTPRoute` or `TLSRoute` has no `Accepted` parent in its status, the host data plane returns the controller's no-route response, and `kubectl describe` on the tenant route shows: ``` Warning RefNotPermitted Gateway API reference not permitted: authorize parentRefs[0]: HTTPRoute in namespace "" is not permitted by Gateway "" in namespace "" allowedRoutes ``` ### Cause[​](#cause "Direct link to Cause") The tenant `Gateway` listener's `allowedRoutes.namespaces` does not permit the route's namespace. Either `allowedRoutes.namespaces.from: Same` is set and the route lives in a different namespace, or `from: Selector` is set and the route's namespace does not match the selector. ### Resolution[​](#resolution "Direct link to Resolution") Either move the route into a permitted namespace, label the route's namespace so it matches the listener selector, or relax the listener's `allowedRoutes.namespaces`. See [Route attachment by namespace selector](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md#route-attachment-by-namespace-selector) for the selector pattern. ## `RefNotPermitted` on a cross-namespace backend reference[​](#refnotpermitted-backend "Direct link to refnotpermitted-backend") ### Symptoms[​](#symptoms-1 "Direct link to Symptoms") A tenant `HTTPRoute` or `TLSRoute` references a Service in a different namespace, has no `Accepted` parent in its status, and `kubectl describe` shows: ``` Warning RefNotPermitted Gateway API reference not permitted: translate rules[0]: translate backendRefs[0]: HTTPRoute in namespace "" is not permitted to reference Service "" in namespace "" ``` ### Cause[​](#cause-1 "Direct link to Cause") Gateway API requires a `ReferenceGrant` in the target namespace whenever a route references a backend Service in a different namespace. vCluster enforces the same rule before sync. ### Resolution[​](#resolution-1 "Direct link to Resolution") This event means the tenant `ReferenceGrant` did not authorize the reference at the time vCluster evaluated the route. Add a matching `ReferenceGrant` to the backend namespace that names the route's namespace, group, and kind. The grant pattern is in [Backend reference via ReferenceGrant](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md#backend-reference-via-referencegrant). ## `RefNotPermitted` on a cross-namespace certificate reference[​](#refnotpermitted-cert "Direct link to refnotpermitted-cert") ### Symptoms[​](#symptoms-2 "Direct link to Symptoms") A tenant HTTPS `Gateway` references a TLS Secret in a different namespace, the listener does not become `Accepted`, and `kubectl describe` on the tenant `Gateway` shows: ``` Warning RefNotPermitted Gateway API reference not permitted: authorize listeners[0].tls.certificateRefs[0]: Gateway in namespace "" is not permitted to reference Secret "" in namespace "" ``` ### Cause[​](#cause-2 "Direct link to Cause") Same Gateway API rule as backend references: a `ReferenceGrant` in the certificate namespace must permit the `Gateway` to reference the Secret. vCluster enforces it before sync. ### Resolution[​](#resolution-2 "Direct link to Resolution") Add a `ReferenceGrant` to the certificate namespace that names the Gateway's namespace, group (`gateway.networking.k8s.io`), and kind (`Gateway`). The full pattern is in [TLS certificate via ReferenceGrant](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/networking/gateway-api.md#tls-certificate-via-referencegrant). ## `SyncError: referenced ConfigMap has no synced host object`[​](#syncerror-missing-ca-configmap "Direct link to syncerror-missing-ca-configmap") ### Symptoms[​](#symptoms-3 "Direct link to Symptoms") A `BackendTLSPolicy` has empty status, the host Gateway controller reports no policy, and `kubectl describe` shows: ``` Warning SyncError Error syncing: translate validation.caCertificateRefs[0]: referenced ConfigMap "" in namespace "" has no synced host object ``` ### Cause[​](#cause-3 "Direct link to Cause") vCluster cannot translate the `BackendTLSPolicy`'s `validation.caCertificateRefs` because the referenced ConfigMap either does not exist in the tenant cluster or is not synced to the host. The same class of error occurs for Secret references that have not been synced. ### Resolution[​](#resolution-3 "Direct link to Resolution") 1. Confirm the tenant ConfigMap exists: ``` kubectl get configmap -n ``` 2. Make sure `sync.toHost.configMaps` is enabled and includes the CA ConfigMap. Default config syncing is opt-in, so either set `sync.toHost.configMaps.all: true` or label the ConfigMap so it matches the configured selector. 3. Re-apply the `BackendTLSPolicy` and confirm the event clears. Timing tip: while the dependent Service is still being synced for the first time, the first emitted warning may name the Service instead of the CA. After the Service is synced, the stable error names the missing ConfigMap. ## `SyncError: unsupported parentRef or backendRef`[​](#syncerror-unsupported-ref "Direct link to syncerror-unsupported-ref") ### Symptoms[​](#symptoms-4 "Direct link to Symptoms") An `HTTPRoute` or `TLSRoute` has empty status and `kubectl describe` shows one of: ``` Warning SyncError Error syncing: translate parentRefs[0]: parentRef group "" kind "" is not supported Warning SyncError Error syncing: translate rules[0]: translate backendRefs[0]: backendRef group "" kind "" is not supported ``` ### Cause[​](#cause-4 "Direct link to Cause") vCluster's native Gateway API sync only translates core Gateway API kinds: `Gateway` for `parentRef`, and core Service for `backendRef`. Custom resource `parentRef` and `backendRef` kinds (such as Istio waypoints or service-mesh overlays) cannot be translated. ### Resolution[​](#resolution-4 "Direct link to Resolution") Either move the unsupported reference to a supported kind, or sync the custom resource separately using [`sync.toHost.customResources`](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/to-host/advanced/custom-resources.md). The custom-resource path still works for non-core Gateway API extensions; native sync is for core Gateway, HTTPRoute, TLSRoute, ReferenceGrant, and BackendTLSPolicy. ## `SyncWarning: Gateway sync skipped (GatewayClass not selected)`[​](#syncwarning-gatewayclass-not-selected "Direct link to syncwarning-gatewayclass-not-selected") ### Symptoms[​](#symptoms-5 "Direct link to Symptoms") A tenant `Gateway` is not visible on the host, its status is empty, and `kubectl describe` shows: ``` Warning SyncWarning Gateway "" sync skipped. The GatewayClass "" does not match the selector under 'sync.fromHost.gatewayClasses.selector' ``` ### Cause[​](#cause-5 "Direct link to Cause") `sync.fromHost.gatewayClasses` is enabled with a selector, and the referenced `GatewayClass` does not have the required labels in the control plane cluster. vCluster did not sync the class into the tenant cluster, so the tenant `Gateway` has no valid `gatewayClassName`. ### Resolution[​](#resolution-5 "Direct link to Resolution") 1. Confirm the host `GatewayClass` carries the expected labels: ``` kubectl --context "$HOST_CONTEXT" get gatewayclass --show-labels ``` 2. Either label the host `GatewayClass` so it matches the selector, or change the tenant `Gateway` to reference a class that does. See [Gateway classes](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync/from-host/gateway-classes.md) for the selector pattern. ## Tenant Gateway has no host address[​](#no-host-address "Direct link to Tenant Gateway has no host address") ### Symptoms[​](#symptoms-6 "Direct link to Symptoms") The tenant `Gateway` reports `Accepted=True` but `status.addresses` stays empty, and `curl` to a discovered address fails. ### Cause[​](#cause-6 "Direct link to Cause") The host Gateway controller has not yet assigned an address, or the host platform does not provide one. For some controllers, an address only appears once a corresponding LoadBalancer Service is provisioned by the cloud provider. On clusters without LoadBalancer support, the host controller may publish a NodePort or require manual port-forwarding to the controller's Envoy/data-plane Service. ### Resolution[​](#resolution-6 "Direct link to Resolution") 1. Inspect the host Gateway and its generated Service: ``` kubectl --context "$HOST_CONTEXT" -n get gateway -o wide kubectl --context "$HOST_CONTEXT" -n get svc ``` 2. If the host platform does not provide LoadBalancer addresses, port-forward the controller's data-plane Service and curl that address directly. The tenant `Gateway` only mirrors what the host controller publishes; vCluster does not synthesize an address. --- # Resolve kubectl port-forward issues with Ingress When the vCluster API server is exposed through an Ingress or LoadBalancer, `kubectl port-forward` commands may fail with connection upgrade errors, even though the network connectivity and firewall configuration are correct. This issue is characterized by the following symptoms: * `kubectl port-forward` fails with "error upgrading connection" messages when using the vCluster kubeconfig * Other kubectl commands work normally (like `kubectl get pods`) * Direct connection to the vCluster using cluster IP works * Network connectivity and firewall rules are verified to be correct * The error occurs specifically when accessing vCluster through the Ingress hostname or LoadBalancer endpoint ## Error message[​](#error-message "Direct link to Error message") Common error messages include: kubectl port-forward connection upgrade error ``` error: error upgrading connection: unable to upgrade connection: Upgrade request required ``` Alternative error message ``` error: error upgrading connection: websocket: bad handshake ``` ## Cause[​](#cause "Direct link to Cause") This issue occurs because `kubectl port-forward` requires HTTP protocol upgrade capabilities (SPDY or WebSocket) to establish its tunneled connection. Some ingress controllers and load balancers don't properly support these upgrade mechanisms or strip the necessary headers. The root causes include: * **HTTP/2 and WebSocket protocol limitations**: Some load balancers don't properly handle the HTTP upgrade requests that `kubectl port-forward` requires * **Missing upgrade header support**: Ingress controllers may strip or fail to forward the `Upgrade: SPDY` header * **SSL/TLS termination issues**: When SSL is terminated at the ingress level, the backend connection may not support the required upgrade protocols * **Load balancer type restrictions**: Some load balancers such as AWS Classic Load Balancer have limited support for HTTP protocol upgrade connections ## Solution[​](#solution "Direct link to Solution") The most reliable solution is to connect to vCluster using the vCluster CLI: ``` vcluster connect -n # Now port-forward works normally within the vCluster context kubectl port-forward service/ : ``` The vCluster CLI uses different connection methods (direct exposure, background proxy, or port-forwarding) and updates the kubeconfig to point to a local endpoint, bypassing the ingress layer for all kubectl operations within the vCluster. ## Alternative solutions[​](#alternative-solutions "Direct link to Alternative solutions") If the vCluster CLI is not available or you need to maintain ingress-level access: ### Fix ingress configuration[​](#fix-ingress-configuration "Direct link to Fix ingress configuration") 1. **Enable WebSocket support in your ingress controller**. Configure your ingress controller to support WebSocket and HTTP upgrade headers. Most modern ingress controllers support this, but it may need to be explicitly enabled. Consult your ingress controller's documentation for WebSocket/HTTP upgrade configuration options. 2. **Configure load balancer for protocol upgrade support** (if applicable). If using cloud load balancers, ensure they support HTTP protocol upgrades (WebSocket/SPDY). Some older or basic load balancers may not properly handle these upgrade mechanisms. Check your cloud provider's documentation for WebSocket/HTTP upgrade support in their load balancing services. 3. **Check ingress controller logs**. Review your ingress controller logs for upgrade-related errors: ``` # Check logs for errors related to "upgrade", "websocket", or "SPDY" kubectl logs -n ``` Look for error messages indicating WebSocket or HTTP upgrade failures. ## Verification[​](#verification "Direct link to Verification") After implementing the solution: 1. **Verify vCluster connection**: ``` # Connect to vCluster vcluster connect # Confirm you're connected to vCluster context kubectl config current-context ``` 2. **Test port-forward capability**: ``` # Try port-forwarding to a pod or service in vCluster kubectl port-forward service/ : # Or test with a pod kubectl port-forward pod/ : ``` 3. **Confirm no connection upgrade errors**: * The port-forward command should establish successfully without "error upgrading connection" messages * You should be able to access the forwarded service at `localhost:` * The connection should remain stable ## Best practices[​](#best-practices "Direct link to Best practices") * **Use vCluster CLI for routine access** instead of relying on port-forward through ingress * **Use load balancers that support HTTP protocol upgrades** when exposing vCluster externally * **Enable WebSocket support** in your ingress controller configuration when exposing vCluster * **Monitor ingress controller logs** for upgrade-related issues Ensure your ingress controller and load balancer properly support HTTP protocol upgrades, or use the vCluster CLI for the most reliable connection. --- # Resolve permission denied errors with noexec volumes When deploying vCluster in Kubernetes environments where emptyDir volumes are mounted with `noexec` flag, the vCluster pod fails to start with permission denied errors. This occurs because vCluster uses init containers to copy Kubernetes binaries into emptyDir volumes, and these binaries cannot be executed when the volume is mounted with `noexec`. ## Error messages[​](#error-messages "Direct link to Error messages") The following error messages indicate this issue: Container runtime error ``` Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/binaries/vcluster": permission denied: unknown ``` When debugging with strace ``` / # /binaries/vcluster sh: /binaries/vcluster: Permission denied / # strace /binaries/vcluster execve("/binaries/vcluster", ["/binaries/vcluster"], [/* 27 vars */]) = -1 EACCES (Permission denied) ``` ## Cause[​](#cause "Direct link to Cause") vCluster uses init containers to copy the vCluster binary and Kubernetes components (kube-apiserver, kube-controller-manager) into emptyDir volumes. The main container then attempts to execute these binaries. When emptyDir volumes are mounted with `noexec`, the binaries cannot be executed. This happens in: * Security-hardened Kubernetes environments * Clusters where the host filesystem for emptyDir has `noexec` mount option * Environments with security policies enforcing `noexec` on temporary storage ## Solution[​](#solution "Direct link to Solution") ### Use pre-built images with embedded binaries[​](#use-pre-built-images-with-embedded-binaries "Direct link to Use pre-built images with embedded binaries") The most reliable solution is to use vCluster images that have all required binaries embedded directly in the container image, eliminating the need for emptyDir volumes. Temporary workaround available A temporary workaround using pre-built images is available: * GitHub workflow: * Pre-built images: * Helm charts: Add alternative Helm repository and deploy ``` # Add the alternative Helm repository helm repo add vcluster-prebuilt https://loft-demos.github.io/vcluster-charts/ helm repo update # Deploy vCluster with pre-built images helm install my-vcluster vcluster-prebuilt/vcluster \ --namespace vcluster-namespace \ --create-namespace ``` Or specify the image directly in your values file: vcluster.yaml ``` # Use pre-built image with embedded binaries controlPlane: distro: k8s: image: repository: ghcr.io/loft-demos/vcluster-pro tag: -noexec ``` ## Alternative solutions[​](#alternative-solutions "Direct link to Alternative solutions") If pre-built images aren't available or suitable: ### Configure volume mounts without noexec[​](#configure-volume-mounts-without-noexec "Direct link to Configure volume mounts without noexec") If you have cluster admin access, modify the mount options for the emptyDir base directory: Remount with exec permissions ``` # On the Kubernetes node mount -o remount,exec /var/lib/kubelet ``` ### Use persistent volumes[​](#use-persistent-volumes "Direct link to Use persistent volumes") Configure vCluster to use persistent storage instead of emptyDir: vcluster.yaml ``` controlPlane: statefulSet: persistence: volumeClaim: enabled: true size: 10Gi ``` ### Deploy to specific nodes[​](#deploy-to-specific-nodes "Direct link to Deploy to specific nodes") Deploy to nodes that allow exec on emptyDir: vcluster.yaml ``` controlPlane: statefulSet: scheduling: nodeSelector: vcluster-compatible: "true" ``` ## Verification[​](#verification "Direct link to Verification") After implementing a solution: Verify vCluster deployment ``` # Check pod status kubectl get pods -n # Check logs for permission errors kubectl logs -n -c syncer # Connect to vCluster vcluster connect -n kubectl get nodes ``` ## Known limitations[​](#known-limitations "Direct link to Known limitations") * **No direct upgrade path**: Cannot upgrade from standard deployment to pre-built images without recreating vCluster * **Image size**: Pre-built images are larger than standard images * **Maintenance overhead**: Custom images require additional maintenance ## Best practices[​](#best-practices "Direct link to Best practices") * Check cluster policies before deploying vCluster * Use pre-built images in security-hardened environments * Test in staging environments matching production security policies ## Related resources[​](#related-resources "Direct link to Related resources") * [Deploy in rootless mode](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/rootless-mode.md) * [Air-gapped deployments](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/air-gapped.md) * [Security hardening guide](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/security/hardening-guide/overview.md) --- # Resolve pods stuck at Ready=False with control plane and tenant version skew This is fixed in vCluster 0.37.0 and later. On earlier versions, synced pods can get stuck reporting `Ready=False`, even though the pod itself is running. This happens when the tenant cluster runs Kubernetes 1.33 or earlier while the control plane cluster runs Kubernetes 1.34 or later. Deployments and StatefulSets never show as ready, because they wait on that pod condition. Upgrade to vCluster 0.37.0 or later to resolve the issue. The rest of this page describes the problem for tenant clusters still running an earlier version. ## Symptoms[​](#symptoms "Direct link to Symptoms") A pod's containers are all `Running`, but its owning Deployment or StatefulSet never reaches the desired ready count: Pod is Running, but the Deployment is not Ready ``` kubectl get pods # NAME READY STATUS RESTARTS AGE # my-app-7cf4ddcbbf-k5m2n 1/1 Running 0 127m kubectl get deployments # NAME READY UP-TO-DATE AVAILABLE AGE # my-app 0/1 1 0 127m ``` The [syncer](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync) logs a reconcile error for the pod, similar to: Syncer log ``` ERROR controller/controller.go:474 Reconciler error {"component": "vcluster", "controller": "pod", ..., "error": "sync: patch host object: update object status: Pod \"my-app-7cf4ddcbbf-k5m2n--b21ec34921\" is invalid: status.qosClass: Invalid value: \"BestEffort\": field is immutable"} ``` Restarting the control plane's [API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) pod can temporarily clear some of these errors, but they come back after creating or updating deployments. ## Cause[​](#cause "Direct link to Cause") This is a known issue tracked in [vCluster GitHub Issue #3578](https://github.com/loft-sh/vcluster/issues/3578), with two contributing root causes: * **QoS class immutability**: the pod syncer copies the tenant pod's QoS class onto the host object before patching. Kubernetes 1.32 and later rejects that patch because `status.qosClass` is immutable once set. * **`observedGeneration` mismatch**: starting in Kubernetes 1.34, the host kubelet sets `observedGeneration` on pod status and conditions. A tenant API server on Kubernetes older than 1.34 drops that field on write, since `PodObservedGenerationTracking` is alpha or off by default there. The syncer then sees the object cache and the informer disagree on every reconcile, reads that as a real condition change, and keeps overwriting the tenant's `Ready=True` condition with stale data. Both root causes only trigger when the tenant cluster runs Kubernetes 1.33 or earlier and the control plane cluster runs Kubernetes 1.34 or later. See the [Kubernetes compatibility matrix](https://vcluster.com/docs/vcluster/manage/upgrade/supported_versions.md#kubernetes-compatibility-matrix) for the specific version combinations flagged with this known issue. ## Workaround for vCluster versions earlier than 0.37.0[​](#workaround-for-vcluster-versions-earlier-than-0370 "Direct link to Workaround for vCluster versions earlier than 0.37.0") Upgrading to vCluster 0.37.0 or later is the fix. If you can't upgrade immediately, restarting the control plane's API server pod clears the immediate errors, but they recur as soon as new pods sync. The most effective mitigation until you can upgrade is to keep the tenant cluster on Kubernetes 1.34 or later. Apply this whenever the control plane cluster runs Kubernetes 1.34 or later. ## Fix status[​](#fix-status "Direct link to Fix status") Fixed in vCluster 0.37.0 by [vcluster#4037](https://github.com/loft-sh/vcluster/pull/4037). The fix stops the syncer from writing the immutable QoS class to the host, and it ignores `observedGeneration` when the tenant cluster doesn't persist it. ## Related resources[​](#related-resources "Direct link to Related resources") * [Kubernetes compatibility matrix](https://vcluster.com/docs/vcluster/manage/upgrade/supported_versions.md#kubernetes-compatibility-matrix) * [vcluster#3578](https://github.com/loft-sh/vcluster/issues/3578) * [vcluster#4037](https://github.com/loft-sh/vcluster/pull/4037) --- # Resolve Pro feature license errors When a tenant cluster requests a Pro feature such as DRA sync, the control plane can crashloop at startup with an error that names a "vCluster pro feature". The wording reads like a paywall, but it is almost always an order-of-operations problem. The feature needs the Pro image and an attached license to be present together, and one of them is missing. Many of these features, including DRA sync, are entitled by the **Free tier** at no cost. In most cases the fix is to connect the tenant cluster to the vCluster Platform so the image receives a license, not to upgrade a plan. For how tiers and licensing work, see [Open Source vs Free tier](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md). ## Error messages[​](#error-messages "Direct link to Error messages") The control plane crashloops during syncer initialization with a message like the following: Control plane startup error ``` start managers: init syncers: create pro syncers: cannot create resource claim syncer: you are trying to use a vCluster pro feature 'DRA Sync' (dra-sync) that is not available in the open source vcluster or disabled in your license. Please use the vCluster pro image and specify a license that allows using this feature or reach out to support@loft.sh ``` Whichever Pro syncer initializes first names itself in the message, so the feature name varies. The underlying cause is the same. ## Cause[​](#cause "Direct link to Cause") A Pro feature needs two things present together. Missing either one produces this error. 1. **The Pro image** (`ghcr.io/loft-sh/vcluster-pro`). The open source image (`ghcr.io/loft-sh/vcluster-oss`) does not compile the Pro syncers at all, so requesting a Pro feature on the OSS image is a no-op rather than a crash. For the available image builds, see [vCluster image options](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/deployment/statefulset.md#vcluster-image-options). 2. **An attached license** that enables the feature. The Pro image reads the license from the `vcluster-platform-api-key` Secret in the tenant cluster's namespace. The `vcluster-platform-api-key` Secret is the unlock. It is created automatically when the tenant cluster is provisioned through the vCluster Platform, either as a VirtualClusterInstance or through the Platform UI. A tenant cluster created standalone with `vcluster create` or a bare `helm install`, and never connected to the Platform, has no such Secret. The Pro binary then has no license to read, and it crashloops with the error above. This is the combination that triggers the error. The Pro image is present, the feature is requested, but no Secret exists to license it. Crashloop versus graceful degradation A tenant cluster that was previously licensed and then loses Platform connectivity falls back to the OSS feature set and keeps running. That graceful degradation does not apply here. A Pro image that never had a license Secret at all crashloops at startup instead of falling back. ## Solution[​](#solution "Direct link to Solution") You can license an already-running tenant cluster in place. No recreation and no data loss are required. If the tenant cluster is still on the OSS image, switch it to the Pro image with `helm upgrade` first, setting `controlPlane.statefulSet.image.repository` to `ghcr.io/loft-sh/vcluster-pro`. Then add the tenant cluster to the Platform, which creates the Secret and restarts the control plane so the Pro binary reads the license. Modify the following with your specific values to generate a copyable command: NAMEmy-vcluster NAMESPACEmy-namespace PROJECTmy-project Add an existing tenant cluster to the Platform ``` vcluster platform login vcluster platform add vcluster my-vcluster --namespace my-namespace --project my-project --restart ``` `--restart` is the default and restarts the control plane so it picks up the new license. For the full workflow, see [Add existing tenant clusters](https://vcluster.com/docs/platform/use-platform/virtual-clusters/add-virtual-clusters.md). Log in with the Platform's external URL `vcluster platform add vcluster` records whatever URL the CLI logged in with into the Secret. Logging in through a port-forward (`localhost:`) writes an address the in-cluster pod cannot reach, so license validation fails. Log in with the Platform's external URL so the in-cluster pod can connect. ## Verification[​](#verification "Direct link to Verification") After adding the tenant cluster, confirm the control plane is running and licensed. Modify the following with your specific values to generate a copyable command: NAMESPACEmy-namespace Verify the tenant cluster is licensed ``` # Confirm the control plane is no longer crashlooping kubectl get pods -n my-namespace # Confirm the license Secret now exists kubectl get secret vcluster-platform-api-key -n my-namespace # Confirm the tenant cluster shows a license status in the Platform vcluster platform list vclusters ``` The control plane logs the enabled features at startup once the license is attached. Startup log on a licensed control plane ``` loader/loader.go Using online license with host loft.vcluster-platform online/license.go Enabled features: online/license.go dra-sync ``` ## Related resources[​](#related-resources "Direct link to Related resources") * [Open Source vs Free tier](https://vcluster.com/docs/vcluster/introduction/oss-vs-free.md) * [vCluster image options](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/deployment/statefulset.md#vcluster-image-options) * [Platform API key configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/platform.md) * [Add existing tenant clusters](https://vcluster.com/docs/platform/use-platform/virtual-clusters/add-virtual-clusters.md) --- Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** # Resolve ingress update conflicts with Rancher When deploying the Rancher agent in both vCluster and the control plane cluster with a single shared ingress, both agents repeatedly update the ingress object. This causes excessive load on the database and API server. You might observe the following signs of ingress update conflicts: * Endless updates to ingress resources in the control plane cluster * Rapid growth of the API server database (may reach several GB) * API server slowdown (simple commands like `kubectl get pods` take several seconds) ## Cause[​](#cause "Direct link to Cause") This issue occurs due to a conflict between Rancher agents deployed in both vCluster and the control plane clusters. Both agents attempt to update annotations on the ingress using different naming conventions, creating a continuous update loop. ## Solution[​](#solution "Direct link to Solution") This issue is resolved in vCluster version 0.19.8 and later. To fix the problem: 1. Upgrade your vCluster installation to version 0.19.8 or later 2. If your API server is slow, compact your etcd database to reduce its size (specific to your Kubernetes distribution) ### Upgrade vCluster using Helm[​](#upgrade-vcluster-using-helm "Direct link to Upgrade vCluster using Helm") 1. **Check your current vCluster version**. ``` helm list -A | grep vcluster ``` 2. **Upgrade to version 0.19.8 or later**. If you installed vCluster using Helm, run the following command: ``` helm upgrade loft/vcluster --reuse-values --version 0.19.8 ``` Replace `` with the name of your vCluster release. 3. **Verify the upgrade**. ``` helm list -A | grep vcluster ``` Confirm the version is now 0.19.8 or later. ### Verify the fix[​](#verify-the-fix "Direct link to Verify the fix") After upgrading, monitor your cluster to ensure: * Ingress updates have stabilized * The database size growth has slowed or stopped * API server performance has returned to normal note You must [compact the etcd](https://vcluster.com/docs/vcluster/troubleshoot/etcd-compaction.md) database before seeing performance improvements. ### Check API server performance[​](#check-api-server-performance "Direct link to Check API server performance") Run a simple command and measure response time: ``` time kubectl get pods -A ``` The command should complete in under a second if the API server is performing normally. ## Additional resources[​](#additional-resources "Direct link to Additional resources") For more details about this fix and other changes in vCluster 0.19.8, refer to the [official release notes](https://github.com/loft-sh/vcluster/releases/tag/v0.19.8). --- # Troubleshoot the resource proxy Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. vCluster Platform required This feature requires [vCluster Platform](https://vcluster.com/docs/platform/.md). Both the client and target tenant clusters must be managed as [VirtualClusterInstance](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance/.md) within the platform. The resource proxy lets a client tenant cluster transparently forward custom resource requests to a target tenant cluster, which stores the objects and can run controllers against them. When the target, the platform network, or the proxy itself degrades, callers see symptoms like 503s, stale discovery, or unexpected 404s instead of a clear cause. Start with the quick triage to decide whether the proxy or the target is at fault, then jump to the matching symptom playbook. For how the proxy is built, see [How the resource proxy works](https://vcluster.com/docs/vcluster/understand/resource-proxy.md). ## Quick triage: proxy degraded or target down[​](#quick-triage "Direct link to Quick triage: proxy degraded or target down") **Step 1: Is the target actually up?** Check the target tenant cluster directly, using its own kubeconfig or the platform UI. Are the control plane pods `Ready`? **Step 2: What do the proxy metrics say?** Run this against the client tenant cluster API: Check proxy metrics ``` kubectl get --raw /metrics/syncer | grep vcluster_proxy ``` See the [metrics reference](https://vcluster.com/docs/vcluster/manage/resource-proxy-operations.md#metrics-reference) for what each series means. **Step 3: What do the client logs say?** Check the client's vCluster control plane pod logs for the messages listed in the [log reference](#log-reference): Check client control plane logs ``` kubectl logs -n vcluster- -l app=vcluster --tail=100 --prefix ``` In an HA deployment, every replica runs its own proxy and keeps its own peer cache, even though only the elected leader registers the APIService. A symptom can therefore be replica-local. `--prefix` labels each line with the pod that emitted it, so check all replicas rather than concluding from one. Then read the verdict from this table: | Target pods | Metrics or logs on the client | Verdict | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Not `Ready` | Anything | The target is down. Fix the target. The proxy recovers on its own once a peer is dialable. | | `Ready` | `vcluster_proxy_peers` shows no `healthy` peers, `vcluster_proxy_target_all_peers_quarantined_total` increased recently, quarantine log lines | Proxy-path degradation: the endpoints are stale or unreachable from the client. Check peer publication and the direct and DERP network paths. See [503 errors persist after a target restart or upgrade](#stale-peers). | | `Ready` | `vcluster_proxy_peer_cache_synced` is `0`, with `peer cache did not sync in time for the startup probe` or `peer watch subscription is down` in logs | The client has no current peer snapshot because of a license, network, or platform issue. Target pod readiness doesn't validate the target API, but restore the client's peer feed first. | | `Ready` | No quarantine activity, requests still return 503 | Check the `reason` label on `vcluster_proxy_request_total`. `upstream` with code `503` usually means the target returned a 5xx response, but `upstream` is also the default for failures the proxy didn't classify more specifically. Check the client proxy logs before attributing it to the target. `timeout`, `deadline`, `peer_departed`, `ts_server_unavailable`, `error`, and `ownership_check_failed` identify failures classified during proxy handling. | | `Ready` | Requests consistently take about 60 seconds, then fail | The target accepted the connection but didn't finish the response before the proxy's request deadline. See [Requests time out after about 60 seconds](#requests-hang). | Don't classify an immediate 503 by latency alone. A peer precheck can fail immediately, but the target can also return a 5xx immediately; use the request's `reason` label to distinguish them. Failures near the 5-second dial timeout point to the peer path. Failures near the 60-second request deadline point to a slow target response path. ## Target tenant cluster returns 503 unavailable[​](#target-unavailable "Direct link to Target tenant cluster returns 503 unavailable") ### Symptoms[​](#symptoms "Direct link to Symptoms") Requests fail with `target tenant cluster "" for / is unavailable`. This is the generic degraded-path error. The proxy sets `Retry-After: 1` on unavailability. ### Cause[​](#cause "Direct link to Cause") The proxy uses the same retryable 503 status for five conditions, most common first: 1. **Target down or restarting.** No target peer accepts a connection. 2. **No peer available.** The client hasn't received a peer list from the platform since startup, or the latest list is empty. 3. **All peers quarantined.** Every known endpoint failed 3 consecutive dials. 4. **Peer list stale.** The platform peer subscription is down and the client's last peer list has expired. 5. **Target returned 5xx.** The proxy maps upstream 5xx responses to 503. ### Resolution[​](#resolution "Direct link to Resolution") Prechecks and upstream 5xx responses use the generic unavailable text. Transport failures can include the underlying dial error. Use the `reason` label on `vcluster_proxy_request_total` or the logs to identify the cause, then take the matching action: 1. **Target down or restarting.** Verify the target pods. No action is needed on the client. Recovery is automatic when a peer accepts a dial. 2. **No peer available.** Check `vcluster_proxy_peer_cache_synced`. A value of `0` means no current, trusted snapshot, while `1` with zero peers means the platform published a fresh list with no target endpoint. Check that the vCluster has platform connectivity and a valid license, that the platform is reachable from the client pod, and that the target's control plane peers are registered. The cold-start case logs `peer cache did not sync in time for the startup probe, requests fail with 503 until the first netmap arrives`. 3. **All peers quarantined.** Look for `all peer hosts for the proxy target are quarantined; requests fail fast with 503 until a peer becomes dialable`. If the target pods are `Ready`, check peer publication, NetworkPolicies, firewall rules, and DERP reachability. See [503 errors persist after a target restart or upgrade](#stale-peers). 4. **Peer list stale.** Look for `peer watch subscription is down and no fresh netmap is available; requests to the target fail with 503 until it recovers`, including a `lastNetmapAge` field. Restore client-to-platform connectivity. 5. **Target returned 5xx.** Check the target [API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/), including its backing store, webhooks, and load. The `reason` label distinguishes `no_peer`, `all_quarantined`, `stale_peers`, `upstream`, and classified failures such as `timeout`, `deadline`, `peer_departed`, `ts_server_unavailable`, `error`, or `ownership_check_failed`. See the [outcome reasons reference](https://vcluster.com/docs/vcluster/manage/resource-proxy-operations.md#outcome-reasons) for what each one means. Callers, especially controllers, should back off on 503 responses instead of retrying in a tight loop. The proxy always sets `Retry-After: 1` on unavailability, and honoring it avoids adding load to an already-degraded target. ## 503 errors persist after a target restart or upgrade[​](#stale-peers "Direct link to 503 errors persist after a target restart or upgrade") ### Symptoms[​](#symptoms-1 "Direct link to Symptoms") Requests keep returning 503 after the target's control plane pods become `Ready`. ### Cause[​](#cause-1 "Direct link to Cause") When the target restarts or upgrades, its control plane pods come back as new peers. The client can keep dialing old endpoints until the platform publishes the replacements. ### Resolution[​](#resolution-1 "Direct link to Resolution") The sequence is self-healing when the platform publishes the replacement peers and the client can reach them. Stale endpoints are quarantined (`vcluster_proxy_peer_quarantines_total` increases), requests fail fast instead of hanging, and a fresh peer list replaces stale endpoints with healthy ones. Confirm recovery with `vcluster_proxy_peers{state="healthy"}` and a successful resource request. `vcluster_proxy_peer_recoveries_total` increases only when the same tracked endpoint succeeds after failures. A new replacement endpoint doesn't increment it. If the proxy doesn't recover, check both peer publication and network reachability: platform health, client-to-platform connectivity, NetworkPolicies, firewall rules, and DERP availability. ## Requests time out after about 60 seconds[​](#requests-hang "Direct link to Requests time out after about 60 seconds") ### Symptoms[​](#symptoms-2 "Direct link to Symptoms") A non-watch request consistently fails after about 60 seconds. Failures near 5 seconds are a different symptom and normally indicate a peer dial failure. ### Cause[​](#cause-2 "Direct link to Cause") Non-watch requests are hard-capped at 60 seconds. A peer dial has its own 5-second timeout. A request that reaches the 60-second cap normally connected to the target but didn't receive a complete response in time. ### Resolution[​](#resolution-2 "Direct link to Resolution") * Check the target API server, its backing store, and admission webhooks for latency. * Compare `vcluster_proxy_request_duration_seconds` with `vcluster_proxy_peer_dial_duration_seconds`. A request near 60 seconds with a short peer dial isolates the delay to the target response path. * Treat repeated failures near 5 seconds as dial failures instead. After 3 consecutive failed dials to an endpoint, the proxy quarantines it and subsequent requests fail fast while it remains quarantined. ## Requests return 401 `auth failed`[​](#auth-failed "Direct link to auth-failed") ### Symptoms[​](#symptoms-3 "Direct link to Symptoms") A proxied request returns 401 with `auth failed`. ### Cause[​](#cause-3 "Direct link to Cause") The target rejected the client's platform access key. This is a platform trust issue, not a Kubernetes RBAC issue. ### Resolution[​](#resolution-3 "Direct link to Resolution") * Confirm both the client and the target are connected to the same platform and the project pairing is correct. `targetVirtualCluster.project` defaults to the client's own project when unset. * Restart the client's vCluster control plane pod to refresh the platform connection if the access key was rotated. ## Requests return 403 permission denied[​](#permission-denied "Direct link to Requests return 403 permission denied") ### Symptoms[​](#symptoms-4 "Direct link to Symptoms") An active proxied resource request returns 403, or vCluster Platform rejects the proxy target during configuration or registration. ### Cause[​](#cause-4 "Direct link to Cause") Kubernetes RBAC in the target tenant cluster governs active proxied requests. vCluster Platform target access is separate and checked with the submitting user's identity during client creation or update, then with the client tenant cluster's identity during registration. ### Resolution[​](#resolution-4 "Direct link to Resolution") The checks below that use `--as` require the current credentials to impersonate users in the cluster being queried. Run them with administrative credentials that have that permission; otherwise an impersonation denial tests the caller, not the client identity. 1. **Check target RBAC**: Verify that the target binding uses the client identity format `loft:vcluster::`, then test the target permission directly. For example, a client named `client` whose project namespace is `p-default` uses `loft:vcluster:p-default:client`: Test target RBAC permissions ``` vcluster connect --driver platform --project -- kubectl auth can-i create myresources.example.com \ --as="loft:vcluster:p-default:client" ``` 2. **Check creator or updater Platform RBAC**: If Platform rejected the proxy target configuration while creating or updating the client, check the identity currently authenticated to Platform. It needs `use` access to the target VirtualClusterInstance in the target project namespace: Check creator or updater Platform RBAC ``` kubectl auth can-i use virtualclusterinstances.management.loft.sh/target \ -n \ --context ``` 3. **Check client registration Platform RBAC**: If registration rejects the proxy target, run the same check as the client identity. For example: Check client registration Platform RBAC ``` kubectl auth can-i use virtualclusterinstances.management.loft.sh/target \ --as="loft:vcluster:p-default:client" \ -n \ --context ``` ## Existing target resources return 404[​](#resources-not-appearing "Direct link to Existing target resources return 404") ### Symptoms[​](#symptoms-5 "Direct link to Symptoms") A named request returns `404 resource not found` even though the object exists on the target, or a list or watch omits objects that exist there. ### Cause[​](#cause-5 "Direct link to Cause") With `accessResources: owned` (the default), the proxy hides anything it didn't create: * Named GET, PUT, PATCH, and DELETE requests for objects without both ownership labels return `404 resource not found` by design, even though the object exists on the target. Create requests are forwarded to the target, so a name collision normally returns 409. * Lists and watches are silently filtered to owned objects through an injected label selector. * **Ownership GET denied.** In `owned` mode, the proxy performs a target GET before a named PUT, PATCH, or DELETE to verify ownership. A 401 or 403 from that GET is masked as a 404. * **Namespace missing on the client.** Namespaced POST, PUT, and PATCH requests require the namespace to exist in the client tenant cluster first. For these requests, a namespace missing on the client returns 404. * **Genuine 404 from the target.** The object or the CustomResourceDefinition is truly absent, for example when the CRD isn't installed on the target. ### Resolution[​](#resolution-5 "Direct link to Resolution") * If the client should manage pre-existing objects, label them on the target with `vcluster.com/owner-vcluster=` and `vcluster.com/owner-project=`. Use the Platform project name for the second value, not the project namespace used in the client's authentication identity. These are plain labels, not an access-controlled boundary. Setting them grants the named client proxied management of the object, so restrict who can label objects on the target the same way you restrict other privileged operations there. - If the client should see everything, set `accessResources: all` for that entry. Understand the blast radius first. `all` allows mutating objects owned by other tenant clusters. - If the ownership GET is denied, grant target RBAC `get` together with the mutation verbs. If `get` is already allowed, check the platform access key as described in [Requests return 401 `auth failed`](#auth-failed). - If the namespace is missing on the client, create it there and retry the request. The proxy then creates it on the target on demand (log: `creating namespace on target cluster`). - For a genuine target 404, verify that the CustomResourceDefinition exists: Check CustomResourceDefinition in target ``` vcluster connect --driver platform --project -- kubectl get crd . ``` ## Discovery lists resources with only a get verb, or an empty list[​](#discovery-fallback "Direct link to Discovery lists resources with only a get verb, or an empty list") ### Symptoms[​](#symptoms-6 "Direct link to Symptoms") Discovery lists configured resources with only a `get` verb, or it returns an empty resource list for a wildcard entry. ### Cause[​](#cause-6 "Direct link to Cause") This is the locally served fallback discovery document, which means the target is unreachable, or answered discovery with an error, right now: * With explicit resource keys, the fallback lists your configured resources with a single `get` verb. * With a wildcard key (`*./`), the fallback is an empty resource list, because the proxy can't enumerate resources without the target. `kubectl api-resources` and other discovery-based commands return no resource kinds for that group and version during the outage. ### Resolution[​](#resolution-6 "Direct link to Resolution") `vcluster_proxy_discovery_fallbacks_total` confirms fallback without having to inspect the discovery document. Its `reason` label identifies where fallback began. `no_peer`, `stale_peers`, and `all_quarantined` mean the initial peer check failed, while `upstream_error` means the subsequent proxied discovery attempt returned 401, 404, or 5xx. That 5xx can be a target response or a transport failure converted to 5xx after the initial check. Check whether discovery is being served from the fallback ``` kubectl get --raw /metrics/syncer | grep vcluster_proxy_discovery_fallbacks_total ``` Full, accurate discovery returns automatically once the target is reachable. Treat fallback-shaped discovery as a symptom, and triage it with the [503 playbook](#target-unavailable). ## Connection upgrades return 501[​](#connection-upgrades "Direct link to Connection upgrades return 501") ### Symptoms[​](#symptoms-7 "Direct link to Symptoms") `exec`, `attach`, `port-forward`, or another upgrade-based request returns 501 with `resource proxy does not support connection upgrades`. ### Cause[​](#cause-7 "Direct link to Cause") The resource proxy doesn't support connection upgrades by design. ### Resolution[​](#resolution-7 "Direct link to Resolution") Access the target directly for those operations. ## The proxy doesn't start[​](#proxy-does-not-start "Direct link to The proxy doesn't start") ### Symptoms[​](#symptoms-8 "Direct link to Symptoms") The configured proxy routes aren't available, and the client logs don't show the expected startup sequence for their group and version. ### Cause[​](#cause-8 "Direct link to Cause") Check the client's vCluster control plane pod logs from startup: * **License**: The feature requires vCluster Enterprise with an online platform connection. Look for a feature or license error right at startup. The error text names a "vCluster pro feature", which reads like a paywall but is usually an image or license wiring problem. See [Resolve Pro feature license errors](https://vcluster.com/docs/vcluster/troubleshoot/pro-feature-license-error.md). * **Configuration rejected**: Invalid `experimental.proxy.customResources` entries fail validation at startup. The key format is `./` or `*./`, a group is required, `targetVirtualCluster.name` is required, and all entries for the same group and version must agree on target and access mode. * **Nothing enabled**: Entries with `enabled: false` don't produce proxy routes. They must still pass per-entry validation, including specifying `targetVirtualCluster.name`. If every entry is disabled, the proxy doesn't start. ### Resolution[​](#resolution-8 "Direct link to Resolution") Correct the license, platform connection, validation error, or disabled configuration identified in the logs, then apply the updated configuration. For each group and version, healthy startup logs show `Starting proxy server`, then `Starting target transport`, and at verbose log level `Connected to target` once the first probe succeeds. ## Deleting the proxy APIService or Service is denied[​](#deletion-denied "Direct link to Deleting the proxy APIService or Service is denied") ### Symptoms[​](#symptoms-9 "Direct link to Symptoms") Attempts to delete the proxy's APIService or its `kube-system` Service inside the client tenant cluster fail with: Admission denial message ``` deletion of vCluster-protected resources is denied; remove the corresponding entry from your vCluster configuration ``` ### Cause[​](#cause-9 "Direct link to Cause") Both objects are labeled `vcluster.loft.sh/protected=resource-proxy`, and a ValidatingAdmissionPolicy (`vcluster-protected-apiservices`, binding `vcluster-protected-resource-proxy`) intentionally denies DELETE requests on labeled objects. Deleting the APIService by hand doesn't fix proxy problems anyway. It only breaks discovery for the group and version until the leader re-registers it. ### Resolution[​](#resolution-9 "Direct link to Resolution") To decommission a proxied API group and version on purpose: 1. Remove every enabled entry for that API group and version from `experimental.proxy.customResources`, then apply the updated `vcluster.yaml`. The control plane pod restarts. If another enabled entry uses the same group and version, the shared APIService remains active. 2. The APIService for the removed group and version isn't garbage collected automatically. If it lingers and keeps that group and version unavailable, remove the protection label (updates are allowed, only deletes are denied). Then delete the APIService and the Service named in its `spec.service` field: Look the names up first rather than constructing them. Long Service names are shortened with a hash suffix, so `proxy--` doesn't always exist. Both objects carry the protection label: Find the orphaned proxy objects ``` kubectl get apiservice -l vcluster.loft.sh/protected=resource-proxy kubectl get service -n kube-system -l vcluster.loft.sh/protected=resource-proxy ``` Inspect the selected APIService to confirm the exact Service namespace and name before deleting either object: Find the Service used by an APIService ``` kubectl get apiservice \ -o jsonpath='{.spec.service.namespace}{"/"}{.spec.service.name}{"\n"}' ``` Remove orphaned proxy objects ``` kubectl label apiservice vcluster.loft.sh/protected- kubectl delete apiservice kubectl label -n kube-system service vcluster.loft.sh/protected- kubectl delete -n kube-system service ``` ## Log reference[​](#log-reference "Direct link to Log reference") All messages appear in the client's vCluster control plane pod ([syncer](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/sync)) logs. Search for the substrings in the first column: | Message | Meaning | Action | | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Starting proxy server` and `Starting target transport` | The proxy for a group and version started. | None. One pair per configured group and version. | | `peer cache did not sync in time for the startup probe` | No peer list arrived from the platform within 1 minute of startup. Resource requests return 503 until it arrives; discovery requests use the synthetic fallback. | Check platform connectivity and license. | | `initial probe of proxy target failed, requests will keep retrying` | The startup reachability check to the target failed. Serving continues, and requests retry individually. | Check target health. Transient during target restarts. | | `quarantining proxy target peer host after consecutive dial failures` | One endpoint left rotation for 20 seconds. The entry includes `host`, `consecutiveFailures`, and `target` fields. | Expected during target restarts. For sustained repetition, see [503 errors persist after a target restart or upgrade](#stale-peers). | | `all peer hosts for the proxy target are quarantined` | Resource requests to that target now fail fast with 503; discovery requests use the synthetic fallback. | Incident-grade. Triage with the [verdict table](#quick-triage). | | `peer watch subscription is down and no fresh netmap is available` | The platform peer feed dropped and the cached list expired. The entry includes a `lastNetmapAge` field. | Client-to-platform problem. Check platform reachability. | | `creating namespace on target cluster` | First namespaced POST, PUT, or PATCH after the namespace is present on the client but absent from the target. The proxy mirrored the namespace to the target. | None. | | `inject source labels failed` | A write body could not be parsed for ownership labeling (the ownership labels are called "source labels" internally), and the write proceeded without ownership labels. | Rare. In `owned` mode the created object is invisible to its creator. Inspect the caller's request encoding, and label the object manually to adopt it. | A note on hostnames like `kube...vcluster.invalid`. The `.invalid` address is an internal placeholder that is rewritten to a concrete peer endpoint on every request. Seeing it in error messages is normal and doesn't indicate a DNS misconfiguration. The error text around it, such as a dial timeout or no healthy peer, carries the real signal. For the proxy's fixed timing constants referenced above (dial timeout, quarantine duration, staleness tolerance), see [Fixed thresholds](https://vcluster.com/docs/vcluster/understand/resource-proxy.md#fixed-thresholds). --- # What happens during control plane outages? Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** * **[Private Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)** Running the control plane as a binary with **[vCluster Standalone](https://vcluster.com/docs/vcluster/deploy/control-plane/binary.md)**. When scaling with additional worker nodes, they are joined as **[private nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#private-nodes)**. Tenant cluster behavior during an outage depends on which control plane is unavailable. A vCluster deployment can include the tenant cluster control plane and the control plane cluster that hosts it. Each outage affects a different part of the system. Control plane outage failure domains for vCluster ## Tenant cluster control plane outage[​](#tenant-cluster-control-plane-outage "Direct link to Tenant cluster control plane outage") The tenant cluster control plane is the Kubernetes API server, controller manager, data store, and syncer for a specific tenant cluster. If this control plane is unavailable, tenant users can't reliably use the tenant cluster API. New API requests, scheduling decisions, controller reconciliation, status updates, and sync operations pause until the control plane becomes available again. | Capability | Status | Impact | | ----------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------ | | Tenant API access | Unavailable | Tenant users cannot reliably use kubectl, clients, or API-driven workflows. | | Scheduling and reconciliation | Unavailable | New scheduling decisions, controller reconciliation, and status propagation pause. | | Shared-node workloads | Limited | Existing pods can continue if the control plane cluster and its workers are healthy. | | Private-node workloads | Limited | Existing containers can continue, but kubelets and controllers depend on the tenant API for normal behavior. | | Standalone workloads | Limited | Workloads already running on worker nodes may continue, but Kubernetes API-driven operations pause. | Existing workloads are not deleted just because the tenant control plane is unavailable. What continues running depends on the worker node model: * **Shared nodes**: Workloads run as translated resources on the control plane cluster. If the control plane cluster and its worker nodes are still healthy, existing pods can continue running, but vCluster cannot create new translated resources or sync status while the tenant control plane is unavailable. * **Private nodes**: Workloads run on nodes dedicated to the tenant cluster. Existing containers can continue running at the node level, but kubelets and controllers need the tenant API server for normal Kubernetes behavior such as new scheduling, self-healing, scaling, and status updates. * **Standalone**: The tenant cluster behaves like a regular Kubernetes cluster. If its control plane is down, workloads already running on worker nodes may continue, but Kubernetes API-driven operations pause. Deploying vCluster in high availability mode reduces this risk by running multiple tenant control plane replicas with a highly available backing store. See [Deploy in high availability](https://vcluster.com/docs/vcluster/deploy/control-plane/kubernetes-pod/high-availability.md) for host nodes and private nodes, or [Deploy Standalone in high availability](https://vcluster.com/docs/vcluster/deploy/control-plane/binary/high-availability.md) for standalone. ## Control plane cluster outage[​](#control-plane-cluster-outage "Direct link to Control plane cluster outage") For vCluster deployments that run as pods on Kubernetes, the control plane cluster hosts the tenant cluster control plane pods and related resources. If the entire control plane cluster is unavailable, tenant control planes hosted there are also unavailable. | Capability | Status | Impact | | ---------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------- | | Hosted tenant control planes | Unavailable | Tenant control plane pods hosted on the unavailable cluster are unreachable. | | Tenant API access | Unavailable | Tenant API access is unavailable for tenant clusters hosted on that cluster. | | Resource syncing | Unavailable | The syncer cannot synchronize resources between the tenant cluster and the control plane cluster. | | Shared-node workloads | Limited | Usually affected because the translated workloads run on the same cluster. | | Private-node workloads | Limited | Runtime may continue on private nodes, but normal Kubernetes control-loop behavior depends on the tenant API. | High availability for an individual vCluster protects against failures such as a single vCluster pod or eligible node becoming unavailable. It does not protect tenant clusters from a full outage of the Kubernetes cluster that hosts their control planes. Existing workloads are not intentionally deleted by vCluster because of the outage. Recovery resumes when the control plane cluster and the tenant control plane pods become available again. The impact on workloads depends on where they run. With shared nodes, the same cluster usually hosts both tenant control planes and translated workloads, so a full control plane cluster outage can also affect workload execution. With private nodes, workloads run on separate tenant nodes, but normal Kubernetes control-loop behavior still depends on the tenant control plane being reachable. ## Prepare for outages[​](#prepare-for-outages "Direct link to Prepare for outages") For production tenant clusters, plan availability at each layer: * Run the tenant cluster control plane in high availability mode. * Use a highly available backing store such as deployed etcd, embedded etcd, or an external database. * For shared nodes and private nodes, run the control plane cluster itself as a highly available Kubernetes cluster. For standalone, run multiple control plane nodes. * Test recovery behavior for shared nodes, private nodes, and standalone, because workload impact differs by worker node model. --- # Why does CoreDNS become a DNS bottleneck? CoreDNS handles DNS resolution for every pod in the tenant cluster. By default, it runs a single replica (`controlPlane.coredns.deployment.replicas: 1`) as a separate Deployment, sized for small and medium tenant clusters. See [CoreDNS configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/coredns.md) for the full deployment and configuration reference, and [Scale CoreDNS for large node counts](https://vcluster.com/docs/vcluster/deploy/control-plane/coredns-scaling.md) for how to raise the ceiling described below. ## Why total query volume matters more than node count[​](#why-total-query-volume-matters-more-than-node-count "Direct link to Why total query volume matters more than node count") A single CoreDNS replica is CPU-bound at vCluster's default 1 CPU limit. What matters is total query volume, not node count on its own. Pod count, lookups per pod, and search-domain expansion all factor in. `ndots:5` search-domain expansion means a single unqualified lookup can generate several sequential queries before it resolves, so it's a significant multiplier on the raw pod-and-lookup count. Measure your own fleet's query rate rather than reasoning from node count alone. See [What to watch](https://vcluster.com/docs/vcluster/deploy/control-plane/coredns-scaling.md#what-to-watch) for the exact queries. ## Why GPU node pools are a common trigger, but not for the obvious reason[​](#why-gpu-node-pools-are-a-common-trigger-but-not-for-the-obvious-reason "Direct link to Why GPU node pools are a common trigger, but not for the obvious reason") GPU node pools are a common trigger for this, but not for the reason pod density suggests. GPU nodes typically run fewer pods per node than general-purpose nodes, so steady-state DNS load per node is often light. The pressure point is the job-launch burst. A batch job scheduling hundreds of pods at once produces a synchronized spike in DNS lookups. Every pod resolves the same handful of names within seconds of each other, so query rate spikes around scheduling events, not just steady-state averages. ## What an overloaded CoreDNS looks like[​](#what-an-overloaded-coredns-looks-like "Direct link to What an overloaded CoreDNS looks like") Tenant workloads see an overloaded CoreDNS as DNS resolution timeouts, intermittent `SERVFAIL` responses, and slower pod startup when init containers or sidecars wait on DNS lookups. Sustained `NXDOMAIN` for a Service the tenant API server can still resolve is a real signal. Ordinary `ndots:5` expansion doesn't explain a resolution failure for a name that exists. Treat it alongside `SERVFAIL` and latency rather than as a standalone capacity trigger, since `ndots:5` search-path expansion also produces `NXDOMAIN` as a normal byproduct of everyday lookups, whether or not CoreDNS is under load. Already seeing these symptoms right now? See [Resolve CoreDNS DNS timeouts and SERVFAIL errors](https://vcluster.com/docs/vcluster/troubleshoot/coredns-dns-timeouts.md) to diagnose and fix an active incident. ## Why embedded CoreDNS can't scale the same way[​](#why-embedded-coredns-cant-scale-the-same-way "Direct link to Why embedded CoreDNS can't scale the same way") With `controlPlane.coredns.embedded: true` (a Pro feature), CoreDNS components run as child processes of the syncer, in the same pod, instead of as their own Deployment. There's no `coredns` Deployment in `kube-system` for `controlPlane.coredns.deployment.replicas` or a Deployment-scaling autoscaler to act on, so both are silently ignored rather than rejected with an error. If you're running embedded CoreDNS and hitting a DNS capacity wall, two options add capacity instead of a replica count. Raising `controlPlane.statefulSet.resources` resizes the whole control-plane pod, not CoreDNS specifically, covered in [Choose a resource profile](https://vcluster.com/docs/vcluster/deploy/control-plane/sizing-and-performance.md#choose-a-resource-profile). Raising `controlPlane.statefulSet.highAvailability.replicas` also adds DNS capacity. More syncer replicas means more active CoreDNS instances, since the `kube-dns` Service points at the syncer pods (`TCP`/`UDP` 1053) rather than at CoreDNS Deployment pods when embedded is on. You can switch a live tenant cluster between embedded and separate CoreDNS without redeploying it. You may see a brief DNS gap during the switch. ## Related[​](#related "Direct link to Related") * [Scale CoreDNS for large node counts](https://vcluster.com/docs/vcluster/deploy/control-plane/coredns-scaling.md) — how to raise the replica count or run an autoscaler. * [Resolve CoreDNS DNS timeouts and SERVFAIL errors](https://vcluster.com/docs/vcluster/troubleshoot/coredns-dns-timeouts.md) — diagnose and fix an active incident. * [CoreDNS configuration](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/components/coredns.md) — full deployment and configuration reference. --- Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** # What is vCluster pod creation? In a vCluster, pod creation is not a direct process like in a standard Kubernetes cluster. Instead, it involves a unique mechanism that bridges your tenant cluster with the underlying control plane cluster. Imagine a vCluster as a virtual layer on top of a physical Kubernetes cluster. When you create a pod, it doesn't magically appear in the control plane cluster. Instead, it goes through a process that ensures your tenant cluster remains isolated while still using the resources of the control plane cluster. A pod in a vCluster isn't created directly in the control plane cluster but follows a well-defined process. This document explains the behind-the-scenes mechanism of pod creation in vCluster. ## Key components[​](#key-components "Direct link to Key components") The crucial components involved in vCluster pod creation are: * **Kube-apiserver (control plane cluster)**: Manages resources in the underlying physical cluster. * **Kube-apiserver (vCluster)**: Handles API requests within the tenant cluster. * **Syncer**: Bridges communication between vCluster and control plane cluster. 1. **Pod Creation in vCluster** Create a pod by making an API call to the vCluster kube-apiserver. The API server stores the pod in the vCluster backend storage (etcd). Example: ``` kubectl create -f my-pod.yaml -n my-vcluster ``` 2. **Syncer synchronization**. The syncer monitors events from the vCluster `kube-apiserver`. It communicates with the control plane cluster `kube-apiserver` to synchronize the pod creation. Key actions: * Watch vCluster events. * Translate vCluster pod configuration. * Initiate pod creation in control plane cluster. 3. **Control plane cluster pod deployment**. The control plane cluster processes the pod creation request: * Brings the pod up using standard Kubernetes procedures. * Sets the pod state to `ready`. * Treats the pod like any other cluster pod. Verification: ``` kubectl get pods -n host-cluster-namespace ``` 4. **Status synchronization**. After the pod reaches `state=ready`, the syncer: * Copies the pod's status from the control plane cluster. * Applies the status to the corresponding pod in the vCluster namespace. Status check: ``` kubectl describe pod -n my-vcluster ``` --- # How the resource proxy works Enterprise-Only Feature This feature is an Enterprise feature. See our [pricing plans](https://www.vcluster.com/pricing) or [contact our sales team](https://www.vcluster.com/enterprise-demo) for more information. vCluster Platform required This feature requires [vCluster Platform](https://vcluster.com/docs/platform/.md). Both the client and target tenant clusters must be managed as [VirtualClusterInstance](https://vcluster.com/docs/platform/api/resources/virtualclusterinstance/.md) within the platform. The Resource Proxy feature enables vCluster to proxy requests for custom resources defined by CustomResourceDefinitions (CRDs) to other tenant clusters. When enabled, the client tenant cluster transparently forwards resource operations to a target tenant cluster. The objects are stored there, where controllers running on the target can manage them. This enables cross-cluster communication patterns, centralized resource management, and tenant-isolated architectures. To configure the proxy, see [Configure the resource proxy](https://vcluster.com/docs/vcluster/configure/vcluster-yaml/experimental/proxy.md). To set one up end to end, see [Set up the resource proxy](https://vcluster.com/docs/vcluster/learn-how-to/resource-proxy-setup.md). ## Key capabilities[​](#key-capabilities "Direct link to Key capabilities") * **Transparent access**: Users interact with custom resources as if they were local to their tenant cluster. * **Centralized storage**: A target tenant cluster stores the custom resources routed to it. * **Tenant isolation**: Each client tenant cluster only sees objects with its ownership labels by default. * **RBAC enforcement**: The target tenant cluster enforces its own RBAC policies on proxied requests. ## How it works[​](#how-it-works "Direct link to How it works") When you configure a client tenant cluster to proxy custom resources, vCluster intercepts API requests for those resources. It forwards them to the target tenant cluster through vCluster Platform. Resource requests cross the vCluster Platform network from the client aggregation layer to the target API and custom resource The proxy performs several key functions: 1. **Request interception**: The client's Kubernetes [API server](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/) intercepts requests for configured API groups and versions and routes them to the proxy. One enabled resource entry activates routing for its entire group and version; target RBAC restricts access to individual resource kinds. 2. **Authentication**: The proxy authenticates to the target using the client's vCluster Platform identity. Its format is `loft:vcluster::`, for example `loft:vcluster:p-default:client` when the project namespace is `p-default`. 3. **Owner labeling**: On create and update operations, the proxy attempts to add owner labels that identify the client responsible for each resource. 4. **Visibility filtering**: For `owned` access, the proxy filters lists and watches and masks named GET, PUT, PATCH, and DELETE requests for objects that the client doesn't own. The `all` mode disables this filtering. 5. **Namespace synchronization**: Before a namespaced POST, PUT, or PATCH request, the proxy verifies that the namespace exists in the client tenant cluster and creates it in the target if needed. Mutual proxying is supported. Two tenant clusters can proxy resources from each other, each acting as the client in one direction and the target in the other. The two directions are independent, and the `target` metric label distinguishes them. See [Monitor the proxy](https://vcluster.com/docs/vcluster/manage/resource-proxy-operations.md#monitoring). ### Multi-client isolation[​](#multi-client-isolation-overview "Direct link to Multi-client isolation") When multiple client tenant clusters proxy to the same target, each client only sees objects with its ownership labels by default. The proxy normally adds those labels when the client writes an object and injects a matching label selector on lists and watches. Owner labels and selector injection give each client a filtered view of resources stored in the shared target To set this up, see [Set up multi-client isolation](https://vcluster.com/docs/vcluster/learn-how-to/resource-proxy-setup.md#multi-client). ### Proxy objects in the client tenant cluster[​](#proxy-objects "Direct link to Proxy objects in the client tenant cluster") The proxy wires itself into the client's Kubernetes aggregation layer. The [troubleshooting guide](https://vcluster.com/docs/vcluster/troubleshoot/resource-proxy.md) references these objects and conventions: | What | Where | Notes | | --------------------------------- | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | APIService `.` | Client tenant cluster | Registered by the elected leader replica. For example, `example.com/v1` uses `v1.example.com`. `kubectl get apiservice` is the first health signal. | | Service `proxy--` | `kube-system` namespace in the client tenant cluster | ExternalName Service pointing to `localhost`. Dots in the group become dashes, and long names are shortened with a hash suffix. | | Proxy port | `localhost:9100+n` inside the vCluster control plane pod | One proxy per group and version, sorted alphabetically so all replicas agree on port assignment. Not reachable from outside the pod. | | Ownership labels | Objects in the target tenant cluster | `vcluster.com/owner-vcluster=` and `vcluster.com/owner-project=`. The project label uses the Platform project name, not its Kubernetes namespace. The proxy attempts to inject both labels into POST, PUT, and PATCH bodies. If injection fails, the request continues without them. | | Deletion protection | Client tenant cluster | The APIService and proxy Service carry the `vcluster.loft.sh/protected=resource-proxy` label, and a ValidatingAdmissionPolicy denies deleting them. See [Deleting the proxy APIService or Service is denied](https://vcluster.com/docs/vcluster/troubleshoot/resource-proxy.md#deletion-denied). | ### Connection path and peer health[​](#connection-path "Direct link to Connection path and peer health") A **caller** is anything that talks to the client tenant cluster's API server: `kubectl`, a controller, or a CI job. A proxied request travels this path: 1. The caller sends a request to the client tenant cluster's API server. 2. The APIService registration routes the request through the `proxy--` Service to the resource proxy, which listens on a local port inside the client's vCluster control plane pod. 3. The proxy forwards the request over the platform network, using a direct WireGuard peer-to-peer connection where possible and the vCluster Platform DERP relay as a fallback. See [Prefer direct connections over the DERP relay](https://vcluster.com/docs/vcluster/manage/resource-proxy-operations.md#direct-connections). 4. The target tenant cluster's API server serves the request. The proxy discovers the target's control plane pods as peers on the platform network and distributes requests across them round-robin. Peer health is tracked per host. #### Fixed thresholds[​](#fixed-thresholds "Direct link to Fixed thresholds") This timing is fixed and not configurable: | Constant | Value | Effect | | ----------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Peer dial timeout | 5s | A single dial attempt gives up after this long, converting a hanging dial into a countable failure. | | Quarantine threshold | 3 consecutive dial failures | The endpoint leaves rotation. | | Quarantine duration | 20s | Then a single trial dial. Requests fail fast instead of hanging while a peer is quarantined. | | Peer list staleness tolerance | 60s | Measured from the moment the peer watch subscription drops, not from the last publication. A successful publication resets the clock. Past the tolerance, the proxy treats the cached peer list as stale and fails requests with 503 rather than trusting it. | | Non-watch request timeout | 60s | Hard cap on any single proxied request and response, end to end. Watches stream indefinitely and are exempt. | | `Retry-After` on 503 | 1s | Caller backoff hint. | | Proxy ports | 9100 and up | One per group and version, alphabetical, identical across replicas. | ### Behavior when the target is unreachable[​](#degraded-behavior "Direct link to Behavior when the target is unreachable") Two design decisions explain most confusing symptoms: 1. **Discovery stays available during common target failures.** When the target is unreachable, or answers discovery with 401, 404, or 5xx, the proxy serves discovery requests locally so the APIService stays `Available` and the client tenant cluster's API server keeps functioning. `kubectl api-resources` can therefore look healthy while every non-discovery request fails with 503. Use [metrics](https://vcluster.com/docs/vcluster/manage/resource-proxy-operations.md#monitoring) and [logs](https://vcluster.com/docs/vcluster/troubleshoot/resource-proxy.md#log-reference), not discovery, to judge proxy health. 2. **Non-watch resource failures degrade to 503 with `Retry-After: 1`.** Peer discovery problems, quarantined peers, a stale peer list, and target-side 5xx responses for non-watch resource requests surface as `503 Service Unavailable`. The discovery statuses listed above use the local fallback, and watch responses stream directly from the target. The cause of a non-watch 503 is distinguishable from logs and metrics, not from the error the caller sees. If you're chasing a specific symptom rather than reading up on the mechanism, start at the [troubleshooting guide](https://vcluster.com/docs/vcluster/troubleshoot/resource-proxy.md) instead. --- # What is allocatable in vCluster virtual scheduler Supported Configurations Running the control plane as a container with: * **[Shared Nodes](https://vcluster.com/docs/vcluster/introduction/architecture.md#shared-nodes)** ## What are allocatable resources on a node?[​](#what-are-allocatable-resources-on-a-node "Direct link to What are allocatable resources on a node?") Allocatable on a Kubernetes node represents the compute resources available for pods. The scheduler prevents over-subscribing resources beyond the allocatable value. It supports three primary resource types: * CPU * Memory * Ephemeral storage ## What is vCluster virtual scheduler?[​](#what-is-vcluster-virtual-scheduler "Direct link to What is vCluster virtual scheduler?") By default, vCluster uses the control plane cluster scheduler to determine pod placement. This approach offers two key benefits: * Saves computing resources. * Allows pods to run on any node. However, the control plane cluster scheduler has limitations detailed in the [virtual scheduler documentation](https://www.vcluster.com/docs/vcluster/configure/vcluster-yaml/control-plane/other/advanced/virtual-scheduler). The virtual scheduler provides an alternative by: * Making independent pod allocation decisions. * Allowing taints and labels on tenant cluster nodes. * Preventing modifications to the control plane cluster nodes. ## The challenge with allocatable values[​](#the-challenge-with-allocatable-values "Direct link to The challenge with allocatable values") When using the vCluster virtual scheduler, you'll notice a significant difference between node allocatable values in the control plane cluster and vCluster context. Consider this scenario: * Kubernetes cluster with three nodes * vCluster assigned to use only two nodes * Focus on CPU allocation for simplicity ### Control plane cluster perspective[​](#control-plane-cluster-perspective "Direct link to Control plane cluster perspective") In the control plane Kubernetes cluster context, the first node shows 62.5 allocatable cpus. This value is calculated by: * Total node cpus: 64 * Minus `kubeReserved` and `systemReserved` values ### vCluster perspective[​](#vcluster-perspective "Direct link to vCluster perspective") In the vCluster context, the same node shows an allocatable value of 30.5 or lower. ## Why the allocatable values differ[​](#why-the-allocatable-values-differ "Direct link to Why the allocatable values differ") The vCluster scheduler uses the allocatable value to prevent over-subscribing resources. * Must reflect resources available specifically to vCluster * Cannot use the entire control plane cluster's allocatable value * Node details show only what vCluster knows and controls Using the full allocatable value of the control plane cluster would cause the vCluster scheduler to over-subscribe the node. ## How vCluster manages allocatable values[​](#how-vcluster-manages-allocatable-values "Direct link to How vCluster manages allocatable values") The vCluster syncer implements a dynamic update mechanism: * Regularly reads node information from the control plane cluster. * Updates node information in the vCluster. * Ensures allocatable values change as pods start and stop in the control plane cluster context. --- # How vCluster and vMetal fit together vCluster helps AI cloud providers deliver highly scalable, resilient compute clusters as self-service tenant products, including managed Kubernetes, Slurm, inference, Ray, and other cluster types. Clusters provide the compute experience, while vCluster Platform manages tenants, capacity, templates, access, multi-tenant observability, and lifecycle operations. vMetal connects vCluster to physical and virtual infrastructure. It presents a stable Machine API to the layers above it and translates Machine requests through infrastructure-specific compute and network drivers. This page maps these responsibilities and routes you to the canonical guide for each product. This is a routing page, not an implementation guide. For setup and configuration, follow the links in each section. ## From product offering to infrastructure[​](#from-product-offering-to-infrastructure "Direct link to From product offering to infrastructure") When the products are used together, a provider publishes cluster templates, access rules, and capacity policies through vCluster Platform. A tenant selects a managed Kubernetes, Slurm, inference, or other compute offering, and the platform applies the appropriate configuration and capacity. When a cluster needs infrastructure, demand becomes a Machine request through the stable Machine API. vMetal sends that request to the selected compute driver, which provisions a physical or virtual Machine. Network drivers and integrations provide the required connectivity during that process. ## Products and what they own[​](#products-and-what-they-own "Direct link to Products and what they own") ### vCluster Platform manages tenants and capacity[​](#vcluster-platform-manages-tenants-and-capacity "Direct link to vCluster Platform manages tenants and capacity") Platform is the self-service management plane for clusters. It manages projects, templates, capacity policy, access control, multi-tenant observability, and billing across connected control plane clusters. Platform is where a provider defines which cluster products tenants can create and what capacity those products can consume. Start with [What is vCluster Platform?](https://vcluster.com/docs/platform/.md) ### vCluster powers clusters[​](#vcluster-powers-clusters "Direct link to vCluster powers clusters") vCluster powers managed Kubernetes, Slurm, inference, Ray, and other compute cluster products. Each tenant receives an isolated cluster experience based on the selected offering. For managed Kubernetes, workloads can use shared or private nodes, and Auto Nodes can request Machines through the stable Machine API as demand changes. Start with [What is vCluster?](https://vcluster.com/docs/vcluster/introduction/what-is-vcluster.md) ### vMetal provisions and manages Machines[​](#vmetal-provisions-and-manages-machines "Direct link to vMetal provisions and manages Machines") vMetal exposes a stable Machine API independent of the infrastructure underneath it. Each Machine is a claim on one physical or virtual server that moves through a driver-neutral lifecycle from request to deletion. Compute drivers translate those requests into infrastructure-specific operations, including bare metal provisioning with Metal3 and virtual-machine provisioning with KubeVirt. Network drivers and integrations support connectivity during provisioning. Naming a driver here describes its role in the architecture, not its shipping or support status. Check the vMetal docs for what's available today. Auto Nodes is one way vCluster requests Machines from vMetal, not the definition of Machine provisioning. Start with the [vMetal overview](https://www.vmetal.ai/docs/) and the [Machine and server concepts](https://www.vmetal.ai/docs/plan/concepts/machine). ## Where to go next[​](#where-to-go-next "Direct link to Where to go next") | If you want to | Go to | | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | Match your use case, inference, AI cloud, CI/CD, and more, to a build and operate path | [Production guide](https://vcluster.com/docs/vcluster/production-guide/overview.md) | | Design a tenant-cluster platform on infrastructure you already run | [vCluster architecture](https://vcluster.com/docs/vcluster/introduction/architecture.md) | | Offer self-service clusters and manage tenants, capacity, access, and observability | [vCluster Platform](https://vcluster.com/docs/platform/.md) | | Provision physical or virtual Machines for clusters | [vMetal overview](https://www.vmetal.ai/docs/) | | Understand the Machine and server model before choosing a driver | [Machine and server concepts](https://www.vmetal.ai/docs/plan/concepts/machine) | | Create an isolated Kubernetes cluster for local development or CI | [Docker (vind) quick start](https://vcluster.com/docs/vcluster/quick-start/docker.md) | --- # vCluster Labs egress IP addresses Allowlist all four IPv4 addresses so vCluster Labs can access your infrastructure. ``` 63.183.170.86/32 18.196.66.114/32 3.19.214.126/32 18.227.187.66/32 ``` --- # vCluster Docs ![vCluster](/docs/media/rebranding/vCluster_horizontal-orange.svg) vCluster powers highly scalable, resilient compute clusters and gives AI cloud providers a self-service platform for offering them to tenants. Deliver managed Kubernetes, Slurm, inference, and other cluster products with tenant management, capacity management, multi-tenant observability, access control, and lifecycle automation built in. Add vMetal to expose physical and virtual infrastructure through a stable Machine API backed by pluggable compute and network drivers. Search... ## Where do you want to start?[​](#where-do-you-want-to-start "Direct link to Where do you want to start?") [rocket\_launch](https://vcluster.com/docs/vcluster/.md) ### [Prove vCluster on your infrastructure](https://vcluster.com/docs/vcluster/.md) [Start with the hardware you have or Docker and launch your first managed compute cluster.](https://vcluster.com/docs/vcluster/.md)[route](https://vcluster.com/docs/vcluster/production-guide/overview.md) ### [Find your production path](https://vcluster.com/docs/vcluster/production-guide/overview.md) [Match your use case, inference, AI cloud, CI/CD, and more, to a concrete build and operate path.](https://vcluster.com/docs/vcluster/production-guide/overview.md)[dns](https://www.vmetal.ai/docs/) ### [Provision your infrastructure](https://www.vmetal.ai/docs/) [Turn bare metal or virtual machines, GPUs included, into cluster capacity.](https://www.vmetal.ai/docs/)[hub](https://vcluster.com/docs/platform/.md) ### [Offer self-service clusters](https://vcluster.com/docs/platform/.md) [Manage tenants, capacity, templates, access, and multi-tenant observability from one platform.](https://vcluster.com/docs/platform/.md) Not sure where to start? See [how the products fit together](https://vcluster.com/docs/how-the-products-fit-together.md) to understand how they divide responsibility, then pick the entry point that matches your task. ## How the products fit together[​](#how-the-products-fit-together "Direct link to How the products fit together") ### vCluster Managed Kubernetes, Slurm, inference, and other compute clusters with hyperscaler-grade isolation and reliability. [What is vCluster?](https://vcluster.com/docs/vcluster/introduction/what-is-vcluster.md) ### vCluster Platform Provides self-service tenant management, capacity management, templates, multi-tenant observability, policy, and access control. [What is vCluster Platform?](https://vcluster.com/docs/platform/.md) ### vMetal Exposes physical and virtual infrastructure through a stable Machine API backed by interchangeable provisioning and network drivers. [vMetal overview](https://www.vmetal.ai/docs/) Clusters and vCluster Platform combine to form vCluster. vMetal connects vCluster to physical and virtual infrastructure through a stable Machine API. See [how the products fit together](https://vcluster.com/docs/how-the-products-fit-together.md) for the full request-to-infrastructure workflow, the boundaries between each product, and where to go next. ## Try vCluster Platform Create your own platform instance and offer self-service compute clusters to tenants. Manage tenants, capacity, access, and operations across all your environments. [Get Started →](https://vcluster.cloud) ---